// Analyse · Understand · Learn

Run the analysis. Understand every result.

Bioinformaniac is where your sequencing data becomes answers, and where you learn the science behind them along the way. Seven expert pipelines run on our servers, from microbiomes and RNA-seq to bacterial, fungal, insect and human genomes. A 53-chapter handbook and a game you can play explain every step, every statistic and every figure they produce. No installing, no code, no black box.

  • CompaniesStrain safety dossiers, genome QC and reports you can hand to a regulator or a client.
  • Researchers & studentsFigures, statistics and a methods section with references, ready for your paper or thesis.
  • Curious mindsPlay the Learning Lab: 13 topics, 80 levels and a robot guide, from DNA basics to expert statistics.
  • Free alpha: every tool with real example results
  • Handbook and Learning Lab free, no card
  • Your data stays private to your account
OutputFig.01 · example runMR-01 / BF-02
AGenus composition · control vs fibre
BComplete genome · 4.2 Mb
PERMANOVA R²0.446
p value0.005
Contigs1
Complete99.6%
01 // Start with your data

What do you have?

Pick the data you have. Each tool opens with an example study, so you can see the results before you upload anything.

Want every method and setting? Compare the seven tools in detail.

02 // Why Bioinformaniac

The analysis you need, done properly

No setup, defensible methods, our servers, private files, everything downloadable, and you learn as you go.

No setup at all

No software to install, no servers or  environments to manage. Open the page and start.

Methods you can defend

Established tools (, ANCOM-BC, , , …) with sensible defaults, plus a written methods section with references.

Runs on our machines

Heavy jobs are queued and run on our servers. Close the tab: the job keeps going, and the results wait in your account.

Private by default

Your files live in your own space. Nobody else sees them, and you can download or delete everything at any time.

Everything downloadable

Figures, tables, genomes, reports, or a whole results folder as one .zip.

Learn as you go

Each step explains what it does. The 53-chapter handbook covers every method and the statistics behind it, and the Learning Lab turns it into a game.

03 // How it works

Three steps to a result

Try or sign up, upload your data, get your results.

STEP 01

Try or sign up

Explore the example studies as a guest. A free account takes your name, e-mail and a password.

STEP 02

Upload your data

Drag in tables or reads. Large files are sent in pieces, so a dropped connection does not lose your upload.

STEP 03

Get your results

Run step by step or start a pipeline and come back later. Download figures, tables and reports.

04 // Plans

Free during the alpha. More on request.

Guests and free accounts pay nothing; full access to every tool on request.

Guest

€0
  • Every example study
  • The handbook
  • Learning Lab: General topic

Free account

€0
  • All 13 Learning Lab topics
  • MicroRes on your own tables
  • Ask for a tool: 7-hour run windows
  • No credit card

On request

Ask us
  • Full access to every tool
  • Up to 12 CPUs · 28 GB RAM
  • Group, company and course accounts
  • Downloads and full reports
See the details
05 // Questions

Good to know

Coding, privacy, papers, storage and how long a genome takes.

Do I need to know how to code?

No. Everything runs from the browser. Each step explains its settings in plain words, and the defaults follow current practice.

Is my data private?

Yes. Your uploads and results sit in a folder only your account can open. You can download everything or delete your account and all its files from the Account page.

Can I use the results in a paper?

Yes. MicroRes, RNA-realm, FungFlow, Humaniac and Insectra write a methods section that names every tool, version and setting, with references. Please cite the underlying tools as listed.

What happens when my storage is full?

You can use all of it. When it is full, new uploads and runs stop with a clear message; delete old files in My data or ask us for more space.

How long does a genome take?

A bacterial genome usually takes under an hour, a fungal genome a few hours. Jobs run on our servers, so you can close the page and come back.

I have a question that is not here.

Write to us from the Contact page, or from Messages (the envelope at the top) when you are signed in. The answer arrives in the same conversation.

Your data deserves a proper analysis. You deserve to understand it.

Free account, no card. Explore real results today, and learn the science while you are at it.

// Product specs

What each engine does

All six come with your account. Steps marked demo are open to guests.

MR-01

MicroRes

A guided pipeline for microbiome data with two modules, taxonomic analysis and functional metagenomics. Every step writes a methods paragraph with references into your report.

  • InputOne table or many. Several files are merged on shared features, missing ones are 0, and each one-sample file becomes a column named after the file.
  • FiguresInteractive, recolourable, export as PNG, PDF or SVG
  • ReportFull HTML or PDF report, Cytoscape files for networks

Taxonomic analysis 13 steps

  1. Import: counts, metadata, taxonomy, tree and sequences demo
  2. Quality control with a live filter preview demo
  3. Contaminant screening (decontam) demo
  4. Normalise and transform demo
  5. Composition demo
  6. Alpha diversity demo
  7. Beta diversity: PCoA, NMDS, dbRDA, PERMANOVA demo
  8. Phylogeny
  9. Core microbiome
  10. Differential abundance with consensus
  11. Associations
  12. Co-occurrence networks
  13. Biomarkers and prediction

Functional metagenomics 7 steps

  1. Import HUMAnN, PICRUSt2,  or any function table demo
  2. Clean and normalise demo
  3. Profiles and functional diversity demo
  4. Functional beta diversity: PCoA, PCA, NMDS, dbRDA, PERMANOVA, dispersion demo
  5. Differential functions with covariates and random effects
  6. Contributions and functional redundancy
  7. Species to function networks, nestedness and keystone carriers
BF-02

BactFlow

A graphical  workflow for bacterial isolate genomes. Upload reads to your private folder, choose the steps you need and follow each run live.

  • ReadsShort reads, long reads or both (hybrid assembly)
  • LiveTerminal, CPU and memory gauges while a run is going
  • QueueOne pipeline at a time on the server, so each run gets the whole machine

Quality control demo

  • Illumina, Nanopore or PacBio reads
  • Pool barcode folders into one file per sample
  • Length and quality filtering
  • Read statistics and quality plots

Assembly demo results

  • , ,  and PacBio 
  •  polishing for older flow cells (R9.4.1, R10.3; off for R10) and genome circularisation
  •  assembly metrics and contig reports

Post-assembly demo results

  •  annotation,  completeness,  taxonomy
  •  typing, plasmids and viruses with 
  • SNP and structural variant calling against a reference
  • Linear and circular genome maps, gene heatmaps, HTML and PDF report
RR-03

RNA-realm

Bulk RNA-seq analysis following current Bioconductor practice, without writing R. Every step adds a methods paragraph with references.

  • InputFASTQ () or counts from , , , , ,  or any matrix; one file or one per sample
  • SpeciesAny. Human and mouse IDs get symbols, gene types and mitochondrial genes automatically
  • ExportsFiltered counts, normalised matrix, DE tables, enrichment tables, modules, methods text

Data and quality 4 steps

  1. Reads to counts: ,  (selective alignment, GC and sequence bias), 
  2. Import and merge count files demo
  3. Quality control: depth, gene types, mitochondrial and rRNA shares, outliers demo
  4. , TMM or median-of-ratios, VST demo

Analysis 4 steps

  1. PCA, drivers of each component, clustering, batch-removed views demo
  2. Differential expression: , -, Wilcoxon, consensus demo
  3. Pathways: preranked  and ORA demo
  4. Co-expression modules, trait correlations, hub genes demo
FF-04

FungFlow

A guided pipeline for fungal isolate genomes. Upload reads to your private folder, choose the stages and follow the run live; results open in an interactive report.

  • ReadsIllumina paired-end, Nanopore, PacBio HiFi, or hybrid
  • Or start fromAn assembly you already have (FASTA)
  • ReportAssembly and completeness metrics, gene and repeat statistics, CAZymes, clusters, downloads

Reads and assembly demo results

  • , , 
  • k-mer profile with : size, heterozygosity, ploidy
  •  (long reads) or  (short reads);  polishing

Quality demo results

  • ,  (fungi or a narrower lineage)
  • Telomere repeats per contig
  •  contig classes, mitochondrial contigs, ITS with 

Annotation demo results

  •  and 
  • Gene prediction ( or ), tRNAs
  • , CAZymes (),  clusters
HM-05

Humaniac

Human genomics on GRCh38, from reads to an interpreted genome. Six modules run alone or chained, each step its own  job; results open in an interactive report with an IGV genome viewer, and a PDF report with methods and the run log.

  • InputIllumina, Nanopore or PacBio HiFi reads of a genome, exome or panel; or a BAM/CRAM, a VCF or gVCFs of a family
  • ReferenceGRCh38 no-alt analysis set, Ensembl  116, , , , HPO, 
  • DemoThe GIAB Ashkenazi trio (HG002 and parents) over a gene panel, from raw reads

Alignment and small variants demo results

  •  or ,  or , duplicate marking
  •  coverage, genetic sex,  contamination
  • ,   or ;  phasing; GIAB benchmark with 

Structural variants demo results

  •  or ,  or 
  •  or  for disease repeats (HTT, FMR1, C9orf72 …)
  •  for SMN1/2, GBA1 and other paralogous genes

Interpretation, families, cancer demo results

  •  with MANE, , , ; ACMG/AMP evidence; HPO gene ranking; ACMG SF v3.2
  •  pharmacogenomics, , genetic ancestry, runs of homozygosity
  • ,  and  for trios;  for tumour–normal pairs
IN-06

Insectra

Insect genomics: one insect to a chromosome-level, annotated genome, or many insects to the structure and selection of their populations. Five modules, run alone or chained, each step its own  job.

  • ReadsPacBio HiFi or Nanopore R10.4.1, R10.3 or R9.4.1 (assembly), Illumina pairs (k-mers, QV, populations), Hi-C pairs (scaffolding)
  • Or start fromAn assembly (FASTA), or a reference genome and its GFF3 for population studies
  • DemoA fruit-fly genome slice and two fly populations with insecticide-resistance variants (simulated reads)

Reads and assembly demo results

  • , , ;  genome profile
  •  or , ,  Hi-C scaffolding
  •  ( odb12),  QV, telomeres,  contaminant contigs (Wolbachia …)

Mitogenome and genes demo results

  • : circularised, annotated mitogenome (invertebrate code)
  •  and ;  invertebrate gene models
  •  on proteins,  names, GO and KEGG

Population genomics demo results

  •  per individual,  joint calling with invariant sites
  • PCA, , KING kinship; pi, Fst, dxy and Tajima's D in windows ()
  •  effects and protein changes in resistance genes (para/kdr, Ace, Rdl, Cyp6, GSTs)
BM-07 · free preview

Bactomaniac

The bacterial strain safety dossier: is this strain what you think it is, is its genome good enough, does it carry resistance, toxin or virulence genes and could they spread, and is it still the same strain? Four verdicts with their evidence, laid out along the EFSA requirements for whole-genome analysis of microorganisms in the food chain.

  • ReadsIllumina pairs, Nanopore (choose R10.4.1, R10.3 or R9.4.1) or PacBio HiFi, or long and short reads together (hybrid)
  • Or start fromAn assembly (FASTA); add a reference genome (FASTA or ) to check stability
  • ReportHTML, PDF and editable Word: verdicts, genome map, EFSA checklist, methods, run log, input checksums and tool version history
  • ValidatedA reference panel of public genomes with known resistance and toxin genes, scored on every release

Quality and identity

  • , ; , or  with  ( only for R9.4.1 and R10.3); 
  •  completeness and contamination, depth per contig
  •  ANI against every  r232 species representative, type material flagged; 

Safety, in context

  •  genes and point mutations;  virulence; genus toxin panels (e.g. Bacillus cereus Hbl, Nhe, CytK, cereulide)
  •  and  plasmids and prophages; 
  • Every resistance gene placed on its replicon and rated for transfer risk

Stability

  • Comparison with a reference or parent strain: SNPs, indels, structural changes and lost replicons (,  + )
// Validation of pipeline

Does it get known strains right?

We run our pipelines on public genomes with a documented biology: strains that make toxins, strains that are safe, strains that carry resistance on a plasmid. Every answer is scored against a truth that does not come from our own software, against criteria we wrote down before the first result existed.

Loading the validation…

Questions a partner asks

Can I run the validation myself?

Yes. The evidence pack lists every strain by its public  accession and every expected result. Run those genomes through our tools, or through your own, and compare.

Can you validate on our strains or our standard?

Yes. Send us the reference strains or your acceptance criteria through Contact. We run them and give you the same kind of report, with your criteria fixed before we look at the result.

How often is it repeated?

Whenever a release changes the analysis. The date of the last run and the software versions are shown above.

Does a passed validation mean a strain is safe?

No. It shows that the tools find what is known to be there, and nothing that is not. Whether a strain may be released is your decision; this is evidence for it.

// Pricing

Plans

During the free alpha every account is free. You can explore every tool with real example results, use MicroRes on your own tables, and play every Learning Lab topic. To run a tool on your own data, ask for access from inside the tool: we open it for 7-hour windows while we budget the server load. Full capacity for every account is coming soon.

Runs are only limited by your free storage and the CPUs and memory of your account. Jobs wait in a queue when the servers are busy and start as soon as there is room.

IncludedFree accountOn request
MicroRes, RNA-realm, BactFlow, FungFlow, Humaniac and InsectraAll stepsAll steps
Runs (genomes, read jobs, pipelines)No limitNo limit
CPUs and RAM per analysis4 CPUs · 8 GBUp to 12 CPUs · 28 GB
Storage5 GBUp to 1 TB
HelpMessages from your accountMessages, plus help with methods

Questions

How many runs can I start?

As many as you like. A run needs free storage for its results, and it uses at most the CPUs and memory of your account. When the servers are busy, your job waits in the queue and starts on its own.

How do I get more resources?

Click Send a request above, or write from Messages (the envelope at the top). Say what you analyse and what you need; we answer in the same conversation and raise the limits of your account.

Will it stay free?

The Free account stays. Paid plans with more resources will come later; you will hear from us before anything changes, and nothing is charged without your consent.

Where is my data processed?

On our server in the EU. Your files are private to your account and are never used to train models or shared. See the Privacy Policy.

Do you offer group or course accounts?

Yes. Send a request for a lab or institution and tell us how many people need access.

// Messages

Messages

Write to the Bioinformaniac team: a question, a problem, or a request for more storage, CPUs or memory. Answers arrive below.

Send a message or request

Received

Your conversations with the team. Reply, mark as read, or delete them from your inbox.
Loading…
Download PDF
Part I · Foundations

Microbial bioinformatics in one picture

Every analysis in this handbook turns DNA sequence into a biological statement. Knowing where each step sits in that chain makes it easier to choose methods, and to notice when a result is coming from the pipeline rather than from the biology.

Two questions, two kinds of data

Most microbial sequencing projects ask one of two questions. The first is who is there and what are they doing in a mixed community: a stool sample, a soil core, a fermenting food. This is the domain of microbiome analysis. It uses either marker-gene (amplicon) sequencing, usually of the 16S rRNA gene for bacteria and archaea or ITS and 18S for fungi and other eukaryotes, or shotgun metagenomics, where all DNA in the sample is sequenced. The second question is what is this organism: the complete genome of a single bacterial isolate, its taxonomy, its genes, its plasmids and how it differs from its relatives. This is isolate genomics.

A third question has become routine in industry and in clinical research: which viruses are present, and is the sample safe. Viral metagenomics asks it of a stool sample, a wastewater sample, a fermentation or a phage preparation, and it needs its own methods, because viruses have no universal marker gene, their genomes are often incomplete in an assembly, and a virus that infects a bacterium can carry toxin or resistance genes into it. Two safety questions cut across all three fields: is a resistance gene present (antimicrobial resistance, AMR) and is it where it can spread.

The fields share tools and ideas, such as quality control of reads, reference databases and the need for careful controls, but their statistics are very different. A microbiome study compares many samples and has to deal with sparse, compositional count tables. A genome project usually deals with a handful of genomes, and the hard part is getting each sequence right.

The chain of steps

For a microbiome study the chain runs: study design and sampling, DNA extraction, library preparation and sequencing, read quality control, feature construction (amplicon sequence variants or taxonomic profiles), taxonomy and function assignment, filtering and contaminant removal, normalisation and transformation, then the ecological and statistical questions: diversity, community differences, differentially abundant features, associations, networks and prediction. Chapters 2 to 13 follow this chain. For a virome the same chain holds, with two additions: a check that each viral genome is complete enough and really present (not just a few reads on one gene), and a link from each virus to its host and lifestyle. For resistance the chain adds a screen of the functional profile or of the assembled genes against a curated list of resistance determinants, followed by a verdict that says how far down it can be trusted: drug class, mechanism, pathway and gene.

For an isolate genome it runs: culture and DNA extraction, sequencing with short reads, long reads or both, read quality control, assembly, polishing, quality assessment, taxonomic placement, annotation, then targeted questions such as typing, plasmids, resistance genes and comparison with reference strains. Chapters 14 to 21 follow that chain.

What "literature-based" means here

Each chapter explains the reasoning behind a step, the main methods and the trade-offs between them, with references to the papers that introduced or tested them. Where the field has not settled on one answer, for example on rarefaction or on which differential abundance test to trust, the chapter says so and describes what careful authors do instead of pretending there is a single correct choice. Boxes marked In practice give concrete recommendations; boxes marked Pitfall describe mistakes that are common in published work.

How the chapters map onto the tools

The handbook and the tools follow the same chains, so you can read a chapter and then run its step. Bioinformaniac has eight tools; each answers one of the questions above.

ToolQuestionWhat is in itChapters
MicroRes · taxonomic moduleWho is there in a mixed community?Steps 01 to 14: import, quality control, contaminants, normalisation, composition, alpha and beta diversity, phylogeny, core microbiome, differential abundance, associations, networks, biomarkers, pangenome.4 to 13
MicroRes · functional moduleWhat can the community do, and does it carry resistance?Steps F1 to F8: import (HUMAnN, PICRUSt2, ), cleaning, profiles, functional beta diversity, differential functions, who carries each function, the species-function atlas, and the antimicrobial resistance screen with a verdict for each sample.10 to 13
MicroRes · viral moduleWhich viruses are present, whom do they infect, are they safe?Steps V1 to V7: import, genome quality and presence, composition, hosts and virus-host links, lifestyle and safety screen (toxin, resistance and lysogeny genes), a study report and a clinical report.V1 to V3
BactFlowWhat is this isolate?Quality control, assembly and post-assembly: typing, plasmids, resistance, comparison.14 to 21
BactomaniacIs this bacterial strain safe to use?A strain safety dossier: identity, resistance with its genomic context, toxins, quality and stability.14 to 21
RNA-realmWhich genes change between conditions?RNA-seq from counts or reads to differential expression, pathways and co-expression, with covariates and a random effect for repeated measures.24 to 29
FungFlow, Humaniac, InsectraWhat is this fungal, human or insect genome?Assembly, annotation and the genome browser, with biosynthetic clusters for fungi.30 to 35

You can read the handbook on its own, but every chapter ends with a short note on where the step lives in the tools.

Two things that make a result reusable: the report and the code

A result that cannot be explained or repeated is not an industrial result. Every tool therefore ends in a report, and MicroRes and RNA-realm write theirs as a story: chapters in a fixed order, the findings first, each figure with a sentence that says what it shows, the cautions next to the findings they qualify, and a methods chapter that lists every setting, every software version and the unique ID of the report. Because the text is produced from the measured results by fixed rules, a number in the report is always a number from your data. The same layout is available as a web page, a PDF and a Markdown file.

Reveal the code is the second half of the promise. Each tool can show, for the analysis you just ran, the exact code or command that produced it, as a script that you can read, run on your own machine and keep with the paper or the dossier. Reading it is also the fastest way to learn what a step really does. Reveal the code is available to administrators and to accounts with full access; in the free release the button is visible but dimmed.

The same care applies to the analysis itself. When your samples are not independent (the same person sampled before and after a treatment, animals in the same cage), say so: MicroRes and RNA-realm let you name the covariates that must be adjusted for and a random effect for the repeated subject, and the report records the choice.

How to use these docs to learn

Each chapter starts with what you will learn, explains the idea before the tool, works through at least one small example with real numbers, lists the common mistakes and ends with questions to test yourself. Book references in the text and in the reference list name the chapter to read next: the docs are a map, the books are the territory. A sensible order for a newcomer is Part I (foundations), then the part for your data type, then Part IV on practice. The statistics are best learned from Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R and Xia & Sun, Bioinformatic and Statistical Analysis of Microbiome Data for microbiomes, Efron & Hastie, Computer Age Statistical Inference for large-scale testing, and James et al., An Introduction to Statistical Learning for machine learning; the sequence side from Ismail, Bioinformatics: A Practical Guide to NGS Data Analysis, Low & Tammi, Practical Bioinformatics for Beginners and Christensen (ed.), Introduction to Bioinformatics in Microbiology.

Check your understanding

Why is a microbiome study not just a list of species?

Because counts are relative, noisy and shaped by every step from extraction to database. A conclusion is only as good as the weakest step of the chain.

What separates an isolate genome project from a microbiome project?

An isolate project reconstructs one organism completely (assembly, annotation); a microbiome project measures many organisms partially and compares samples statistically.

Key references

  1. Knight R, Vrbanac A, Taylor BC, et al. Best practices for analysing microbiomes. Nature Reviews Microbiology 16, 410–422 (2018).
  2. Quince C, Walker AW, Simpson JT, Loman NJ, Segata N. Shotgun metagenomics, from sampling to analysis. Nature Biotechnology 35, 833–844 (2017).
  3. Bharti R, Grimm DG. Current challenges and best-practice protocols for microbiome analysis. Briefings in Bioinformatics 22, 178–193 (2021).
  4. Christensen H (ed.). Introduction to Bioinformatics in Microbiology. Learning Materials in Biosciences, Springer, 2018. Read: Ch. 1 Introduction to Bioinformatics in Microbiology: why the field exists and how sequence data flow through it.
  5. Wang Z. Introduction to Computational Metagenomics. World Scientific, 2022. Read: Ch. 1–3: what metagenomics is and why its data are a computational problem.
  6. Xia Y, Sun J, Chen D-G. Statistical Analysis of Microbiome Data with R. ICSA Book Series in Statistics, Springer, 2018. Read: Ch. 1–3: bioinformatic analysis, what microbiome data are, and an overview of their statistics.
  7. Xia Y, Sun J. Bioinformatic and Statistical Analysis of Microbiome Data: From Raw Sequences to Advanced Modeling with QIIME 2 and R. Springer, 2023.
  8. Ismail HD. Bioinformatics: A Practical Guide to Next Generation Sequencing Data Analysis. CRC Press, 2023.
  9. Low LWY, Tammi MT (eds.). Practical Bioinformatics for Beginners: From Raw Sequence Analysis to Machine Learning Applications. World Scientific, 2023.
  10. Efron B, Hastie T. Computer Age Statistical Inference: Algorithms, Evidence, and Data Science. Cambridge University Press, 2016.
  11. James G, Witten D, Hastie T, Tibshirani R. An Introduction to Statistical Learning with Applications in R, 2nd edition. Springer, 2021.
Part I · Foundations

Study design, controls and metadata

The most important decisions in a microbiome study are made before any DNA is extracted. Design determines which questions can be answered at all; controls determine whether the answer can be trusted.

Replication and the unit of analysis

The unit of replication is the thing that was independently assigned to a treatment or sampled from a population: a person, an animal, a pen, a field plot. Several samples from the same unit (repeated time points, technical replicates, several gut sections) are not independent and must be modelled as such, usually with a random effect or by averaging. Treating them as independent inflates the apparent sample size and produces false positives, a problem known as pseudoreplication.

Microbial communities vary a lot between individuals, so effects that are clear within an individual can be small compared with the differences between individuals. Longitudinal designs, where each subject is its own control, are therefore much more powerful than cross-sectional ones for many questions. Animal studies add a well-known complication: animals housed together share microbes through coprophagy and the environment, so the cage, not the animal, is often the true unit.

Confounders that matter in practice

Several factors are known to shift community composition strongly enough to create spurious associations: antibiotics and other medication (proton-pump inhibitors, metformin, laxatives), diet in the days before sampling, stool consistency and transit time, age, and host genotype in animal models. Technical factors matter just as much: the extraction kit, the sequencing run, the person who processed the samples, and the time a sample spent at room temperature. The safest approach is to randomise samples across extraction batches and sequencing runs, so that technical batches cannot line up with biological groups, and to record every one of these variables.

Negative and positive controls

Reagents and laboratory surfaces carry their own low-level microbial DNA. In samples with high biomass, such as stool, this "kitome" is swamped by real signal. In low-biomass samples, such as skin swabs, tissue biopsies, blood, milk or air, contaminants can make up a large share of the reads and have produced headline findings that later proved to be artefacts (Salter et al., 2014; Eisenhofer et al., 2019). Every extraction batch should include an extraction blank (no sample) and every PCR plate a no-template control. These controls are sequenced like real samples and used later to identify contaminants (chapter 6).

Positive controls are mock communities: defined mixtures of known strains or of their DNA, such as the ZymoBIOMICS standards. Sequencing a mock community shows how far your workflow distorts the true composition. Extraction bias against Gram-positive cells, primer mismatches and GC bias all show up as a gap between expected and observed abundances. A mock community is also the best way to compare two pipelines on your own data.

Relative and absolute abundance

Sequencing returns a sample of reads whose total depends on the library, not on how many microbes were in the sample. Counts are therefore relative: if one taxon doubles, every other taxon appears to shrink. Several methods recover absolute numbers. Flow cytometry measures total cell load per gram (Vandeputte et al., 2017), qPCR of the 16S gene measures total gene copies, and spike-ins add a known number of foreign cells or synthetic DNA before extraction. If the scientific question is about absolute change (did the antibiotic reduce the total bacterial load?), plan for one of these from the start.

Metadata

A sample without metadata is almost worthless. Record subject, time point, treatment, the confounders above, extraction batch, sequencing run and any deviation from protocol, in a single tidy table with one row per sample. Use controlled vocabularies, units in the column names and no merged cells. Standards such as MIxS from the Genomic Standards Consortium describe the minimum information expected when data are deposited in public archives, and following them from the start saves work at publication time (Wilkinson et al., 2016).

Check your understanding

You sequence one stool sample from each of 3 patients 10 times. How many independent observations do you have?

Three. Technical replicates measure the laboratory noise, not the variation between people.

Why do extraction blanks matter most for low-biomass samples?

Kit and laboratory contaminants make up a larger share of the reads when there is little real DNA; without blanks they look like genuine taxa.

Part I · Foundations

Sequencing technologies and read quality

Short reads are cheap and accurate; long reads span repeats and reveal structure. Most choices downstream follow from which kind of read you have and from how good it is.

Short reads

Illumina instruments produce reads of 100 to 300 bases, often as pairs from the two ends of a fragment. Per-base accuracy is high (typically better than 99.9 percent) and errors are mostly substitutions, which rise towards the end of each read. Short reads are the standard for amplicon sequencing, for shotgun metagenomic profiling and for polishing bacterial genomes. Their weakness is length: any repeat longer than a read or a fragment, such as rRNA operons, insertion sequences and duplicated genes, cannot be resolved, so short-read genome assemblies usually break into tens to hundreds of contigs.

Long reads

Oxford Nanopore (ONT) reads DNA as it passes through a protein pore and can produce reads from a few kilobases to more than a hundred kilobases. Older chemistries had error rates of several percent, dominated by insertions and deletions in homopolymers. The R10.4.1 pores with recent basecallers reach modal accuracies above 99 percent, enough to produce near-finished bacterial genomes without short-read polishing (Sereika et al., 2022). PacBio HiFi reads are 10 to 25 kb long with accuracy above 99.9 percent, from circular consensus sequencing of the same molecule. Older PacBio CLR reads are longer but much noisier. A long read that spans every repeat in a genome allows assembly into one circular chromosome plus plasmids.

Quality scores

Each base comes with a Phred quality score Q, where the probability that the base is wrong is

P(error) = 10−Q/10

so Q20 means a 1 in 100 chance of error and Q30 a 1 in 1000 chance. Quality plots per position show where reads degrade; read-length histograms show whether long-read libraries contain the long molecules you paid for. Simple summary tools such as  report read counts, total bases, N50 and mean quality per file (Shen et al., 2016);  does the same with plots tailored to long reads (De Coster et al., 2018).

Ismail works through FASTQ encodings, quality plots and trimming in detail, and Low & Tammi give a hands-on version of the same checks (Ismail, Bioinformatics: A Practical Guide to NGS Data Analysis, ch. 1; Low & Tammi, Practical Bioinformatics for Beginners, ch. 3).

What read QC should do

  • Remove adapters and primers. Adapter read-through happens whenever a fragment is shorter than the read. In amplicon data the primer sequences must also be removed because they do not represent the template.
  • Trim or filter by quality. For short reads, trim low-quality tails (, ). For long reads, filter whole reads below a mean quality or length threshold rather than trimming.
  • Remove host reads in shotgun samples from host-associated sites by mapping to the host genome, both to save compute and for privacy in human studies.
  • Check depth. Samples far below the run median are often failed libraries.

How much sequencing?

For 16S amplicons, 10,000 to 50,000 reads per sample usually captures the dominant community; rare taxa need more. For shotgun profiling of gut samples, a few million read pairs per sample are enough for species-level taxonomic profiles, while functional profiling and genome reconstruction from metagenomes benefit from 10 to 20 million or more. For isolate genomes, 50 to 100-fold short-read coverage or 30 to 50-fold long-read coverage is typical; very high long-read coverage can be subsampled to the longest, best reads.

Check your understanding

A base has Q = 30. What is the chance it is wrong?

10−30/10 = 0.001, one error in a thousand bases. Q20 is one in a hundred.

Why are Nanopore errors harder to remove than Illumina errors?

Illumina errors are mostly substitutions near read ends, easy to trim; Nanopore errors include insertions and deletions, often in homopolymers, spread along the read.

Core skills · Sequence analysis

Aligning sequences: scoring, dynamic programming and 

Almost every sequence question starts with a comparison: is this gene like one we know, where does this read belong, how far apart are these two strains? An alignment is the answer in its most explicit form, and  is how we ask it of millions of sequences at once.

Homology and similarity are not the same word

Two sequences are homologous if they descend from a common ancestor. That is a yes-or-no statement about history: a pair is homologous or it is not, and "70 % homologous" is a category error. What we can measure is similarity, the fraction of aligned positions that are identical or chemically alike. Similarity is evidence for homology, never the definition of it. Pearson puts the practical consequence plainly: statistically significant similarity lets us infer homology, and absence of significant similarity does not prove its absence (Pearson, 2013).

Homologs come in two kinds that matter for naming genes. Orthologs diverged when species diverged, so they are "the same gene" in two organisms. Paralogs diverged when a gene was duplicated inside one lineage, and often take on different jobs. Calling a paralog by its relative's name is one of the commonest sources of wrong annotation, which chapter A3 returns to.

What an alignment is

An alignment writes two sequences in rows so that positions believed to share an ancestor sit in the same column. A column holds a match, a mismatch or, when one sequence has an inserted or deleted stretch, a gap (a dash). Many alignments are possible for the same pair, so we need a rule to say which is best: a score that adds a reward for each match, a penalty for each mismatch and a penalty for each gap, and we keep the alignment with the highest total.

Scoring DNA

DNA scoring is simple because there are only four letters: a positive score for a match and a negative one for a mismatch. The ratio sets what the alignment is tuned to find. 's megablast uses +1 and −2, which rewards long, nearly identical matches; the default uses +2 and −3, which tolerates somewhat more divergence (the defaults of  +; Camacho et al., 2009).

Scoring proteins: substitution matrices

For proteins, not every mismatch is equally bad. Replacing leucine with isoleucine, two similar hydrophobic residues, happens often in evolution and costs little; replacing tryptophan with alanine happens rarely and costs a lot. A substitution matrix stores one score for every pair of the 20 amino acids as a log-odds ratio: how often the pair is seen in real alignments of related proteins, divided by how often chance alone would put them together. The two families you will meet are PAM, built from closely related proteins and extrapolated to greater distances (Dayhoff et al., 1978), and BLOSUM, built directly from conserved blocks of real alignments (Henikoff & Henikoff, 1992). The number in a BLOSUM name is the identity at which sequences in the blocks were merged: BLOSUM62 comes from blocks clustered at 62 %, so it suits moderately related proteins; BLOSUM45 suits more distant ones and BLOSUM80 closer ones. BLOSUM62 is the default of  for proteins.

PairBLOSUM62 scoreWhy
W–W+11tryptophan is rare, so matching it is strong evidence
C–C+9cysteines often form structural disulfide bonds
A–A+4alanine is common, so matching it says less
I–V+3both small, branched and hydrophobic
D–E+2both acidic and negatively charged
A–W−3small against large and aromatic: rarely tolerated

Scoring gaps

An insertion or deletion is usually one event that removes or adds several residues in one go, so a gap of length 5 is much less than five times as unlikely as a gap of length 1. Affine gap penalties capture this with a large cost to open a gap and a small cost for each further position (Gotoh, 1982). For protein searches  charges 11 to open and 1 to extend with BLOSUM62. The gap penalties matter as much as the matrix: too cheap and the program stitches unrelated fragments together, too dear and it cannot cross a real insertion.

Dynamic programming: the idea behind every aligner

There are astronomically many ways to place gaps, yet the best one can be found without trying them all, because the best alignment of two prefixes is built from the best alignments of shorter prefixes. Fill a table whose cell (i, j) holds the best score for the first i letters of one sequence against the first j of the other. Each cell is the best of three moves: diagonal (align the two letters, add the match or mismatch score), down (a gap in the second sequence, add the gap penalty) or right (a gap in the first). When the table is full, the bottom-right cell is the optimal score and following the choices back to the top-left gives the alignment itself. This is the Needleman–Wunsch algorithm for global alignment, which aligns both sequences from end to end (Needleman & Wunsch, 1970).

Global alignment is right when the two sequences are expected to be homologous along their whole length, such as two versions of one gene. When only a region is shared, for instance a protein domain inside two different proteins, forcing the ends to align drags the score down. Local alignment, the Smith–Waterman algorithm, changes two rules: no cell may fall below zero, which lets an alignment start anywhere, and the answer is the highest cell anywhere in the table, not the corner (Smith & Waterman, 1981). Tracing back from that cell until a zero is reached recovers the best-matching region.

Dynamic programming is exact but slow: the table has m × n cells, which is trivial for two genes of a few kilobases and impossible for two human chromosomes, where aligning chromosome 1 with itself would need about 6 × 1016 cells. Everything in the next sections is a way to avoid filling the whole table.

Is the alignment real? Bit scores and E-values

A raw alignment score depends on the matrix and gap penalties, so a score of 120 means nothing on its own. Karlin and Altschul showed that, for local alignments without gaps, the scores of unrelated sequences follow a known distribution, so the chance of seeing a score at least this high can be calculated (Karlin & Altschul, 1990). Two numbers come out of that theory.

  • The bit score S′ rescales the raw score so that it can be compared across matrices and searches. Each extra bit halves the chance of a random hit.
  • The E-value is the number of hits with a score at least this good that you would expect by chance in a search of this size: E ≈ m · n · 2−S′, where m is the query length and n the total length of the database.

Remember what the E-value measures: how surprising the score is under a random model of sequences. Real proteins are not random. Low-complexity stretches (poly-Q, collagen repeats) and unusual amino-acid composition produce high scores between unrelated proteins, which is why  masks low-complexity regions by default and applies a composition-based adjustment to protein scores (Schäffer et al., 2001). A significant E-value is evidence of homology, but it is evidence about the score, not a verdict on the biology.

: finding similar sequences fast

 avoids the table by looking only where similarity is likely. It cuts the query into short words (11 letters for , 28 for megablast, 3 amino acids for protein searches), finds the places in the database where a word matches exactly or, for proteins, scores above a threshold, and tries to extend each such seed into a longer alignment, stopping when the score drops too far. It is a heuristic: it can miss a true alignment that has no good seed, in return for being thousands of times faster than Smith–Waterman (Altschul et al., 1990). Gapped  requires two nearby seeds on the same diagonal before it extends, which cut the run time again, and PSI- repeats the search with a profile built from the hits of the previous round, which finds more distant relatives (Altschul et al., 1997). The modern command-line program is + (Camacho et al., 2009).

ProgramQuery → databaseUse it to…
 / megablastnucleotide → nucleotidefind the same or a very close sequence; megablast for near-identical hits, dc-megablast or for more divergent ones
protein → proteinfind homologous proteins, including distant ones
nucleotide, translated in 6 frames → proteinannotate a gene or read of unknown frame
protein → nucleotide, translated in 6 framesfind where a known protein hides in an unannotated genome
tblastxtranslated → translatedcompare coding sequences when neither has been annotated (slow)

Searching at the protein level is more sensitive for divergent sequences than searching DNA: the 20-letter alphabet carries more information per position, and the genetic code is degenerate, so third-codon positions drift faster than the amino acids they code for. Between distantly related species a DNA search often finds nothing where a protein search finds a clear homolog.

Reading a hit table

The tabular format (-outfmt 6) gives one row per aligned region. Read the columns together, never one alone.

ColumnMeaningCommon trap
pidentidentical positions divided by alignment length100 % over 20 residues of a 600-residue protein is a hit to a motif, not to the gene
length, qcovsaligned length; share of the query covereda high identity on low coverage is a domain match
evalueexpected random hits at this score or betterdepends on database size: do not compare across databases
bitscorescore on a common scaleuse it, not the E-value, to rank hits within one search

For transferring a function from a hit to your protein, identity and coverage both matter. Tian and Skolnick found that enzyme function is usually conserved above about 40 % identity but is increasingly unreliable below it (Tian & Skolnick, 2003). Between roughly 20 and 35 % identity over a long alignment lies what Rost called the twilight zone, where identity alone cannot separate homologs from chance and a profile or structure is needed (Rost, 1999). These are starting points, not laws: a catalytic residue lost to a single substitution can silence an enzyme at 90 % identity.

When  is the wrong tool

  • Millions of queries. Metagenomic reads and whole proteomes are searched with faster protein aligners.  reaches -like sensitivity in its sensitive modes at a small fraction of the time (Buchfink et al., 2015; Buchfink et al., 2021), and  is faster again and can cluster sequences (Steinegger & Söding, 2017).
  • Mapping reads to a reference genome. This needs a read mapper, not a database search:  for long reads and genome-to-genome alignment (Li, 2018), and  for short reads, both covered with variant calling in chapter 21 (Variant calling and comparative genomics).
  • Distant relationships. When pairwise similarity runs out, use profiles: PSI-, or profile hidden Markov models such as  domains searched with . They are the subject of chapter A3.

For a second explanation with exercises, read Baxevanis, Bader and Wishart, ch. 3, on pairwise similarity with  and FASTA (Bioinformatics, 4th ed.); Christensen, ch. 4, which links pairwise alignment, multiple alignment and  in a microbiology setting (Introduction to Bioinformatics in Microbiology); and Pevzner and Shamir, ch. 4, for dynamic programming as a general idea (Bioinformatics for Biologists).

Check your understanding

Two proteins are 85 % identical over 40 residues of a 500-residue sequence. Are they homologous?

Possibly a shared domain, but the evidence is about 8 % of the sequence. Coverage matters as much as identity: this is a domain-level hit, and the rest of the proteins may be unrelated.

Why does Smith–Waterman never let a cell fall below zero?

A zero lets an alignment start afresh anywhere. A poor region cannot drag the score of a good one below zero, so the best-matching region is found regardless of what lies on either side.

A hit has bit score 40 against a database of 108 residues. The same query and hit is later searched against a database ten times larger. What happens to the bit score and to the E-value?

The bit score is unchanged; it depends on the alignment, not the database. The E-value rises about tenfold, because there are ten times as many chances for a random hit.

Which program finds where a known protein lies in a newly sequenced, unannotated genome?

: the protein is the query and the genome is translated in all six frames.

Key references

  1. Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ. Basic local alignment search tool. Journal of Molecular Biology 215, 403–410 (1990).
  2. Altschul SF, Madden TL, Schäffer AA, et al. Gapped  and PSI-: a new generation of protein database search programs. Nucleic Acids Research 25, 3389–3402 (1997).
  3. Buchfink B, Xie C, Huson DH. Fast and sensitive protein alignment using . Nature Methods 12, 59–60 (2015).
  4. Buchfink B, Reuter K, Drost H-G. Sensitive protein alignments at tree-of-life scale using . Nature Methods 18, 366–368 (2021).
  5. Camacho C, Coulouris G, Avagyan V, et al. +: architecture and applications. BMC Bioinformatics 10, 421 (2009).
  6. Dayhoff MO, Schwartz RM, Orcutt BC. A model of evolutionary change in proteins. In: Atlas of Protein Sequence and Structure, vol. 5, suppl. 3, 345–352. National Biomedical Research Foundation, 1978.
  7. Gotoh O. An improved algorithm for matching biological sequences. Journal of Molecular Biology 162, 705–708 (1982).
  8. Henikoff S, Henikoff JG. Amino acid substitution matrices from protein blocks. PNAS 89, 10915–10919 (1992).
  9. Karlin S, Altschul SF. Methods for assessing the statistical significance of molecular sequence features by using general scoring schemes. PNAS 87, 2264–2268 (1990).
  10. Li H. : pairwise alignment for nucleotide sequences. Bioinformatics 34, 3094–3100 (2018).
  11. Needleman SB, Wunsch CD. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology 48, 443–453 (1970).
  12. Pearson WR. An introduction to sequence similarity (“homology”) searching. Current Protocols in Bioinformatics 42, 3.1.1–3.1.8 (2013).
  13. Rost B. Twilight zone of protein sequence alignments. Protein Engineering 12, 85–94 (1999).
  14. Schäffer AA, Aravind L, Madden TL, et al. Improving the accuracy of PSI- protein database searches with composition-based statistics and other refinements. Nucleic Acids Research 29, 2994–3005 (2001).
  15. Shah N, Nute MG, Warnow T, Pop M. Misunderstood parameter of   impacts the correctness of bioinformatics workflows. Bioinformatics 35, 1613–1614 (2019).
  16. Smith TF, Waterman MS. Identification of common molecular subsequences. Journal of Molecular Biology 147, 195–197 (1981).
  17. Steinegger M, Söding J.  enables sensitive protein sequence searching for the analysis of massive data sets. Nature Biotechnology 35, 1026–1028 (2017).
  18. Tian W, Skolnick J. How well is enzyme function conserved as a function of pairwise sequence identity? Journal of Molecular Biology 333, 863–882 (2003).
  19. Baxevanis AD, Bader GD, Wishart DS (eds.). Bioinformatics, 4th edition. Wiley, 2020. Read: Ch. 3, assessing pairwise sequence similarity with  and FASTA.
  20. Christensen H (ed.). Introduction to Bioinformatics in Microbiology. Learning Materials in Biosciences, Springer, 2018. Read: Ch. 4, pairwise alignment, multiple alignment and .
  21. Pevzner P, Shamir R (eds.). Bioinformatics for Biologists. Cambridge University Press, 2011. Read: Ch. 4, dynamic programming as one key for many biological problems.
Core skills · Sequence analysis

Multiple alignment and phylogenetic trees

A pairwise alignment compares two sequences. A multiple alignment lines up many at once, and it is the raw material for almost everything that follows: conserved sites, profile models, and the family trees that tell you how genes, strains and species are related.

From two sequences to many

In a multiple sequence alignment (MSA) each column is a claim: the residues in it descend from one residue of the common ancestor. A well-built MSA shows which positions are conserved, which vary, and where insertions and deletions have happened. Conserved columns point to structure and function; variable columns carry the signal that separates strains and species. The same alignment feeds profile models (chapter A3) and phylogenetic trees (below), so a mistake here is inherited by everything built on it.

Building the alignment

Exact multiple alignment by dynamic programming grows exponentially with the number of sequences and is not used in practice. Almost every method is progressive: align the two most similar sequences first, then add the next most similar sequence or group, following a guide tree from the closest pairs to the most distant ones. The weakness is built in. A gap placed in an early alignment is never revisited, so an early error spreads to every later step, a problem often summarised as "once a gap, always a gap". Better methods soften this with iterative refinement, which repeatedly splits the alignment and realigns the parts, as in MUSCLE (Edgar, 2004), or with a consistency score that prefers alignments supported by many pairwise comparisons.

ProgramStrengthUse it when
fast, with settings from quick to accurate (Katoh & Standley, 2013)the default choice; --auto picks a strategy for the data size
MUSCLEaccurate; version 5 gives an ensemble of alignments to test how much a result depends on the alignment (Edgar, 2022)moderate numbers of sequences; checking robustness
Clustal Omegascales to many thousands of protein sequences (Sievers et al., 2011)very large protein sets

Align proteins, then go back to DNA. A coding sequence aligned as DNA treats a deleted codon as a stretch of three nucleotides that can be misplaced by one position, and the codon structure is lost. The reliable order is to translate, align the proteins, and map the gaps back onto the nucleotides in whole codons, which preserves the reading frame and is the correct input for tests of selection. PAL2NAL does this step (Suyama et al., 2006).

Checking and trimming

Look at the alignment before using it. Scroll through it in a viewer: long gap-rich blocks, a single sequence that does not fit anywhere, or a large block of mismatches at one end usually mean a wrong sequence (a fragment, a contaminant, a different gene) or a poor alignment. Remove or fix those first. Automatic trimming of unreliable columns, for example with trimAl (Capella-Gutiérrez et al., 2009), is common practice, but it is not free: a benchmark of trimming tools found that they frequently worsened single-gene trees, because the columns removed carry signal as well as noise (Tan et al., 2015). Trim lightly and record what you did.

What a tree is

A phylogenetic tree is a diagram of descent. Tips are the sequences you sampled. Each internal node is a common ancestor of everything below it and each branch is a lineage. A clade is a node and all its descendants; a clade is monophyletic when it contains an ancestor and all of its descendants. Three things are easy to get wrong:

  • The order of tips means nothing. Any node can be rotated like the arm of a mobile without changing the tree. Do not read "closer on the page" as "closer relatives"; follow the branches back to the node they share.
  • Branch lengths are optional. A cladogram shows only branching order. A phylogram draws branch length proportional to the amount of change, usually in substitutions per site.
  • A tree is rooted or not. An unrooted tree shows relationships without a direction in time. A root adds that direction and is placed with an outgroup, a lineage known to be outside the group of interest, or by assumptions such as a constant rate (the midpoint).

Ways of building a tree

ApproachIdeaTrade-off
Distance (neighbor-joining)convert the alignment to pairwise distances and join the closest pairs, correcting for unequal rates (Saitou & Nei, 1987)very fast, one tree, loses information when sequences are very divergent
Maximum parsimonychoose the tree that needs the fewest changesno model of evolution; can mislead when rates differ between lineages
Maximum likelihoodchoose the tree and branch lengths that make the data most probable under an explicit model (Felsenstein, 1981)the standard for most work; slower, needs a model
Bayesian inferencesample trees in proportion to their posterior probability (Ronquist et al., 2012)gives a probability for each clade and handles dating; slow, needs care that runs have converged

For most projects use maximum likelihood.  (Minh et al., 2020) and -NG (Kozlov et al., 2019) are the usual programs;  is a much faster approximate method suited to very large alignments, where it trades some accuracy for speed (Price et al., 2010).

Why raw differences understate distance

Count the share p of positions at which two sequences differ. It is tempting to use p as the distance, but a position can change twice (A to G and back, or A to G to T), and the second change may be invisible. The longer the time, the more changes are hidden. A substitution model corrects for this. The simplest, Jukes–Cantor, assumes every change is equally likely and gives the corrected distance d = −¾ ln(1 − 4p/3).

Real models add the facts of real evolution: transitions (A↔G, C↔T) are more frequent than transversions, base frequencies are not equal, and some sites change faster than others, usually modelled by a gamma distribution of rates (written +G). The general time-reversible model with gamma rates (GTR+G) is a common DNA model. Rather than guessing, let the program choose: ModelFinder, built into , compares models by an information criterion and reports the best (Kalyaanamoorthy et al., 2017).

How much to trust a branch

The bootstrap resamples the columns of the alignment with replacement, builds a tree from each resample, and reports for every branch the percentage of resampled trees that contain it (Felsenstein, 1985). A branch with 87 means that 87 of 100 resampled alignments gave that grouping. It measures how consistently the data support the grouping, not the probability that the grouping is true: a consistent error, such as a wrong model or a contaminated sequence, can be supported at 100. The standard bootstrap needs many full tree searches, so  offers an ultrafast version, UFBoot, whose values are read differently, with 95 or more usually taken as well supported (Hoang et al., 2018), and the SH-aLRT test, where 80 or more is the common mark (Guindon et al., 2010). Report which kind of support a figure shows, and treat branches below these marks as unresolved, not as wrong.

Gene trees and species trees

A tree built from one gene is a gene tree. It need not match the tree of the species. Paralogs make the gene tree describe duplications as well as speciations, so trees of gene families must be read with orthology in mind. Horizontal gene transfer moves genes between lineages, which is common in bacteria. Incomplete lineage sorting makes gene trees differ from the species tree in closely related species. The usual remedies are to build the species tree from many single-copy genes, either by joining their alignments or by reconciling their individual trees, and to check how well individual genes agree. The 16S rRNA gene is a convenient single marker for bacteria, but a tree from one marker is a tree of that marker.

Baxevanis, Bader and Wishart, ch. 8, covers multiple alignment and how to measure its quality (Bioinformatics, 4th ed.). Claverie and Notredame take the practical route, from building an alignment to building a tree, in ch. 9 and ch. 13 (Bioinformatics For Dummies). For the methods behind tree estimation read Warnow (Computational Phylogenetics), and for doing it in R, Paradis (Analysis of Phylogenetics and Evolution with R).

Check your understanding

Two taxa sit next to each other at the bottom of a printed tree. Are they close relatives?

Not necessarily. The order of tips is arbitrary because nodes can be rotated. Close relatives are those that share a recent common ancestor: follow the branches up to the node they share.

Two DNA sequences differ at half their positions. Is the evolutionary distance 0.5 substitutions per site?

No. Some positions have changed more than once and the extra changes are invisible. Under Jukes–Cantor the corrected distance is about 0.82.

A branch has bootstrap support 100. Does that prove the grouping is correct?

No. It shows that the data consistently support it under the model used. A wrong model, a contaminated sequence or long-branch attraction can be supported at 100.

You want a tree of a gene family that has been duplicated in several species. What must you keep in mind?

The gene tree contains duplication events as well as speciation events, so it does not equal the species tree. Separate orthologs from paralogs before reading it as a species phylogeny.

Key references

  1. Capella-Gutiérrez S, Silla-Martínez JM, Gabaldón T. trimAl: a tool for automated alignment trimming in large-scale phylogenetic analyses. Bioinformatics 25, 1972–1973 (2009).
  2. Edgar RC. MUSCLE: multiple sequence alignment with high accuracy and high throughput. Nucleic Acids Research 32, 1792–1797 (2004).
  3. Edgar RC. Muscle5: high-accuracy alignment ensembles enable unbiased assessments of sequence homology and phylogeny. Nature Communications 13, 6968 (2022).
  4. Felsenstein J. Cases in which parsimony or compatibility methods will be positively misleading. Systematic Zoology 27, 401–410 (1978).
  5. Felsenstein J. Evolutionary trees from DNA sequences: a maximum likelihood approach. Journal of Molecular Evolution 17, 368–376 (1981).
  6. Felsenstein J. Confidence limits on phylogenies: an approach using the bootstrap. Evolution 39, 783–791 (1985).
  7. Guindon S, Dufayard J-F, Lefort V, et al. New algorithms and methods to estimate maximum-likelihood phylogenies: assessing the performance of PhyML 3.0. Systematic Biology 59, 307–321 (2010).
  8. Hoang DT, Chernomor O, von Haeseler A, Minh BQ, Vinh LS. UFBoot2: improving the ultrafast bootstrap approximation. Molecular Biology and Evolution 35, 518–522 (2018).
  9. Kalyaanamoorthy S, Minh BQ, Wong TKF, von Haeseler A, Jermiin LS. ModelFinder: fast model selection for accurate phylogenetic estimates. Nature Methods 14, 587–589 (2017).
  10. Katoh K, Standley DM.  multiple sequence alignment software version 7: improvements in performance and usability. Molecular Biology and Evolution 30, 772–780 (2013).
  11. Kozlov AM, Darriba D, Flouri T, Morel B, Stamatakis A. -NG: a fast, scalable and user-friendly tool for maximum likelihood phylogenetic inference. Bioinformatics 35, 4453–4455 (2019).
  12. Minh BQ, Schmidt HA, Chernomor O, et al.  2: new models and efficient methods for phylogenetic inference in the genomic era. Molecular Biology and Evolution 37, 1530–1534 (2020).
  13. Price MN, Dehal PS, Arkin AP.  2: approximately maximum-likelihood trees for large alignments. PLoS ONE 5, e9490 (2010).
  14. Ronquist F, Teslenko M, van der Mark P, et al. MrBayes 3.2: efficient Bayesian phylogenetic inference and model choice across a large model space. Systematic Biology 61, 539–542 (2012).
  15. Saitou N, Nei M. The neighbor-joining method: a new method for reconstructing phylogenetic trees. Molecular Biology and Evolution 4, 406–425 (1987).
  16. Sievers F, Wilm A, Dineen D, et al. Fast, scalable generation of high-quality protein multiple sequence alignments using Clustal Omega. Molecular Systems Biology 7, 539 (2011).
  17. Suyama M, Torrents D, Bork P. PAL2NAL: robust conversion of protein sequence alignments into the corresponding codon alignments. Nucleic Acids Research 34, W609–W612 (2006).
  18. Tan G, Muffato M, Ledergerber C, et al. Current methods for automated filtering of multiple sequence alignments frequently worsen single-gene phylogenetic inference. Systematic Biology 64, 778–791 (2015).
  19. Baxevanis AD, Bader GD, Wishart DS (eds.). Bioinformatics, 4th edition. Wiley, 2020. Read: Ch. 8, multiple sequence alignments and measuring their quality.
  20. Claverie J-M, Notredame C. Bioinformatics For Dummies, 2nd edition. Wiley, 2007. Read: Ch. 9, building a multiple sequence alignment; ch. 13, building phylogenetic trees.
  21. Warnow T. Computational Phylogenetics: An Introduction to Designing Methods for Phylogeny Estimation. Cambridge University Press, 2017. Read: the chapters on distance-based and maximum-likelihood estimation, and on bootstrapping.
  22. Paradis E. Analysis of Phylogenetics and Evolution with R, 2nd edition. Use R, Springer, 2012. Read: the chapters on reading, plotting and estimating trees in R.
Core skills · Sequence analysis

Genes, genomes and annotation: what the evidence can and cannot say

A genome sequence is a string of letters. Annotation turns it into a list of genes with names and functions, and almost every later analysis depends on that list. It is also the step in which the most confident-looking errors are made, so it pays to know what each kind of evidence is worth.

The layout of a genome

Bacterial and archaeal genomes are compact: the great majority of the sequence codes for protein or for a structural RNA, and genes lie close together, often on both strands. Genes with related functions are frequently transcribed together as an operon, one messenger RNA carrying several genes, so the order of genes on the chromosome is itself information. A bacterial gene is a single uninterrupted reading frame, with a ribosome-binding site, the Shine–Dalgarno sequence, a few bases upstream of the start codon.

Eukaryotic genomes are loose. In the human genome, protein-coding exons are only about 1–2 % of the sequence. Genes are split into exons separated by introns, which are removed from the transcript by splicing; introns nearly always begin with GT and end with AG. Genes also have untranslated regions (UTRs) at either end, and one gene can give several transcripts by alternative splicing. Fungi lie in between: compact genomes, but with introns, usually few and short in yeasts and more numerous in filamentous species. Chapter 33 treats fungal gene prediction in practice.

The genetic code and the reading frame

A codon is three nucleotides. Of the 64 codons, 61 specify an amino acid and three (TAA, TAG, TGA) mean stop; ATG codes for methionine and is the usual start. A sequence can be read in three frames on each strand, six in all, and a gene is read in only one. An open reading frame (ORF) is the stretch from a start codon to the next stop in one frame. Not every ORF is a gene: stop codons turn up by chance, so short ORFs are everywhere.

Organisms do not all use the same code. Bacteria and archaea use the standard amino-acid assignments but also accept GTG and TTG as starts ( translation table 11). Mycoplasma reads TGA as tryptophan instead of stop (table 4), and mitochondria have their own tables. Annotating with the wrong table truncates every gene that contains the reassigned codon.

How genes are found

Prokaryotes. Gene finders score candidate ORFs on how much they look like the genome's own coding sequence (codon usage and its pattern across the three codon positions), on the presence of a ribosome-binding site, and on the start codon used, then choose the set of genes that fits together best along the chromosome.  learns all of these from the genome itself without being given a training set (Hyatt et al., 2010). Start positions are the hardest part: the end of a gene is fixed by its stop codon, but several in-frame start codons may be possible, and a wrong choice shortens or lengthens the protein at the N-terminus.

Eukaryotes. The problem is to find exons and join them correctly across introns. Ab initio programs such as  use a hidden Markov model of exon, intron and intergenic sequence (Stanke et al., 2006). They become much more accurate when given evidence: RNA-seq reads aligned to the genome and proteins from related species. Pipelines such as  combine the two and train the model on the genome being annotated (Brůna et al., 2021).

Non-coding genes. Transfer RNAs are found with  (Chan et al., 2021). Other structural RNAs, from ribosomal RNA to riboswitches, are found by covariance models of their conserved secondary structure, built into  and catalogued in  (Nawrocki & Eddy, 2013; Kalvari et al., 2021).

Protein families and profile models

 compares a sequence with another sequence. When two proteins are too distant for that, a profile does better, because it summarises a whole family. A profile hidden Markov model (profile HMM) is built from a multiple alignment of a family (chapter A2). For each column it records how likely each amino acid is, so a conserved tryptophan counts for much and a variable position for little, and it has states for insertions and deletions between columns. A new sequence is scored against the model, and the score says how well it fits the family, not any one member.  searches sequences against such models quickly (Eddy, 2011).

The  database holds profile HMMs for thousands of protein families and domains, each with a curated score cut-off that sets where a hit is counted (Mistry et al., 2021), and  combines  with many other signature databases into one report per protein (Paysan-Lafosse et al., 2023). A domain hit tells you which part of a protein resembles a known module, and not what the whole protein does: a protein with a kinase domain and an unknown N-terminal region is a kinase of unknown specificity.

From a name to a function: ontologies and evidence

Free text such as "putative transporter" is hard to compare across genomes. The Gene Ontology (GO) gives controlled terms in three branches: molecular function (what the product does), biological process (what larger activity it takes part in) and cellular component (where it works) (Ashburner et al., 2000; Gene Ontology Consortium et al., 2023). Each assignment carries an evidence code saying how it was reached. Codes beginning with experimental evidence (for instance direct assay) rest on an observation about that protein; the code IEA, inferred from electronic annotation, is computed with no human review. Most of the annotations in any large database are of the second kind. Pathway resources such as KEGG add the question of which reactions a genome can carry out.

Functional annotation then adds up as a ladder of decreasing certainty:

  1. Experimentally shown for this protein, or an identical one.
  2. Orthology: a clear ortholog with experimental evidence, found by phylogeny or a method such as  (Emms & Kelly, 2019) or  (Cantalapiedra et al., 2021). Orthologs usually share function; paralogs often do not.
  3. Homology to a characterised family or domain: the protein belongs to a family, so it is likely to do the family's general job.
  4. Context: neighbouring genes, operon structure, co-expression.
  5. No evidence: "hypothetical protein". An honest label, and a far better one than a guess.

How errors spread

An annotation that is transferred from one genome to the next is copied, and the copy is not checked. Schnoes and colleagues compared the function assigned in public databases with the experimentally supported function across several enzyme superfamilies and found that misannotation was substantial in a number of superfamilies and that it had spread by transfer between similar sequences (Schnoes et al., 2009). Every new genome annotated by homology to a polluted database inherits the pollution and adds to it.

Annotation pipelines

Bacterial genomes are usually annotated with an integrated pipeline: 's PGAP, used for every genome submitted to  (Tatusova et al., 2016),  (Seemann, 2014), or , which searches a curated and versioned database of protein families and returns identifiers that can be traced back to their source (Schwengers et al., 2021). They differ in the databases they use and in how cautiously they name genes, so record the pipeline and database version with the result. Chapter 19 describes bacterial annotation step by step, and chapters 33 and 34 cover fungal gene prediction and function.

Baxevanis, Bader and Wishart, ch. 5, explains gene prediction methods and the structure of an annotation project (Bioinformatics, 4th ed.). Zvelebil and Baum, ch. 10, treat gene detection and annotation across bacteria and eukaryotes (Understanding Bioinformatics). Keith (ed.), vol. I, has a chapter on genome annotation (ch. 5) and one on working with ontologies (ch. 6) (Bioinformatics, Methods in Molecular Biology).

Check your understanding

A newly assembled bacterial genome has 30 % of its genes flagged as pseudogenes with frameshifts. What do you suspect first?

Sequencing or assembly errors, most likely homopolymer indels from long reads, not 30 % true pseudogenes. Polish the assembly and annotate again.

Why does a rule such as "any ORF over 100 codons is a gene" fail in a genome of 70 % GC?

Stop codons are AT-rich, so in a GC-rich genome they occur less often by chance. About one random stop-free stretch in seven is at least 100 codons long, so long false ORFs appear.

A protein has a  kinase domain. What can you say about its function?

That it probably phosphorylates something. The domain does not tell you the substrate, the pathway or whether the catalytic residues are intact. The rest of the protein and other evidence decide.

What is the difference between an IEA and an experimental GO annotation?

IEA is inferred electronically, with no human review of the protein. An experimental code rests on an observation made on that protein, so it is far more reliable.

Key references

  1. Ashburner M, Ball CA, Blake JA, et al. Gene Ontology: tool for the unification of biology. Nature Genetics 25, 25–29 (2000).
  2. Brůna T, Hoff KJ, Lomsadze A, Stanke M, Borodovsky M. BRAKER2: automatic eukaryotic genome annotation with -EP+ and  supported by a protein database. NAR Genomics and Bioinformatics 3, lqaa108 (2021).
  3. Cantalapiedra CP, Hernández-Plaza A, Letunic I, Bork P, Huerta-Cepas J.  v2: functional annotation, orthology assignments, and domain prediction at the metagenomic scale. Molecular Biology and Evolution 38, 5825–5829 (2021).
  4. Chan PP, Lin BY, Mak AJ, Lowe TM. : improved detection and functional classification of transfer RNA genes. Nucleic Acids Research 49, 9077–9096 (2021).
  5. Eddy SR. Accelerated profile HMM searches. PLoS Computational Biology 7, e1002195 (2011).
  6. Emms DM, Kelly S. : phylogenetic orthology inference for comparative genomics. Genome Biology 20, 238 (2019).
  7. Gene Ontology Consortium, Aleksander SA, Balhoff J, et al. The Gene Ontology knowledgebase in 2023. Genetics 224, iyad031 (2023).
  8. Hyatt D, Chen G-L, LoCascio PF, et al. : prokaryotic gene recognition and translation initiation site identification. BMC Bioinformatics 11, 119 (2010).
  9. Kalvari I, Nawrocki EP, Ontiveros-Palacios N, et al.  14: expanded coverage of metagenomic, viral and microRNA families. Nucleic Acids Research 49, D192–D200 (2021).
  10. Mistry J, Chuguransky S, Williams L, et al. : the protein families database in 2021. Nucleic Acids Research 49, D412–D419 (2021).
  11. Nawrocki EP, Eddy SR. : 100-fold faster RNA homology searches. Bioinformatics 29, 2933–2935 (2013).
  12. Paysan-Lafosse T, Blum M, Chuguransky S, et al.  in 2022. Nucleic Acids Research 51, D418–D427 (2023).
  13. Schnoes AM, Brown SD, Dodevski I, Babbitt PC. Annotation error in public databases: misannotation of molecular function in enzyme superfamilies. PLoS Computational Biology 5, e1000605 (2009).
  14. Schwengers O, Jelonek L, Dieckmann MA, et al. : rapid and standardized annotation of bacterial genomes via alignment-free sequence identification. Microbial Genomics 7, 000685 (2021).
  15. Seemann T. : rapid prokaryotic genome annotation. Bioinformatics 30, 2068–2069 (2014).
  16. Stanke M, Keller O, Gunduz I, et al. : ab initio prediction of alternative transcripts. Nucleic Acids Research 34, W435–W439 (2006).
  17. Tatusova T, DiCuccio M, Badretdin A, et al.  Prokaryotic Genome Annotation Pipeline. Nucleic Acids Research 44, 6614–6624 (2016).
  18. Watson M, Warr A. Errors in long-read assemblies can critically affect protein prediction. Nature Biotechnology 37, 124–126 (2019).
  19. Baxevanis AD, Bader GD, Wishart DS (eds.). Bioinformatics, 4th edition. Wiley, 2020. Read: Ch. 5, genome annotation and gene prediction methods.
  20. Zvelebil M, Baum JO. Understanding Bioinformatics. Garland Science, 2007. Read: Ch. 10, gene detection and genome annotation.
  21. Keith JM (ed.). Bioinformatics, Volume I: Data, Sequence Analysis, and Evolution. Methods in Molecular Biology 452. Humana Press, 2008. Read: Ch. 5, genome annotation; ch. 6, working with ontologies.
Core skills · Programming and data

The command line, files and scripts

Bioinformatics tools run on servers, read files too large to open in a spreadsheet, and are chained into pipelines. The command line is where that work happens. A few dozen commands and the habit of writing down what you ran are enough to do most of it, and to be able to repeat it.

Why the shell

Three reasons make the shell indispensable. The machines that do the work are remote servers with no screen, reached by a terminal. The data are large: a sequencing run is gigabytes of text, and a command such as head shows the start of it at once. And a command is a record. A series of clicks cannot be rerun on next year's data, but a script of commands can, and a reviewer can read it (Wilson et al., 2014).

Files and directories

A file system is a tree of directories. You are always in one, the working directory, and a path can be absolute (starting at the root, /home/ana/project/data/reads.fq.gz) or relative to where you are (data/reads.fq.gz).

CommandWhat it does
pwd, ls -lh, cd dirwhere am I, what is here with sizes, move to dir (cd .. goes up)
mkdir -p a/b, cp, mvmake nested directories, copy, move or rename
rm filedelete a file. There is no recycle bin: it is gone
head -n 4 f, tail -n 4 f, less -S fthe first or last lines, and a pager to scroll a large file (q quits)
wc -l f, zcat f.gz | headcount lines, read a compressed file without unpacking it

Two habits save most accidents. Never put spaces in file names: the shell splits a name at the space, so my reads.fq becomes two files. And keep raw data read-only in its own directory, so that every result can be regenerated from it (Noble, 2009).

Pipes: small tools, joined

Each command reads text from its standard input and writes to its standard output. The pipe | connects the output of one to the input of the next; > writes to a file (and overwrites it), >> appends; errors go to a separate stream, 2>. Every command also returns an exit status: 0 means success and anything else means failure, which scripts can test.

ToolUse
cut -f1,3pick columns of a tab-separated file (-d, for commas)
sort, sort -k2,2nr, uniq -corder lines, order by column 2 numerically and descending, count identical lines (the input must be sorted)
grep -E 'pattern', -c, -vlines that match a pattern, count them, or keep the lines that do not match
awk -F'\t' '…'a small language for columns: sums, means, filters
sed 's/a/b/', trreplace text, translate characters

Sort numbers with -n. Without it, sort compares characters, so 5386, 19282 and 48502 come out as 19282, 48502, 5386, because "1" comes before "5". With sort -n they come out in numeric order.

Reading sequence files from the shell

A FASTA record is a header line starting with > followed by the sequence, so grep -c '>' genomes.fna counts the sequences. A FASTQ record is exactly four lines: the read name (starting with @), the bases, a + line, and a quality string with one character per base. The bases are therefore on every fourth line, starting at the second:

awk 'NR%4==2 {n++; t+=length($0)} END {printf "%d reads, mean length %.2f\n", n, t/n}' reads.fq

On a file of three reads of 10, 12 and 6 bases this prints 3 reads, mean length 9.33. A common shortcut fails: grep -c '^@' reads.fq counts lines that start with @, but @ is also a valid quality character (Phred score 31), so on a file in which one read has such a quality string it returned 4 for 3 reads. Count by position, as above, or use a tool built for it such as  (Shen et al., 2016). A BAM file is binary: view it with  view (Danecek et al., 2021), never with less.

Patterns: regular expressions

A regular expression describes a family of strings. ^ and $ anchor a pattern to the start and end of a line, [0-9] is any digit, {6} repeats the previous item six times, and a dot must be written \. to mean a real dot. grep -E '^NC_[0-9]{6}\.[0-9]+$' keeps lines like NC_001416.1 and rejects AB123 and NC_12.3. Test a pattern on a few lines you know should match and a few that should not, before you trust it on a million.

Scripts that stop when something goes wrong

A script is a file of commands. Run on its own, a script carries on after a failed step and produces output from broken input, which is worse than stopping. Start every script with:

#!/usr/bin/env bash
set -euo pipefail

-e stops the script when a command fails, -u stops it on an unset variable, and pipefail makes a pipeline fail if any command in it fails, not only the last. Without pipefail the line false | true reports success; with it the script stops (tested in bash). An empty variable is the one this does not catch. If d is set to an empty string, rm -rf ${d}/* becomes rm -rf /*. Guard it with ${d:?d is empty}, which stops the script with a message, and always quote variables ("$f") so that file names with spaces survive.

A loop does the same thing to many files: for f in data/*.fastq.gz; do echo "$f"; done. Put one step in one place, parameters at the top and outputs in a directory named for the step, and the script becomes its own documentation (Buffalo, ch. 12).

Heavy jobs: schedulers

A shared server has a few login nodes to connect to and many compute nodes to work on. Do not run assemblies or alignments on the login node; hand them to the scheduler, which queues them and runs each where there are resources. On , a job script starts with resource requests (#SBATCH --cpus-per-task=8, --mem=32G, --time=04:00:00), is submitted with sbatch script.sh, watched with squeue -u $USER and cancelled with scancel JOBID. A job that asks for too little memory is killed when it exceeds it, and one that asks for too much waits longer in the queue, so read the log of a finished job and adjust. Workflow managers such as  write these job scripts for you and resume after a failure (Di Tommaso et al., 2017).

Keep it repeatable

Anyone, including you in a year, should be able to rerun an analysis and get the same result. A few habits give most of it: record every command in a script, not in memory; keep raw data read-only and results disposable; record the versions of the tools; put the code under version control; and write a short README that says what each directory holds (Sandve et al., 2013; Noble, 2009).

Buffalo, Bioinformatics Data Skills, is the best single book for this chapter: ch. 2 on setting up a project, ch. 3 on the Unix shell, ch. 4 on working with remote machines, ch. 7 on Unix data tools and ch. 12 on shell scripting, pipelines and running tasks in parallel. Hendrix, Applied Bioinformatics, ch. 1, introduces sequences, Biopython and GNU/Linux together. The next chapter turns from the shell to Python and R.

Check your understanding

You run sort lengths.txt on 5386, 19282 and 48502, and get 19282 first. Why, and what fixes it?

Plain sort compares characters, and "1" comes before "5". sort -n compares numbers.

A FASTQ file has 400 lines. How many reads, and which lines hold the bases?

100 reads, four lines each. The bases are on lines 2, 6, 10, and so on: NR%4==2.

A script contains rm -rf ${out}/* and out was never filled in. What happens with set -u, and what is safer?

If out is unset, set -u stops the script. If it is set to an empty string it does not, and the command becomes rm -rf /*. Use ${out:?}, which stops on empty as well.

Why start a script with set -o pipefail?

Without it a pipeline reports the status of its last command only, so a failure in the first step is hidden and the script goes on with empty or broken input.

Core skills · Programming and data

Python and R for bioinformatics: tables, sequences and plots

The shell moves files; Python and R analyse what is in them. Python is a general-purpose language with good tools for sequences and for machine learning. R was built for statistics and holds most of the packages for sequencing statistics. A bioinformatician reads both, and the same few ideas, tidy tables, groups and plots, work in each.

Which language when

Python is the glue of bioinformatics: it reads files, calls other programs, and has NumPy for fast arrays (Harris et al., 2020), pandas for tables (McKinney, 2010), Biopython for sequences and formats (Cock et al., 2009) and scikit-learn for machine learning. R is the language of statistics, and the Bioconductor project collects thousands of R packages for genomic data, with shared data structures so that tools for counts, ranges and annotation work together (Huber et al., 2015). ,  and , the packages behind differential expression in the RNA-seq chapters (24 to 29), are Bioconductor packages. You do not have to choose: the habits below are the same in both.

Tidy tables

Most time in an analysis goes into getting data into the right shape. Wickham's tidy data rules are the standard: each variable is a column, each observation is a row, and each kind of observational unit is its own table (Wickham, 2014). A count table with one row per gene and one column per sample is wide: convenient to read, and often the form a tool wants. A table with one row per gene and sample, with columns for gene, sample and count, is long: the form for grouping, plotting and modelling. Knowing which you hold, and how to convert, solves a large share of everyday problems.

Tables in Python and R

The same task in both languages: read a tab-separated file, group, summarise. The file is the real  output from the viral demonstration data (80 genomes, with the length and the end structure of each).

pandas

import pandas as pd
v = pd.read_csv("_virus_summary.tsv", sep="\t")
v.groupby("topology")["length"].agg(["count", "median"])

R

v <- read.delim("_virus_summary.tsv")
aggregate(length ~ topology, v, function(x) c(n = length(x), median = median(x)))

Operations you will use constantly: filter rows (v[v["virus_score"] >= 0.99] in pandas, subset(v, virus_score >= 0.99) in R), select columns, merge two tables on a shared key (pd.merge, merge), group and summarise as above, and reshape between wide and long (melt and pivot in pandas; pivot_longer and pivot_wider in the tidyr package). When you merge, check that the number of rows is what you expect: a key that is not unique on one side silently multiplies rows.

Sequences with Biopython

Biopython reads and writes the standard formats and gives sequences methods. SeqIO.parse("genomes.fna", "fasta") iterates over the records of a FASTA file without loading them all, which matters for large files. A Seq object can be reverse-complemented and translated:

from Bio.Seq import Seq
from Bio.SeqUtils import gc_fraction
s = Seq("ATGGCCATTGTAATGGGCCGCTGAAAGGGTGCCCGATAG")
len(s), gc_fraction(s), s.translate(to_stop=True), s.reverse_complement()

This gives a length of 39, a GC fraction of 56.4 %, the protein MAIVMGR (translation stops at the first stop codon, TGA, because of to_stop=True) and the reverse complement CTATCGGGCACCCTTTCAGCGGCCCATTACAATGGCCAT. The genetic code that translation uses must match the organism: Biopython takes a table argument, and the wrong table gives wrong proteins (chapter A3).

R and Bioconductor

R data live in data frames (tables) and vectors; most operations apply to a whole vector at once, so loops are rarely needed. The tidyverse packages (dplyr for grouping and filtering, ggplot2 for plots) give a consistent grammar for the table operations above. Bioconductor adds shared containers: a SummarizedExperiment holds a matrix of counts together with the table describing the samples and the table describing the genes, so that subsetting one subsets all three together. That removes a whole class of errors in which the counts and the sample table drift out of order.

Four R habits cause silent errors. Factor levels: a grouping variable is stored as a factor, and its first level is the reference that a model compares everything with, which is alphabetical unless you set it, so with control and treated the reference is control by luck; with healthy and disease it is disease, and every fold change flips sign. Indexing: R counts from 1 and Python from 0, and a slice in Python excludes its end. Missing values: mean(x) of a vector with an NA is NA unless you ask na.rm = TRUE, and a filter silently drops rows where the test is NA. Recycling: adding a vector of length 2 to one of length 6 repeats the short one, and gives a warning only when the lengths do not divide evenly.

Plots that say what the data say

A plot is an analysis step, not a decoration. Show the individual data points where there are few, not only a bar of the mean. Label axes with units. Use a log scale for quantities that span orders of magnitude, and say so. Start a bar chart's axis at zero. Do not use a colour scheme that a colour-blind reader cannot separate. And report the number of observations behind every summary.

Make it repeatable

Analyse in a script, or in a notebook that you can run from top to bottom in a fresh session. The danger of notebooks is hidden state: a cell run out of order leaves a variable behind that the final code no longer creates. Set a random seed wherever something is random (set.seed(1) in R, np.random.default_rng(1) in Python), and record the versions of the language and packages (sessionInfo() in R, pip freeze or a  environment file in Python) (Sandve et al., 2013).

McKinney, Python for Data Analysis, covers NumPy in ch. 4, pandas in ch. 5, loading data in ch. 6, wrangling in ch. 7 and group operations in ch. 9. For R, see Buffalo, Bioinformatics Data Skills, ch. 8, a rapid introduction to the language, and Irizarry, Introduction to Data Science (the tidyverse and data wrangling parts) and Wickham and Grolemund, R for Data Science (tidy data). Gentleman, R Programming for Bioinformatics, is the reference for Bioconductor's design.

Check your understanding

A count table has one row per gene and one column per sample. Is it tidy? What would the tidy form be?

It is a wide table, convenient to read. The tidy (long) form has one row per gene and sample, with columns gene, sample and count, so that sample can be used for grouping and plotting.

You compare "healthy" and "disease" in R and every fold change has the opposite sign from the paper. What is the likely cause?

The reference level. Factor levels are ordered alphabetically, so "disease" is the reference and the effect is healthy versus disease. Set the level order, or relevel, so that "healthy" is the reference.

After merging a sample table with a results table you have more rows than before. What happened?

A key was not unique on one side, so rows were repeated for each match. Check the keys for duplicates before merging, and the row count after.

Why can a gene list opened and saved in a spreadsheet be wrong?

Some gene symbols, such as SEPT2 or MARCH1, are converted to dates automatically, and the original symbol is lost.

Core skills · Programming and data

Text tools for biological data: grep, awk, sed, join and regular expressions

Almost every file in bioinformatics is text: FASTA, FASTQ, GFF, VCF, count tables, logs. A handful of old Unix tools read all of them, in seconds, without loading anything into memory. Learn these tools well and most questions about a file ("how many?", "which ones?", "what is in column 5?") are one line long.

Look before you act

Every analysis that goes wrong starts with a file that was not what its author thought. Before any tool, look. head -n 5 f shows the start; wc -l f counts lines; file f says whether it is text, gzip or something else; cut -f1-4 f | head shows the first four columns; and cat -A f | head -n 3 shows invisible characters: tabs appear as ^I and the end of every line as $. A file saved on Windows has ^M$ at each line end (a carriage return), which breaks comparisons that look identical on screen. tr -d '\r' < f > f.unix repairs it.

Most tables are tab-separated (TSV) or comma-separated (CSV). The delimiter matters: cut and sort use a tab by default, and awk splits on whitespace unless told otherwise with -F'\t'. A gene description with a space in it will be split into two fields by plain awk, so for tab-separated data always write -F'\t'.

The files this chapter reads

All examples below were run on small files made for the chapter. A count table counts.tsv (genes in rows, samples in columns), a sample table meta.tsv, a gene annotation in GFF3, and a FASTA file:

counts.tsv           meta.tsv          a.fa
gene  s1   s2   s3     sample  group     >seq1 Escherichia coli
geneA 10   0    5      s1      control   ATGCGTACGTTAGC
geneB 200  180  210    s2      treated   ATGGCC
geneC 0    0    3      s3      treated   >seq2 Bacillus subtilis
                                         GGGTTTAAACCC

grep: find the lines that matter

CommandMeaning
grep 'text' flines containing the text
grep -v '^#' flines that do not start with #: drops comments and headers
grep -c '>' a.facount the matching lines (here, the sequences of a FASTA file)
grep -E 'a|b'extended regular expressions: alternation, +, {n}, grouping
grep -w 'gyrB'whole word only, so that gyrB does not match gyrB2
grep -n, -A 2, -B 2show line numbers, and two lines after or before each match
grep -F -f ids.txt flines of f that contain any identifier listed in ids.txt, as fixed strings

grep -f without -F reads each identifier as a pattern, where . matches any character: NC_001416.1 would also match NC_001416x1. For a list of identifiers always add -F.

Regular expressions, properly

A regular expression (regex) is a pattern for text. Most characters match themselves; a few are operators. These are enough for almost all work:

PatternMatches
.any one character. To match a real dot, write \.
^, $the start and the end of the line
[ACGT], [^ACGT], [0-9]one character from the set, one not in the set, one digit
*, +, ?the previous item zero or more times, one or more times, zero or one time
{3}, {2,5}exactly 3 times, between 2 and 5 times
(a|b)a group, with a choice. A group can be reused in sed as \1

Two examples, both run. Accession numbers that look like  nucleotide records, six digits and a version:

$ printf 'NC_001416.1\nAB123\nNC_045512.2\n' | grep -E '^NC_[0-9]{6}\.[0-9]+$'
NC_001416.1
NC_045512.2

And a renaming with sed, which keeps part of the match with a group and writes it back as \1:

$ echo "sample_01_R1.fastq.gz" | sed -E 's/_R([12])\.fastq\.gz/_\1.fq/'
sample_01_1.fq

Two warnings. Anchor your patterns. Without ^ and $ a pattern matches anywhere in the line, which is rarely what you meant. And write the dot as \.: a bare dot is a wildcard. Regular expressions are also the wrong tool for nested formats (for a tree in Newick format or a JSON file, use a parser; see the Python chapters).

cut, sort, uniq, paste: columns and counts

cut -f1,3 f keeps columns 1 and 3 (tab by default; -d, for commas). paste a b glues files side by side. sort -k2,2nr sorts by column 2, numerically, in reverse. uniq -c counts runs of identical lines, which is why its input must be sorted first; sort | uniq -c | sort -rn is the idiom for "count each value, most common first". For a numeric sort always add -n: plain sort puts 19282 before 5386, because it compares characters.

$ cut -f1,3 counts.tsv | paste - <(cut -f4 counts.tsv)
gene    s2      s3
geneA   0       5
geneB   180     210
geneC   0       3

The <( ... ) is process substitution: the output of a command is offered to another command as if it were a file. It avoids temporary files, and it is how you compare the outputs of two commands: diff <(cut -f1 a | sort) <(cut -f1 b | sort) lists identifiers that are in one file and not in the other.

awk: the language for tables

awk reads a file line by line, splits each line into fields $1, $2, … (and $0 is the whole line), and runs your instructions. A program is a set of pattern { action } pairs; BEGIN runs before the first line and END after the last. NR is the line number and NF the number of fields. Variables start at 0 and arrays are built as you go.

sed: edit a stream

sed 's/old/new/' f replaces the first old on each line; add g for every one. sed 1d deletes the first line (the header). sed -n '5,10p' prints lines 5 to 10. sed -E turns on extended regular expressions, and -i edits the file in place, which is dangerous: test without -i first, or write to a new file. A typical use is to pull a field out of a long annotation column:

$ grep -v '^#' anno.gff3 | grep -P '\tgene\t' | sed -E 's/.*Name=([^;]+).*/\1/'
dnaA
gyrB
recA

Read the substitution: match any text (.*), then Name=, then capture one or more characters that are not a semicolon ([^;]+), then anything; replace the whole line by the captured group. For a field in a GFF attribute or a VCF INFO column this idiom is what you need.

join and sort: combine two tables

join merges two files on a common first column, like a database join, but both files must be sorted on that column in the same way. Add -t $'\t' to use a tab; -v1 prints the lines of the first file that found no partner. Joining the count table to an annotation of two genes:

$ sed 1d counts.tsv | sort -k1,1 | join -t $'\t' - ann.tsv
geneA   10      0       5       kinase
geneB   200     180     210     gyrase
$ sed 1d counts.tsv | sort -k1,1 | join -t $'\t' -v1 - ann.tsv
geneC   0       0       3

The second command is the one that protects you: it lists the rows that did not match. Every merge should be followed by a check of what was lost.

Other tools worth knowing

ToolUse
tr 'ACGT' 'TGCA', revtranslate characters; reverse a line. echo ACGT | rev | tr ACGT TGCA gives the reverse complement ACGT (it is its own reverse complement)
fold -w1 | sort | uniq -cone character per line, then count: base composition (A 7, C 8, G 9, T 8 in the example FASTA)
paste -sd, -join lines into one comma-separated line: a quick list of identifiers
find dir -name '*.fq'list files by name, size or age, recursively
xargs -P 4 -I{} cmd {}run a command on each input item, four at a time
zcat, gzip -c, bgzip, read compressed files without unpacking; index a sorted, bgzip-compressed table for random access
, , format-aware tools: use them for FASTA/FASTQ, BAM and VCF rather than awk, because they understand the format (Shen et al., 2016; Danecek et al., 2021)

When to stop using these tools

One-line tools are best for looking at data and for quick, well-defined summaries. Switch to a script when the logic has more than two or three conditions, when you need to read the same file twice, when the format is nested (JSON, Newick, XML), or when others must read and trust it. A 10-line Python or R script is easier to review than a 300-character awk line. The next chapters are about exactly that step.

Check your understanding

A GFF3 file has 3,200 lines, 40 of them comments. Which one-line command counts the genes?

awk -F'\t' '!/^#/ && $3=="gene"' f | wc -l, or grep -v '^#' f | cut -f3 | grep -c '^gene$'. Both skip comments and test the third column exactly.

You ran sort -k1,1 a | join - b and many rows are missing, with no error. What do you check?

That b is sorted on the same key with the same locale, and that the key columns hold the same text (hidden carriage returns, different case). Run join -v1 to list what did not match.

What does sed -E 's/(.*)_(R[12])/\2_\1/' do to sampleA_R1?

It swaps the two parts around the last underscore: R1_sampleA. The groups are written back in the order \2, \1.

Why does grep -f ids.txt table.tsv sometimes return rows that are not in the list?

Each identifier is read as a pattern: a dot matches any character, and an identifier like g1 also matches g10. Use grep -F -w -f.

Core skills · Programming and data

Bash scripting: from one-liners to reliable pipelines

A one-liner answers a question once. A script answers it every time, on every file, for everyone. This chapter builds, step by step, a small but real script with the features of a good one: arguments, checks, loops, functions, clear errors and an exit status other programs can read. The script is run on real files and every output below is its real output.

Variables and quoting

A variable is set without spaces (n=3) and read with a dollar sign ($n or ${n}). Quote every expansion: "$f", not $f. Without quotes the shell splits the value at spaces and expands wildcards in it. The difference is easy to see:

$ x="a b"; for w in $x; do echo "[$w]"; done
[a]
[b]
$ x="a b"; for w in "$x"; do echo "[$w]"; done
[a b]

Single quotes keep text exactly as written; double quotes allow $variable and $(command) to be replaced by their values. $(command) runs a command and substitutes its output: today=$(date +%F).

Parameter expansion: edit names without sed

The shell can cut and replace parts of a variable. These are used constantly to build output names from input names:

$ name=reads_001.fastq.gz
$ echo ${name%.fastq.gz}        # remove a suffix
reads_001
$ echo ${name#reads_}           # remove a prefix
001.fastq.gz
$ echo ${name/reads/trimmed}    # replace the first match
trimmed_001.fastq.gz
$ echo ${#name}                 # length
18

Also useful: ${var:-default} (use a default if unset or empty), ${var:?message} (stop with a message if unset or empty) and basename and dirname for paths.

Tests, conditions and exit status

Every command returns an exit status: 0 for success, anything else for failure. if tests it. The test [[ ... ]] compares strings and checks files: -e exists, -f is a regular file, -d is a directory, -s exists and is not empty, -z "$s" the string is empty, == compares strings, and -eq -lt -gt compare integers. Combine tests with &&, || and !.

if [[ -s "$fastq" && ! -e "$out" ]]; then
    echo "running"
fi
cmd || { echo "cmd failed" >&2; exit 1; }   # stop if cmd fails

A function returns a status the same way: f() { return 3; }; f; echo $? prints 3 (tested). Always send error messages to standard error (>&2), so that they do not end up in a results file.

Loops, arrays and reading lines

for f in data/*.fq; do ...; done          # over files
while read -r line; do ...; done < list.txt  # over lines of a file
arr=(s1 s2 s3); echo ${arr[1]}             # an array: prints s2
for i in "${!arr[@]}"; do echo $i:${arr[$i]}; done
declare -A len; len[s1]=3; len[s2]=2      # an associative array (a dictionary)

Run on three names, the array loop printed 0:s1, 1:s2, 2:s3; the dictionary loop printed s1=3 and s2=2. read -r reads a line without interpreting backslashes: always use -r. Looping over $(ls) breaks on spaces in names; loop over a glob (*.fq) instead.

A complete script, built up

Here is a script that reports the number of reads and the mean read length of every FASTQ file in a directory, as a table. It uses everything above. The file is screen.sh:

#!/usr/bin/env bash
# screen.sh: count reads and mean length for every FASTQ file in a directory.
set -euo pipefail

usage() { echo "usage: $0 -d DIR [-o OUT]" >&2; exit 2; }
out=summary.tsv
while getopts "d:o:h" opt; do
  case $opt in
    d) dir=$OPTARG ;;
    o) out=$OPTARG ;;
    *) usage ;;
  esac
done
: "${dir:?give a directory with -d}"
[[ -d $dir ]] || { echo "not a directory: $dir" >&2; exit 1; }

printf 'file\treads\tmean_length\n' > "$out"
for f in "$dir"/*.fq; do
  [[ -e $f ]] || { echo "no .fq files in $dir" >&2; exit 1; }
  awk -v name="$(basename "$f")" 'NR%4==2 {n++; t+=length($0)} END {printf "%s\t%d\t%.2f\n", name, n, t/n}' "$f" >> "$out"
done
echo "wrote $out" >&2

Read it in blocks. The first line says which interpreter runs the file. set -euo pipefail makes failure visible. getopts reads options: d: means -d takes a value. : "${dir:?...}" stops with a message if -d was not given. Then the directory is checked, the header written, and each file summarised by awk. It was run on two small files, s1.fq (3 reads) and s2.fq (2 reads):

$ ./screen.sh -d data -o results/summary.tsv
wrote results/summary.tsv
$ cat results/summary.tsv
file    reads   mean_length
s1.fq   3       9.33
s2.fq   2       11.00
$ ./screen.sh -d nodir; echo "exit=$?"
not a directory: nodir
exit=1
$ ./screen.sh; echo "exit=$?"
./screen.sh: line 14: dir: give a directory with -d
exit=1

The last two runs matter as much as the first. A script that fails loudly, early and with a non-zero status can be put inside a pipeline, a scheduler job or a  process, which stop when it fails. A script that fails quietly lets everything after it run on wrong input.

Safe habits

  • Start with set -euo pipefail: stop on a failed command, on an unset variable, and on a failure anywhere in a pipeline. false | true reports success without pipefail and failure with it (tested). Know the limit: set -u catches an unset variable but not an empty one; ${var:?} catches both.
  • Never build a deletion from a variable you did not check. rm -rf "${out:?}"/tmp stops if out is empty; rm -rf $out/tmp becomes rm -rf /tmp.
  • Write results to a new file, then move it into place. cmd > out.tmp && mv out.tmp out: an interrupted run then leaves no half-written result that a later step mistakes for a finished one.
  • Skip work that is done. [[ -s "$out" ]] && continue at the top of a loop makes the script resumable.
  • Clean up with trap. trap 'rm -f "$tmp"' EXIT runs when the script ends, even on error. Tested: trap "echo cleaning up" EXIT; echo working prints working then cleaning up.
  • Log what you ran. Print the command, the version of the tools ( --version | head -n 1) and the date at the top of the log.
  • Use shellcheck on every script. It finds unquoted variables, useless cat and many other mistakes, and explains each.

Doing many things at once

xargs -P N runs N copies of a command in parallel, each on one input item. ls data/*.fq | xargs -P 2 -I{} sh -c 'echo "{}: $(grep -c "^+$" {}) records"' counted the records of both files at the same time (s1.fq: 3 records, s2.fq: 2 records). GNU parallel does the same with more control. Count the cores you use: on a shared server you request them from the scheduler and then use no more than you were given. For anything with many steps and dependencies between them, a workflow manager (, ) does what a long script does badly: it knows which outputs exist, reruns only what changed, and records what was run (Di Tommaso et al., 2017).

From script to pipeline

A useful progression is: one-liners to explore; a script per step, each taking inputs and an output directory as arguments; a driver script or a workflow manager that calls the steps; and a README with the commands. At every stage, keep raw data read-only and let results be regenerable. The reproducibility rules of the previous chapter apply here without change (Sandve et al., 2013).

Check your understanding

A loop for f in $(ls *.fq) fails on a file called my reads.fq. Why, and what do you write?

The unquoted command substitution is split at the space into my and reads.fq. Loop over the glob directly, for f in *.fq; do ... "$f" ...; done, and avoid spaces in names.

What does ${name%.fastq.gz} do to reads_001.fastq.gz?

It removes the shortest match of the suffix from the end: reads_001.

Your script ends with exit 0 even though a step in the middle failed. How do you prevent that?

Start with set -euo pipefail, or test each important command with || { echo "failed" >&2; exit 1; }, and never end a script with an exit 0 that hides earlier failures.

Why should error messages go to standard error?

Standard output is what the next command or the results file receives; an error message mixed into it corrupts the data. Standard error is shown on the terminal and kept in the log.

Core skills · Programming and data

Python from the ground up, with sequences as the example

Python is the language in which most bioinformatics tools and pipelines are written or glued together. This chapter teaches the language itself, from the first line to functions, files and errors, using DNA sequences as the running example. Every snippet was run; the printed output is real.

Values, names and types

A name points to a value; the value has a type. Python's built-in types cover most of what a first analysis needs:

TypeExampleUse in bioinformatics
int, float14, 0.5lengths, counts, fractions, p values
str"ATGCGT"sequences, identifiers, file names
list["r1", "r2"]an ordered, changeable collection: reads, samples
tuple("chr1", 100, 900)a fixed record: a coordinate, a name and a length
set{"A", "C"}unique items, fast membership tests, overlaps
dict{"seq1": 20}look up a value by key: length by sequence name
bool, NoneTrue, Noneconditions; "no value"

Indexing starts at 0 and a slice s[a:b] includes a and excludes b. This is the source of most off-by-one errors, and it is also why genomic interval formats differ: BED is 0-based and half-open, GFF and VCF are 1-based and closed. On seq = "ATGCGTACGTTAGC":

>>> len(seq), seq[0:3], seq[-3:], seq[::-1]
(14, 'ATG', 'AGC', 'CGATTGCATGCGTA')

Functions

A function packages a step you will repeat, with a name and a documented purpose. Write one for every idea you use twice.

comp = {"A": "T", "C": "G", "G": "C", "T": "A"}

def revcomp(s):
    """Reverse complement of an unambiguous DNA string."""
    return "".join(comp[b] for b in reversed(s))

def gc(s):
    s = s.upper()
    return (s.count("G") + s.count("C")) / len(s)

print(revcomp(seq))        # GCTAACGTACGCAT
print(f"{gc(seq):.3f}")    # 0.500

A caution that matters in biology: this revcomp fails with a KeyError on N or any ambiguity code, which is the right behaviour for a function that was written for four letters; it is better to stop than to return a wrong sequence. For real work use a library (next chapter) that handles the full alphabet.

Collections and loops

Count bases with a dictionary, then with the standard library:

counts = {}
for b in seq:
    counts[b] = counts.get(b, 0) + 1
print(counts)                      # {'A': 3, 'T': 4, 'G': 4, 'C': 3}

from collections import Counter
print(Counter(seq).most_common(2)) # [('T', 4), ('G', 4)]

A list comprehension builds a list in one line: [i for i in range(len(seq) - 2) if seq[i:i+3] == "CGT"] returns the positions of the motif, [3, 7]. Codons are [seq[i:i+3] for i in range(0, len(seq) - len(seq) % 3, 3)], giving ['ATG', 'CGT', 'ACG', 'TTA']; the last two bases do not fill a codon and are dropped on purpose. sorted(reads, key=lambda r: r[1], reverse=True) sorts [("r1", 10), ("r2", 12), ("r3", 6)] by length, longest first. enumerate gives an index and zip walks two lists together: for i, (n, l) in enumerate(zip(names, lengths), start=1).

Reading a file

Always open files with with, which closes them even when an error occurs. This reads a FASTA file, in which a record can span several lines, into a dictionary of name to sequence:

recs, name = {}, None
with open("a.fa") as fh:
    for line in fh:
        line = line.rstrip("\n")
        if line.startswith(">"):
            name = line[1:].split()[0]
            recs[name] = []
        else:
            recs[name].append(line)
recs = {k: "".join(v) for k, v in recs.items()}
print({k: len(v) for k, v in recs.items()})   # {'seq1': 20, 'seq2': 12}

This parser is fine for a lesson and for a small file. For production work use a tested parser (Biopython, next chapter): it handles blank lines, Windows line endings and the many little ways real files deviate.

Errors

Python stops at an error and tells you which line and why. Read the last line of the message first: it names the error and says what went wrong. Catch an error only when you know how to recover:

try:
    n = int("12x")
except ValueError as e:
    print("ValueError:", e)      # ValueError: invalid literal for int() with base 10: '12x'

Never write a bare except: that swallows everything. It turns a clear failure into silent wrong results.

Traps for beginners (all run)

TrapWhat happens
A mutable default argument: def add(x, acc=[])The list is created once, so add(1), add(2) returns [1, 2] [1, 2]: the calls share state. Use acc=None and make the list inside: [1] [2]
Floating point: 0.1 + 0.2 == 0.3False. Floats are binary approximations. Compare with a tolerance (math.isclose), never with ==
round(2.675, 2)2.67, not 2.68, for the same reason
7 / 2 and 7 // 23.5 and 3; 7 % 2 is 1. Division with / always gives a float
Changing a list while looping over itItems are skipped. Loop over a copy, or build a new list with a comprehension

Modules: write it once, import it, or run it

A file of Python code is a module. Another program can import it and use its functions, and the same file can be run on its own. The condition if __name__ == "__main__": separates the two: its block runs only when the file is executed, not when it is imported. That is how a script becomes a library without losing its command-line use. The file below, seqtools.py, holds three helpers (a FASTA reader, a reverse complement and a k-mer counter) and is used in the rest of this chapter. It reads the genome of chapter P12.

seqtools.py
"""seqtools: a few sequence helpers. Import it, or run it as a script."""
import sys
from collections import Counter

COMP = str.maketrans("ACGTacgt", "TGCAtgca")


def read_fasta(path):
    """Yield (name, sequence) for each record, reading one line at a time."""
    name, parts = None, []
    with open(path) as fh:
        for line in fh:
            line = line.rstrip()
            if line.startswith(">"):
                if name is not None:
                    yield name, "".join(parts)
                name, parts = line[1:].split()[0], []
            elif line:
                parts.append(line)
    if name is not None:
        yield name, "".join(parts)


def revcomp(seq):
    return seq.translate(COMP)[::-1]


def kmer_counts(seq, k):
    return Counter(seq[i:i + k] for i in range(len(seq) - k + 1))


if __name__ == "__main__":                       # runs only when the file is executed, not when it is imported
    for name, seq in read_fasta(sys.argv[1]):
        print(name, len(seq))
$ python3 seqtools.py ../data/phix174.fa
gi|9626372|ref|NC_001422.1| 5386

$ python3 -c "import seqtools; print(seqtools.__name__, [n for n in dir(seqtools) if not n.startswith('_')][:4])"
seqtools ['COMP', 'Counter', 'kmer_counts', 'read_fasta']

Run as a script, it prints the name and length of each record. Imported, it prints nothing and offers its functions. Two habits follow. Put reusable code in modules and keep scripts thin. And never give a module a name that an installed package already has (random.py, csv.py): Python imports your file instead of the library, and the error that follows is baffling.

Iterators and generators: one record at a time

The function read_fasta uses yield instead of return. A function with yield is a generator: it produces one value, pauses, and resumes where it stopped when the next value is asked for. A for loop over it holds a single record in memory, however large the file. This is the difference between a program that handles a 100 GB FASTQ file and one that stops at the first 8 GB. The test below reads the 6,000 reads of chapter P12 twice, once into a list and once through a generator, and measures the peak memory of each with tracemalloc (records is the four-line FASTQ reader of the command-line tool in chapter P6).

import sys, tracemalloc
sys.path.insert(0, ".")
from fqtool import records

def mean_q(qual):
    return sum(ord(c) - 33 for c in qual) / len(qual)

tracemalloc.start()
with open("../data/S1.fq") as fh:
    everything = list(records(fh))                   # all 6,000 reads held in memory at once
a = tracemalloc.get_traced_memory()[1]
del everything
tracemalloc.reset_peak()
with open("../data/S1.fq") as fh:
    total = sum(mean_q(q) for _, _, q in records(fh))   # one read at a time
b = tracemalloc.get_traced_memory()[1]
print(f"peak memory holding the list: {a / 1e6:.2f} MB; with a generator: {b / 1e6:.3f} MB; same answer: {total / 6000:.2f}")

# output
peak memory holding the list: 2.53 MB; with a generator: 0.159 MB; same answer: 33.55

Holding the list takes 2.53 MB for a 1.3 MB file, because every Python string carries overhead, while the generator needs 0.159 MB, and the answer is the same. The rule: stream when you can, load when you must. A generator can be used only once; to go over the data twice, call the function again. Open files with with, which closes them even when an error occurs, and open gzip files with gzip.open(path, "rt"), which reads text from a compressed file without unpacking it.

Counting things: dictionaries, Counter and k-mers

Many questions in bioinformatics are questions of counting: bases, codons, k-mers, reads per gene. A dictionary maps a key to a number, and collections.Counter is a dictionary built for counting (it returns 0 for a key it has not seen, and most_common(n) sorts). A k-mer is a substring of length k. A sequence of length L has L − k + 1 of them, which a generator expression counts directly. The tetranucleotides (k = 4) of a genome form a signature that differs between organisms, and metagenome binning tools use it to group contigs that belong together.

import sys
sys.path.insert(0, ".")
from seqtools import read_fasta, kmer_counts

name, seq = next(read_fasta("../data/phix174.fa"))
c = kmer_counts(seq, 4)
print(len(seq), "bases;", sum(c.values()), "tetranucleotides;", len(c), "of 256 possible occur")
print("most common:", c.most_common(4))
print("GATC:", c["GATC"], "| expected about", round((len(seq) - 3) / 256, 1))
for k, v in sorted(c.items(), key=lambda kv: kv[1])[:3]:
    print("rarest:", k, v)

# output
5386 bases; 5383 tetranucleotides; 255 of 256 possible occur
most common: [('TGCT', 67), ('TGGT', 57), ('TTTT', 55), ('GTTT', 51)]
GATC: 0 | expected about 21.0
rarest: GGGG 3
rarest: CGGG 3
rarest: CTAG 3

The 5,386-base genome holds 5,383 tetranucleotides, and 255 of the 256 possible ones occur. The most common, TGCT, appears 67 times. The one that is absent is GATC, the Dam methylation site that chapter P12 found missing with grep and discussed. Two independent tools reaching the same zero is the kind of check that gives confidence in both.

Patterns: regular expressions in Python

The re module is the same regular expression language as grep -E and sed -E (P3), inside a program. re.finditer gives the position of every match (0-based, like grep -ob), and re.findall a list of matches. Ambiguity codes are turned into character classes with a dictionary. One trap deserves care: matches do not overlap. Counting AA in AAAA gives 2 with a normal pattern but 3 positions, and a look-ahead (?=AA), which matches a position without consuming the characters, counts all three. Use a look-ahead whenever a motif can overlap itself.

import re
seq = "".join(l.strip() for l in open("../data/phix174.fa") if not l.startswith(">"))
iupac = {"R": "[AG]", "Y": "[CT]", "W": "[AT]", "S": "[CG]", "N": "[ACGT]"}
pattern = "".join(iupac.get(c, c) for c in "AGGAGNNNNATG")       # exact bases stay, ambiguity codes become classes
print(pattern)
print([m.start() for m in re.finditer("AGGAG", seq)], "<- AGGAG, 0-based, as grep -ob gave")
print("non-overlapping 'AA' in 'AAAA':", len(re.findall("AA", "AAAA")), "| overlapping with a look-ahead:", len(re.findall("(?=AA)", "AAAA")))
print([m.start() for m in re.finditer("(?=(" + "AGGAG[ACGT]{4,10}ATG" + "))", seq)], "<- ribosome-binding site then ATG")

# output
AGGAG[ACGT][ACGT][ACGT][ACGT]ATG
[232, 835, 2382, 3540, 5062] <- AGGAG, 0-based, as grep -ob gave
non-overlapping 'AA' in 'AAAA': 2 | overlapping with a look-ahead: 3
[835, 2382, 5062] <- ribosome-binding site then ATG

The five AGGAG positions are the offsets that grep -ob printed in P12, and the three ribosome-binding sites followed by ATG are the same three. Positions that two methods agree on need no further argument.

A small class: a genomic interval

A class bundles data and the functions that belong to it. When you find yourself passing the same three values (a start, an end and a strand) around a program, a class is the cleaner design. dataclasses.dataclass writes the boring parts (the constructor, __repr__ and comparison) for you, and order=True makes objects sortable by their fields in order. The class below stores an interval with BED-style 0-based half-open coordinates, converts from a GFF line (the conversion that chapter P12 did with awk) and tests for overlap: two half-open intervals overlap when each starts before the other ends.

from dataclasses import dataclass

@dataclass(order=True)
class Interval:
    start: int            # 0-based, half-open, like BED
    end: int
    name: str = ""
    strand: str = "+"

    def __len__(self):
        return self.end - self.start

    @classmethod
    def from_gff(cls, line):
        f = line.rstrip("\n").split("\t")
        return cls(int(f[3]) - 1, int(f[4]), f[8].split(";")[0].replace("ID=", ""), f[6])

    def overlaps(self, other):
        return self.start < other.end and other.start < self.end

genes = sorted(Interval.from_gff(l) for l in open("../data/genes.gff") if not l.startswith("#"))
print(genes[0])
print("coding bases:", sum(len(g) for g in genes), "| overlapping pairs:", sum(a.overlaps(b) for i, a in enumerate(genes) for b in genes[i + 1:]))

# output
Interval(start=50, end=221, name='1_1', strand='+')
coding bases: 4950 | overlapping pairs: 1

The seven genes cover 4,950 bases, as found, and exactly one pair overlaps: genes 1_2 and 1_3 share one base at position 848. Overlaps of one or a few bases between neighbouring genes are common in compact genomes, where the two proteins are made in a coupled way (translational coupling). The half-open convention is what makes the test self.start < other.end and other.start < self.end correct without a special case for touching intervals.

An algorithm you can write: global alignment

Every sequence-comparison tool rests on dynamic programming (chapter A1): fill a table in which each cell holds the best score of aligning the first i bases of one sequence with the first j of the other, by choosing the best of a diagonal step (match or mismatch) and the two gap steps, then trace back from the corner. It takes 25 lines, and writing it once explains why alignment time grows with the product of the lengths. The function below is the Needleman–Wunsch algorithm with a linear gap penalty. It is checked against Biopython's PairwiseAligner, which does the same job in compiled code.

def needleman_wunsch(a, b, match=1, mismatch=-1, gap=-2):
    """Global alignment by dynamic programming: fill a score table, then trace back."""
    n, m = len(a), len(b)
    S = [[0] * (m + 1) for _ in range(n + 1)]
    for i in range(1, n + 1):
        S[i][0] = i * gap
    for j in range(1, m + 1):
        S[0][j] = j * gap
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            diag = S[i - 1][j - 1] + (match if a[i - 1] == b[j - 1] else mismatch)
            S[i][j] = max(diag, S[i - 1][j] + gap, S[i][j - 1] + gap)
    i, j, x, y = n, m, [], []
    while i or j:
        if i and j and S[i][j] == S[i - 1][j - 1] + (match if a[i - 1] == b[j - 1] else mismatch):
            x.append(a[i - 1]); y.append(b[j - 1]); i -= 1; j -= 1
        elif i and S[i][j] == S[i - 1][j] + gap:
            x.append(a[i - 1]); y.append("-"); i -= 1
        else:
            x.append("-"); y.append(b[j - 1]); j -= 1
    return S[n][m], "".join(reversed(x)), "".join(reversed(y))

a, b = "GATTACAGATTACA", "GATCACAGTTACCA"
score, x, y = needleman_wunsch(a, b)
print(score); print(x); print(y)
from Bio.Align import PairwiseAligner
al = PairwiseAligner(mode="global", match_score=1, mismatch_score=-1, open_gap_score=-2, extend_gap_score=-2)
print("Biopython score:", al.score(a, b), "| same:", al.score(a, b) == score)

# output
7
GATTACAGATTA-CA
GATCACAG-TTACCA
Biopython score: 7.0 | same: True

Both give a score of 7: twelve matches, one mismatch and two gaps (one in each sequence), which is 12 − 1 − 4 with a score of 1 for a match, −1 for a mismatch and −2 for a gap. Keep the hand-written version for understanding; use the library for work, because it is much faster, supports affine gaps, local alignment and substitution matrices, and has been tested by thousands of users. The match between the two is the test, and writing a slow reference to check a fast tool is a habit worth keeping.

Projects and environments

Create one directory per project with data/ (raw, read-only), results/, scripts/ and a README. Create an isolated environment so that the project keeps its own library versions: python -m venv .venv && source .venv/bin/activate && pip install biopython pandas, then pip freeze > requirements.txt. Or use  ( env create -f env.yml), which also installs non-Python tools such as and . The file that lists the versions is part of the result: a paper that says "analysed in Python" and nothing more cannot be reproduced (Wilson et al., 2014). In a Jupyter notebook, restart the kernel and run all cells from the top before you trust a result, since cells run out of order leave hidden state behind.

Check your understanding

What does seq[2:5] return for "ATGCGTACG", and how many characters?

"GCG", three characters: index 2 is the third letter, and the end index 5 is excluded.

Why does def f(x, acc=[]) give surprising results?

The default list is created once, when the function is defined, and every call that does not pass acc appends to the same list. Use None and create the list inside the function.

A p value of exactly 0.05 is computed as 0.1 + 0.2 - 0.25. Why is p == 0.05 unreliable?

Floating-point arithmetic is approximate, so the result differs from 0.05 by a tiny amount. Compare with math.isclose or with a tolerance.

Why use with open(...)?

The file is closed when the block ends, even if an error occurs, so no file is left open and no buffered data is lost.

Core skills · Programming and data

Python for sequences and tables: Biopython, pandas, and a tested command-line tool

Two libraries cover most daily work: Biopython for sequence files and biological objects, and pandas for tables. This chapter uses both on small files whose answers can be checked by hand, then turns the result into a command-line tool with arguments and tests, which is the step that separates a script you run from software others can use.

Sequences with Biopython

Biopython (Cock et al., 2009) parses the formats and knows the biology: the genetic code, ambiguity codes, reverse complements. SeqIO.parse(file, format) yields one record at a time, so it works on files of any size.

from Bio import SeqIO
from Bio.Seq import Seq
from Bio.SeqUtils import gc_fraction

for rec in SeqIO.parse("a.fa", "fasta"):
    print(rec.id, len(rec), round(gc_fraction(rec.seq), 3), rec.description)
# seq1 20 0.55 seq1 Escherichia coli
# seq2 12 0.5 seq2 Bacillus subtilis

s = Seq("ATGGCCATTGTAATGGGCCGCTGA")
print(s.reverse_complement())     # TCAGCGGCCCATTACAATGGCCAT
print(s.translate())              # MAIVMGR*
print(s.translate(to_stop=True))  # MAIVMGR

For FASTQ, rec.letter_annotations["phred_quality"] holds the quality of each base. On the three-read example file, read r2 has a minimum quality of 2 and a mean of 36.8 (it contains the character #, Phred 2), whereas r1 is 40 throughout. A Phred score Q means an error probability of 10−Q/10: Q20 is 1 error in 100, Q30 is 1 in 1,000.

Check three things whenever you translate: that the sequence is in the right frame, that you use the right genetic code table (table=11 for bacteria; table=4 for Mycoplasma), and that an in-frame stop codon in the middle (a *) means a pseudogene, a frameshift or a wrong frame, not a protein.

Tables with pandas

A DataFrame is a table with labelled rows and columns (McKinney, 2010). Read, look, then check:

import pandas as pd, numpy as np
c = pd.read_csv("counts.tsv", sep="\t", index_col=0)   # genes x samples
m = pd.read_csv("meta.tsv",   sep="\t", index_col=0)   # samples x variables
print(c.sum(axis=0).to_dict())        # {'s1': 210, 's2': 180, 's3': 218}
print((c > 0).sum(axis=1).to_dict())  # {'geneA': 2, 'geneB': 3, 'geneC': 1}
assert list(c.columns) == list(m.index)   # the same samples, in the same order

That last line is the most useful line in the chapter. Samples in the count table and in the metadata must be the same, in the same order; a silent mismatch assigns each sample to the wrong group and produces results that look normal. Assert it every time.

Normalising: relative abundance and counts per million

rel = c / c.sum(axis=0) * 100              # percent of each sample
print(rel.round(1))
#          s1     s2    s3
# geneA   4.8    0.0   2.3
# geneB  95.2  100.0  96.3
# geneC   0.0    0.0   1.4

cpm = c.div(c.sum(axis=0), axis=1) * 1e6
print(np.log2(cpm + 1).round(2))           # the +1 is a pseudocount: log(0) is undefined
#          s1     s2     s3
# geneA  15.54   0.00  14.49
# geneB  19.86  19.93  19.88
# geneC   0.00   0.00  13.75

Look at geneA: its relative abundance is 4.8% in s1 and 0% in s2, and in the log scale a zero stays zero. Two cautions follow. Relative abundance is compositional: in s2, geneB is 100% because nothing else was counted, and if one gene rises the others must fall. And a pseudocount is a choice that changes the result for rare features; report it.

Grouping, reshaping and merging

Analyses ask "for each group, what is the mean?". Reshape from wide (one column per sample) to long (one row per gene and sample), attach the metadata, and group (Wickham, 2014):

long = (c.reset_index()
         .melt(id_vars="gene", var_name="sample", value_name="count")
         .merge(m, left_on="sample", right_index=True))
print(long.groupby(["gene", "group"])["count"].mean().unstack().round(1))
# group  control  treated
# gene
# geneA     10.0      2.5
# geneB    200.0    195.0
# geneC      0.0      1.5

A merge is where tables silently go wrong. If the key is not unique on one side, rows multiply: merging a table of 3 rows with a second table in which the key 1 appears twice gave 5 rows, not 3 (tested). After every merge, compare the number of rows before and after, check for missing values (df.isna().sum()) and check keys for duplicates (df["k"].is_unique). Read a merge's validate="one_to_one" argument as a free assertion.

A tested command-line tool

A script becomes a tool when it takes its inputs as arguments, does one thing, reports errors clearly and is tested. This tool, fqstats.py, reads FASTQ files (also gzip-compressed) and reports the number of reads, the mean read length and the mean quality, optionally ignoring low-quality reads. The core is a generator that reads four lines at a time:

def read_fastq(handle):
    """Yield (name, bases, quals) from a FASTQ file, four lines at a time."""
    while True:
        name = handle.readline().rstrip("\n")
        if not name:
            return
        bases = handle.readline().rstrip("\n")
        handle.readline()
        quals = handle.readline().rstrip("\n")
        if len(bases) != len(quals):
            raise ValueError(f"{name}: {len(bases)} bases but {len(quals)} quality characters")
        yield name[1:], bases, quals

The summary function converts each quality character to a number (ord(c) - 33, the Sanger/Illumina 1.8+ offset), averages, and skips reads below a threshold. The main function uses argparse, which builds the help text, checks the arguments and gives a clear error:

def main(argv=None):
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("files", nargs="+", help="FASTQ files, optionally gzipped")
    ap.add_argument("--min-quality", type=float, default=0,
                    help="skip reads with a lower mean quality (default: %(default)s)")
    a = ap.parse_args(argv)
    print("file\treads\tmean_length\tmean_quality")
    for f in a.files:
        s = summarise(f, a.min_quality)
        print(f"{f}\t{s['reads']}\t{s['mean_length']:.2f}\t{s['mean_quality']:.1f}")
    return 0

Run on the two example files, and again with a quality filter:

$ python fqstats.py data/s1.fq data/s2.fq
file        reads  mean_length  mean_quality
data/s1.fq  3      9.33         38.4
data/s2.fq  2      11.00        40.0
$ python fqstats.py --min-quality 37 data/s1.fq
data/s1.fq  2      8.00         39.2

The filter removed read r2 (mean quality 36.8), so the remaining two reads have lengths 10 and 6 and a mean length of 8. The numbers can be checked by hand, which is what a test is. With pytest, each test is a function whose name starts with test_:

def test_reads_are_parsed_four_lines_at_a_time():
    fh = io.StringIO("@r1\nACGT\n+\nIIII\n@r2\nAC\n+\nII\n")
    assert [(n, b) for n, b, _ in read_fastq(fh)] == [("r1", "ACGT"), ("r2", "AC")]

def test_mismatched_quality_length_is_an_error():
    with pytest.raises(ValueError):
        list(read_fastq(io.StringIO("@r1\nACGT\n+\nII\n")))

def test_quality_filter_removes_low_quality_reads():
    assert summarise("data/s1.fq", min_q=37)["reads"] == 2

All four tests of the suite passed (4 passed). Test the cases that matter most: a normal file, a file with a defect, an empty file and the edge of a threshold. A tool that has tests can be changed without fear, and a result that comes from a tested tool is worth more than one that does not (Wilson et al., 2014).

From a script to a tool people can trust

The tool above takes one command. Real tools do several things, and the pattern for that is subcommands, the way git commit and  view work. argparse builds them with add_subparsers; each subcommand has its own arguments, types (type=float) and defaults, and --version and --help come for free. Three more habits separate a tool from a script. It reports a failure on standard error with a short message and returns a non-zero exit status, so that a pipeline or a scheduler can see it failed (P4). It uses the logging module, not print, for progress messages, so that the user decides with -v how much to see. And its main(argv) takes the arguments as a parameter, so that a test can call it directly without starting a new process. The next version, fqtool.py, has two subcommands, stats and filter:

fqtool.py
#!/usr/bin/env python3
"""fqtool: summarise or filter a FASTQ file by quality."""
import argparse
import gzip
import logging
import sys

__version__ = "0.1.0"
log = logging.getLogger("fqtool")


def open_text(path):
    """Open a plain or gzipped text file."""
    return gzip.open(path, "rt") if str(path).endswith(".gz") else open(path)


def records(fh):
    """Yield (header, sequence, quality) from an open FASTQ file."""
    while True:
        head = fh.readline().rstrip()
        if not head:
            return
        seq, _, qual = fh.readline().rstrip(), fh.readline(), fh.readline().rstrip()
        if not head.startswith("@") or len(seq) != len(qual):
            raise ValueError(f"malformed record starting {head[:30]!r}")
        yield head, seq, qual


def mean_quality(qual, offset=33):
    return sum(ord(c) - offset for c in qual) / len(qual)


def cmd_stats(args):
    n = bases = 0
    total_q = 0.0
    with open_text(args.fastq) as fh:
        for _, seq, qual in records(fh):
            n += 1
            bases += len(seq)
            total_q += mean_quality(qual)
    print(f"reads\t{n}\nbases\t{bases}\nmean_quality\t{total_q / n:.2f}" if n else "reads\t0")
    return 0


def cmd_filter(args):
    kept = total = 0
    with open_text(args.fastq) as fh, open(args.out, "w") as out:
        for head, seq, qual in records(fh):
            total += 1
            if mean_quality(qual) >= args.min_quality and len(seq) >= args.min_length:
                out.write(f"{head}\n{seq}\n+\n{qual}\n")
                kept += 1
    log.info("kept %d of %d reads", kept, total)
    return 0


def build_parser():
    p = argparse.ArgumentParser(prog="fqtool", description=__doc__)
    p.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
    p.add_argument("-v", "--verbose", action="store_true", help="log progress to standard error")
    sub = p.add_subparsers(dest="command", required=True)
    s = sub.add_parser("stats", help="count reads and bases, mean quality")
    s.add_argument("fastq")
    s.set_defaults(func=cmd_stats)
    f = sub.add_parser("filter", help="keep reads that pass a quality and length threshold")
    f.add_argument("fastq")
    f.add_argument("-o", "--out", required=True)
    f.add_argument("-q", "--min-quality", type=float, default=30, help="minimum mean Phred quality (default 30)")
    f.add_argument("-l", "--min-length", type=int, default=50, help="minimum read length (default 50)")
    f.set_defaults(func=cmd_filter)
    return p


def main(argv=None):
    args = build_parser().parse_args(argv)
    logging.basicConfig(level=logging.INFO if args.verbose else logging.WARNING, format="%(levelname)s %(message)s")
    try:
        return args.func(args)
    except (OSError, ValueError) as e:
        print(f"fqtool: error: {e}", file=sys.stderr)
        return 1


if __name__ == "__main__":
    sys.exit(main())
$ python3 fqtool.py --help | head -14
usage: fqtool [-h] [--version] [-v] {stats,filter} ...

fqtool: summarise or filter a FASTQ file by quality.

positional arguments:
  {stats,filter}
    stats         count reads and bases, mean quality
    filter        keep reads that pass a quality and length threshold

options:
  -h, --help      show this help message and exit
  --version       show program's version number and exit
  -v, --verbose   log progress to standard error

$ python3 fqtool.py stats ../data/S1.fq
reads	6000
bases	600000
mean_quality	33.55

$ python3 fqtool.py -v filter ../data/S1.fq -o /tmp/pyproj_q34.fq -q 34 -l 100 2>&1; grep -c '^@S1' /tmp/pyproj_q34.fq || true
INFO kept 229 of 6000 reads
229

$ python3 fqtool.py stats missing.fq 2>&1; echo "exit=$?"; python3 fqtool.py 2>&1 | tail -1; echo "exit=${PIPESTATUS[0]}"
fqtool: error: [Errno 2] No such file or directory: 'missing.fq'
exit=1
fqtool: error: the following arguments are required: command
exit=2

The mean quality, 33.55, is the figure that awk and Biopython gave on the same file. The filter at a mean quality of 34 kept 229 of the 6,000 reads and logged that at the INFO level because of -v. Without -v it is silent. A missing file gives exit status 1 and a one-line message; a missing subcommand gives status 2, which is what argparse uses for a usage error. Such conventions are what let other programs build on yours.

Tests that fail when they should

A test is a small program that runs your code on an input whose answer you know and checks the result. pytest finds functions named test_... and reports which pass. Its three most useful tools are shown in the suite below: parametrize runs one test on several inputs, tmp_path gives a temporary directory so that tests never touch real data, and capsys captures what the program prints. Test the cases that matter: normal input, a defective file, a missing file and the edge of a threshold. The values in test_mean_quality are checked by hand: the characters +, 5, ? and I are Q10, Q20, Q30 and Q40, whose mean is 25.

test_fqtool.py
import pytest
import fqtool

FQ = "@r1\nACGT\n+\nIIII\n@r2\nACGT\n+\n!!!!\n"          # one read of Q40, one of Q0


@pytest.mark.parametrize("qual, expected", [("IIII", 40.0), ("!!!!", 0.0), ("5555", 20.0), ("+5?I", 25.0)])
def test_mean_quality(qual, expected):
    # '+' = Q10, '5' = Q20, '?' = Q30, 'I' = Q40, so "+5?I" has the mean (10 + 20 + 30 + 40) / 4 = 25
    assert fqtool.mean_quality(qual) == pytest.approx(expected)


def test_records_reads_four_lines_at_a_time(tmp_path):
    p = tmp_path / "x.fq"
    p.write_text(FQ)
    with open(p) as fh:
        recs = list(fqtool.records(fh))
    assert [r[0] for r in recs] == ["@r1", "@r2"] and recs[0][1] == "ACGT"


def test_truncated_record_is_an_error(tmp_path):
    p = tmp_path / "bad.fq"
    p.write_text("@r1\nACGT\n+\nII\n")                  # quality shorter than sequence
    with open(p) as fh, pytest.raises(ValueError):
        list(fqtool.records(fh))


def test_stats_command(tmp_path, capsys):
    p = tmp_path / "x.fq"
    p.write_text(FQ)
    assert fqtool.main(["stats", str(p)]) == 0
    out = capsys.readouterr().out
    assert "reads\t2" in out and "mean_quality\t20.00" in out


def test_filter_keeps_only_good_reads(tmp_path):
    src, dst = tmp_path / "x.fq", tmp_path / "y.fq"
    src.write_text(FQ)
    assert fqtool.main(["filter", str(src), "-o", str(dst), "-q", "30", "-l", "4"]) == 0
    assert dst.read_text().startswith("@r1") and "@r2" not in dst.read_text()


def test_missing_file_gives_exit_status_1(tmp_path, capsys):
    assert fqtool.main(["stats", str(tmp_path / "nope.fq")]) == 1
    assert "error" in capsys.readouterr().err
$ python3 -m pytest -q test_fqtool.py 2>&1 | tail -6
.........                                                                [100%]
9 passed in 0.02s

A passing suite proves little until you have seen it fail. To check that these tests have teeth, we planted a plausible bug: the quality offset set to 64 (the old Illumina convention) instead of 33. A copy of the tool with that one change was run against the same tests:

$ rm -rf /tmp/bugdemo && mkdir /tmp/bugdemo && cp test_fqtool.py /tmp/bugdemo/ && sed 's/offset=33/offset=64/' fqtool.py > /tmp/bugdemo/fqtool.py
$ cd /tmp/bugdemo && python3 -m pytest -q 2>&1 | grep -E '^FAILED|^[0-9]+ (passed|failed)' | cut -c1-140
FAILED test_fqtool.py::test_mean_quality[IIII-40.0] - assert 9.0 == 40.0 ± 4....
FAILED test_fqtool.py::test_mean_quality[!!!!-0.0] - assert -31.0 == 0.0 ± 1....
FAILED test_fqtool.py::test_mean_quality[5555-20.0] - assert -11.0 == 20.0 ± ...
FAILED test_fqtool.py::test_mean_quality[+5?I-25.0] - assert -6.0 == 25.0 ± 2...
FAILED test_fqtool.py::test_stats_command - AssertionError: assert ('reads\t2...
FAILED test_fqtool.py::test_filter_keeps_only_good_reads - AssertionError: as...
6 failed, 3 passed in 0.06s

Six of the nine tests fail, and the messages show the symptom (a mean quality of 9.0 where 40.0 was expected, and negative qualities). A suite that stays green after a change of that size would not be worth having. Writing the test before the code (test-driven development) and running the whole suite after every change are the habits that make a tool safe to modify (Youens-Clark, Ch. 4; Wilson et al. 2017).

Efficiency without tricks

Three rules cover most speed problems. Vectorise: operate on whole columns (c.sum(axis=0)), not row by row in a Python loop. Stream: process a large file one record at a time rather than loading it. Measure before optimising: time a run, and if a step is slow, check whether a compiled tool (, ) already does it. Then think about parallelism.

Measure before you optimise, and measure with the right tool. timeit runs a function several times and reports the average. The example computes the mean quality of the 6,000 reads in two ways: a Python loop over the characters, and NumPy, which treats the whole quality block as one array of bytes and does the arithmetic in compiled code:

import sys, timeit
import numpy as np
sys.path.insert(0, ".")
from fqtool import records
with open("../data/S1.fq") as fh:
    quals = [q for _, _, q in records(fh)]
def pure():
    return sum(sum(ord(c) - 33 for c in q) / len(q) for q in quals) / len(quals)
arr = np.frombuffer("".join(quals).encode(), dtype=np.uint8).reshape(len(quals), -1)
def vector():
    return float((arr.astype(np.float64) - 33).mean())
t1 = timeit.timeit(pure, number=3) / 3
t2 = timeit.timeit(vector, number=3) / 3
print(f"pure Python {t1 * 1000:.0f} ms, NumPy {t2 * 1000:.1f} ms ({t1 / t2:.0f} times faster); same answer: {abs(pure() - vector()) < 1e-9}")

# output
pure Python 33 ms, NumPy 3.1 ms (11 times faster); same answer: True

NumPy was about 11 times faster here (33 ms against 3.1 ms; your numbers will differ with the machine) and gave the same answer. The condition is that the data have a regular shape: all reads here have 100 bases, so the block reshapes into 6,000 rows and 100 columns. For reads of unequal length you would pad or work per read. The lesson is general: a loop over millions of items in Python is the first place to look when a program is slow, and a vectorised expression is often the cure.

Check your understanding

Your group means look plausible, but the treated group is exactly the control group of the previous experiment. What did you not check?

That the sample order in the metadata equals that of the count table. Assert list(counts.columns) == list(meta.index), or reindex the metadata by the count columns.

A merge turns 1,000 rows into 1,350. What happened and how do you find it?

A key appears more than once on at least one side, so rows were multiplied. Check df["key"].duplicated().sum() on both tables and use validate="one_to_one".

What error probability does Phred Q30 mean?

1 in 1,000 (10−30/10), an accuracy of 99.9%.

Why write tests for a ten-line tool?

They record what the tool is supposed to do, catch a regression when you change it, and let a reader trust the numbers without redoing the work.

Core skills · Programming and data

R from the ground up: vectors, data frames, functions and statistics

R is the language of statistics and of Bioconductor, the collection of packages in which most expression, variant and microbiome methods were first published. It thinks in vectors and tables, which suits biological data. This chapter teaches the language with the same small count table used in the Python chapters, so that you can compare the two. All code was run in R 4.6 and the printed output is real.

Vectors: the basic object

Everything in R is a vector or built from vectors. Operations work on every element at once, and elements can have names:

> x <- c(10, 0, 5); names(x) <- c("s1", "s2", "s3")
> x[x > 0]
s1 s3
10  5
> x / sum(x)
       s1        s2        s3
0.6666667 0.0000000 0.3333333
> 1:6 + c(10, 20)             # the shorter vector is recycled
[1] 11 22 13 24 15 26

Indexing starts at 1 (unlike Python), and x[a:b] includes both ends. Recycling is convenient and dangerous: if the lengths do not divide evenly, R warns, but when they do, it silently repeats. NA is a missing value, and it is contagious: mean(c(1, NA, 3)) is NA; mean(c(1, NA, 3), na.rm = TRUE) is 2. Decide what a missing value means before you remove it.

Factors and the reference level

A factor is a categorical variable with a fixed set of levels. Statistical models use the first level as the reference and report every other level against it, so the order decides the direction of every coefficient and fold change. By default levels are alphabetical:

> g <- factor(c("treated", "control", "treated")); levels(g)
[1] "control" "treated"
> levels(relevel(g, ref = "treated"))
[1] "treated" "control"

With control as the reference a positive coefficient means "higher in treated"; relevel it, and the same data give the opposite sign. If your fold changes have the sign opposite to the paper, check this first.

Data frames

A data frame is a table whose columns are vectors of equal length. Read, look at its structure, check it:

> counts <- read.delim("counts.tsv", row.names = 1, check.names = FALSE)
> meta   <- read.delim("meta.tsv", row.names = 1)
> str(meta)
'data.frame':   3 obs. of  1 variable:
 $ group: chr  "control" "treated" "treated"
> colSums(counts)
 s1  s2  s3
210 180 218
> rowSums(counts > 0)
geneA geneB geneC
    2     3     1
> all(colnames(counts) == rownames(meta))
[1] TRUE

As in Python, the last line is the one to keep: samples in the count table and in the metadata must be the same and in the same order. Relative abundance, filtering and subsetting use the same ideas as before:

> rel <- sweep(counts, 2, colSums(counts), "/") * 100   # percent of each sample
> round(rel, 1)
        s1  s2   s3
geneA  4.8   0  2.3
geneB 95.2 100 96.3
geneC  0.0   0  1.4
> counts[rowSums(counts) > 5, ]          # genes with more than 5 reads in total
       s1  s2  s3
geneA  10   0   5
geneB 200 180 210
> subset(meta, group == "treated")
     group
s2 treated
s3 treated
> tapply(as.numeric(counts["geneA", ]), meta$group, mean)
control treated
   10.0     2.5

apply(counts, 1, f) applies a function to every row (2 for columns), and sapply(counts, sum) to every column of a data frame. They return vectors, so they are the natural way to compute a statistic per gene or per sample. The tidyverse packages (dplyr, tidyr, readr) offer a different, very readable grammar (filter, mutate, group_by, summarise) built on the tidy-data idea that every variable is a column and every observation a row (Wickham, 2014). Base R is enough for everything in this chapter and is always installed; learn tidyverse once you are comfortable with it.

Functions and control flow

gc <- function(s) {
  b <- strsplit(toupper(s), "")[[1]]
  mean(b %in% c("G", "C"))
}
round(gc("ATGCGTACGTTAGC"), 3)      # 0.5

for (i in 1:3) if (i %% 2 == 1) cat("odd", i, "\n")   # odd 1 / odd 3

A function returns its last value. Give arguments sensible defaults, check inputs with stopifnot(), and avoid loops over rows when a vectorised operation exists: colSums(counts) is both shorter and much faster than a loop.

Statistics you must be able to read

Compare two groups with a t test (assumes roughly normal values) or the Wilcoxon rank-sum test (compares ranks; use it for skewed data or small groups). With eight values per group drawn at random from two distributions whose means differ by 1, one run gave:

> set.seed(1); a <- rnorm(8, 10, 1); b <- rnorm(8, 11, 1)
> round(t.test(a, b)$p.value, 4)
[1] 0.0938
> round(wilcox.test(a, b)$p.value, 4)
[1] 0.1049

A true difference of one standard deviation was not significant with eight samples per group: the test lacks power. That is the lesson of every small study: a result that is not significant is not evidence that there is no difference. Plan the sample size before the experiment, not after.

When you test thousands of genes or taxa at once, about 5% of the true nulls will have p < 0.05 by chance. Correct for it. p.adjust(p, "BH") applies the Benjamini–Hochberg procedure, which controls the false-discovery rate (Benjamini and Hochberg, 1995):

> p <- c(0.001, 0.02, 0.04, 0.3)
> round(p.adjust(p, "BH"), 4)
[1] 0.0040 0.0400 0.0533 0.3000

The third gene, at p = 0.04, is no longer below 0.05 after the correction (adjusted 0.0533). Report the adjusted value and say which correction you used.

Check your understanding

You compare "disease" with "healthy" and every log fold change has the opposite sign of the paper. What is the likely cause?

The reference level of the factor: levels are alphabetical, so "disease" came first and became the reference. Use relevel(x, ref = "healthy").

With 8 samples per group a t test gives p = 0.09. Does this show there is no difference?

No. It shows that this study could not detect a difference of this size. Power depends on the sample size, the variability and the effect.

p = 0.04 for gene C among 20,000 tests. Is it a finding?

Not without correction: at a 5% threshold, about 1,000 null genes are expected below it by chance. Use the adjusted value.

What does 1:6 + c(10, 20) return, and why is it risky?

11 22 13 24 15 26: the shorter vector is recycled. If the lengths were mismatched by mistake, you could get a wrong answer without an error.

Core skills · Programming and data

R for omics: ordination, PERMANOVA, ggplot2 and the Bioconductor workflow

This chapter applies R to a typical community or expression dataset: a samples-by-features table, a grouping, and the questions "are the groups different overall?", "which features differ?" and "how do I show it?". The code that was run here uses base R, vegan and ggplot2; the Bioconductor workflows are shown as a map, and the parts that were not run are marked.

A community analysis in 20 lines

The data are simulated so that the truth is known: 12 samples, 8 taxa, two groups of 6, and three taxa made three times more abundant in the treated group. Samples are rows here, as vegan expects.

library(vegan); library(ggplot2)
set.seed(7)
n   <- 12
grp <- factor(rep(c("control", "treated"), each = n / 2))
otu <- matrix(rpois(n * 8, lambda = 20), nrow = n)
otu[grp == "treated", 1:3] <- otu[grp == "treated", 1:3] * 3
rownames(otu) <- paste0("S", 1:n); colnames(otu) <- paste0("taxon", 1:8)

rel <- otu / rowSums(otu)               # relative abundance
d   <- vegdist(rel, "bray")             # Bray-Curtis dissimilarity
adonis2(d ~ grp, permutations = 999)    # PERMANOVA

The output of the PERMANOVA in this run:

         Df SumOfSqs      R2      F Pr(>F)
Model     1  0.18742 0.79231 38.148  0.002 **
Residual 10  0.04913 0.20769
Total    11  0.23655 1.00000

Read the table. R2 is the share of the total variation in the distances explained by the grouping: here 79%, which is large because the effect was planted and strong. F compares the variation between groups with the variation within them, and Pr(>F) is the share of 999 random relabelings of the samples that gave an F at least this large; with 999 permutations the smallest possible p is 0.001. In real microbiome data a single factor usually explains much less, often 2–15%.

PERMANOVA assumes the groups have similar spread. A significant result can come from one group being more variable, not from a different centre. Test it with betadisper(d, grp) and permutest(), and look at the ordination before believing the p value.

Ordination: see the distances

Principal coordinates analysis (PCoA) places the samples in two dimensions so that distances are preserved as well as possible. cmdscale(d, k = 2) does it in base R; ggplot2 draws it:

pc <- cmdscale(d, k = 2)
df <- data.frame(PC1 = pc[, 1], PC2 = pc[, 2], group = grp)
p  <- ggplot(df, aes(PC1, PC2, colour = group)) + geom_point(size = 3) + theme_minimal()
ggsave("pcoa.pdf", p, width = 5, height = 4)

Save figures as PDF or SVG (vector graphics) for publication, so that they stay sharp at any size, and as PNG for the web. In the R installation used for this chapter the default PNG device failed with a "graphics API version mismatch" error, while the PDF device worked; if you meet the same, install the ragg package and use ggsave(..., device = ragg::agg_png), or save a PDF. Colour choices matter: do not use red and green together, and label the axes with units. Show the data points, not only a mean, so that readers see the spread.

Alpha diversity

Diversity within a sample is summarised by richness (how many taxa), the Shannon index (richness and evenness) and others. diversity(otu, "shannon") gives the Shannon index per sample; tapply(sh, grp, median) summarises by group. In this run the medians were 2.063 (control) and 1.919 (treated): the treatment made three taxa more abundant, so the community became less even and its Shannon index fell, even though no taxon was lost. Diversity indices answer different questions, so choose them for the question, report more than one, and do not compare richness between samples with very different sequencing depths without rarefying or standardising by coverage.

Differential abundance: which features differ

The overall test says that groups differ; the next question is which features differ. The methods fall into two families. For count data from RNA-seq, models for counts with a dispersion estimate:  and  (Love et al., 2014). For compositional microbiome data, methods that respect the fact that abundances are proportions, such as ANCOM-BC, ALDEx2 or a model on centred log-ratios. These methods often disagree on microbiome data, so use more than one and report what several agree on. The shape of a  analysis, shown here but not run in this chapter because the package is not installed on the machine where the examples were tested:

# not run here: requires Bioconductor's 
library()
dds <- DESeqDataSetFromMatrix(countData = counts,   # genes x samples, raw integer counts
                              colData   = meta,     # samples x variables, same order
                              design    = ~ group)
dds$group <- relevel(dds$group, ref = "control")    # the reference level decides the sign
dds <- DESeq(dds)
res <- results(dds, alpha = 0.05)                   # adjusted p values (Benjamini-Hochberg)
summary(res)

Three rules for such a workflow. Give the model raw counts, not normalised values or percentages: it normalises internally and needs the counts to estimate noise. Put the factor of interest last in the design, and any batch or covariate before it. And look at the results before trusting them: a plot of log fold change against mean abundance (an MA plot), the distribution of p values (flat with a peak near zero is healthy), and a PCA of the samples to find outliers and batch effects.

The Bioconductor map

Bioconductor is a repository of R packages for genomic data that share common data structures, so that packages can work together (Huber et al., 2015). Install it with BiocManager::install(), and record the version, since packages and R versions move together. The structures to know:

StructureHoldsUsed for
SummarizedExperimenta count matrix, row annotations (genes) and column annotations (samples) kept togetherRNA-seq and most omics; subsetting samples subsets the annotations too, so they cannot drift apart
GRangesgenomic intervals with strand and metadatapeaks, genes, variants; overlaps and distances
DNAStringSet (Biostrings)many DNA sequencessequence work, pattern matching, translation
phyloseqan OTU table, taxonomy, sample data and a treemicrobiome analysis (McMurdie and Holmes, 2013)

The value of these structures is the same as that of the sample-match assertion earlier: the data and their annotations travel together, so that a subset of the samples is a subset of the metadata as well.

Writing it up: scripts and reports

Put the analysis in a script or a Quarto or R Markdown document that reads the raw files and produces every figure and table, so that the whole analysis reruns with one command. Fix the random seed (set.seed()) wherever randomness is used (permutations, subsampling), and finish the document with sessionInfo(), which records the versions of R and of every package. Save the figures as files, not as screenshots. A reviewer, or you in a year, can then check every number (Sandve et al., 2013).

Check your understanding

PERMANOVA gives p = 0.001 and R² = 0.08. How do you describe the result?

The groups differ in their community composition, and the grouping explains about 8% of the variation between samples: a modest but reliable effect. Check the dispersion before concluding that the centres differ.

Why must  receive raw counts?

It models counts with a noise (dispersion) estimate that depends on the count scale, and normalises internally by size factors. Percentages or transformed values break these assumptions.

The ordination shows two clouds, but betadisper is significant. What does that mean?

The groups differ in their spread, so the PERMANOVA result may reflect different variability and not different centres. Report both, and interpret the figure with that in mind.

What is the advantage of a SummarizedExperiment over a bare matrix?

The counts, the gene annotations and the sample table are one object: subsetting samples subsets the sample table too, so they cannot become misaligned.

Core skills · Programming and data

Biopython in depth: sequences, records, features, alignments, motifs and trees

Biopython is the toolbox for the biological objects of Python: a sequence that knows its alphabet and its genetic code, a record that carries annotations and features, parsers for every common file format, pairwise aligners, motif models and phylogenetic trees. This chapter goes through it object by object. Every snippet was run, mostly on the real files of the MicroRes demonstration data (377 representative 16S sequences of 253 bases and their tree), so the numbers quoted are real.

Seq: a sequence with biology attached

A Seq behaves like a Python string (it has a length, slices, find, count) but also knows what it is: it can be complemented, reverse-complemented, transcribed and translated. Two properties matter in daily work. A Seq is immutable, like a string; use MutableSeq if you must change it in place. And its methods do not guess: a character outside the alphabet is an error or an ambiguity code, never silently changed.

from Bio.Seq import Seq
s = Seq("ATGGCCATTGTAATGGGCCGCTGA")
s.reverse_complement()        # Seq('TCAGCGGCCCATTACAATGGCCAT')
s.translate()                 # Seq('MAIVMGR*')
s.translate(to_stop=True)     # Seq('MAIVMGR')
s.transcribe()                # DNA to RNA: T becomes U
Seq("ATGNNNCGT").translate()  # Seq('MXR')   NNN is an unknown codon, translated as X
Seq("ACGTRYN").reverse_complement()   # Seq('NRYACGT')  ambiguity codes are complemented correctly

Ambiguity codes (R = A or G, Y = C or T, N = any) are real: they appear in assemblies and in consensus sequences. A hand-made complement table that only knows A, C, G and T fails on them; Seq does not. The reverse complement matters because DNA is double-stranded: a gene can be on either strand, and a read from the other strand is the reverse complement of the gene.

Translation, done properly

Translation depends on four things that must be right. The frame: a coding sequence starts at a start codon and is read in triplets; starting one base off gives an unrelated protein. The genetic code table: most bacteria use table 11 (the standard code, with alternative start codons such as TTG and CTG); Mycoplasma reads TGA as tryptophan (table 4); mitochondria use their own tables. The stop codon: use to_stop=True to cut the protein at the first stop. And the length: a length that is not a multiple of three raises a warning, because a frameshift or a truncated record is likely.

from Bio.Data import CodonTable
t = CodonTable.unambiguous_dna_by_id[11]
t.start_codons[:3]       # ['TTG', 'CTG', 'ATT']
t.stop_codons            # ['TAA', 'TAG', 'TGA']
Seq("ATGGCCTGA").translate(table=4)   # Seq('MAW'): in table 4, TGA is tryptophan, not a stop

The last line is a classic trap: the same DNA gives a 2-residue protein with a stop under table 11 and a 3-residue protein under table 4. Always state the table in a methods section. An internal stop codon (*) in a gene that should be intact means one of: a pseudogene, a sequencing or assembly error (a frameshift), the wrong frame, or the wrong table.

SeqRecord and SeqFeature: a sequence with its annotation

A SeqRecord wraps a Seq with an identifier (id), a name, a description, free-form annotations, per-letter annotations (the quality scores of a FASTQ read) and a list of features. A SeqFeature is one annotated region: a type (CDS, gene, rRNA), a FeatureLocation with start, end and strand, and qualifiers (product, locus_tag). This is exactly what a  file holds, and what a genome annotation produces.

from Bio.SeqRecord import SeqRecord
from Bio.SeqFeature import SeqFeature, FeatureLocation
r = SeqRecord(Seq("ATGGCCATTGTAATGGGCCGCTGAAAGGGTGCCCGATAG"), id="demo1", description="toy gene")
r.features.append(SeqFeature(FeatureLocation(0, 24, strand=1), type="CDS", qualifiers={"product": "toy protein"}))
cds = r.features[0].extract(r.seq)      # Seq('ATGGCCATTGTAATGGGCCGCTGA')
cds.translate(table=11, to_stop=True)   # Seq('MAIVMGR')

f = SeqFeature(FeatureLocation(10, 31, strand=-1), type="gene")
f.extract(r.seq)                        # Seq('CACCCTTTCAGCGGCCCATTA'): the reverse complement of bases 10 to 30

Coordinates are the main source of errors. Biopython uses Python's convention: 0-based and half-open, so FeatureLocation(0, 24) covers the first 24 bases, positions 1 to 24 in the 1-based, closed convention of  and GFF. When you read coordinates from a GFF or a  table, convert them: a GFF feature at 101 to 200 is FeatureLocation(100, 200). On the minus strand, extract returns the reverse complement for you, which is why a gene on the minus strand is read correctly without extra code. Writing the record out as  (SeqIO.write(r, handle, ""), after setting r.annotations["molecule_type"] = "DNA") produced a file starting LOCUS demo1 39 bp DNA.

SeqIO: reading and writing every common format

SeqIO.parse(file, format) returns an iterator of records: one record in memory at a time, so a file of any size can be processed. SeqIO.read expects exactly one record. SeqIO.write(records, file, format) writes, and SeqIO.convert converts between formats. The formats you will meet: fasta, fastq (Sanger/Illumina 1.8+), , embl, gff (through a helper library), stockholm and clustal for alignments.

On the 377 representative sequences of the demonstration data:

from Bio import SeqIO
from Bio.SeqUtils import gc_fraction
recs = list(SeqIO.parse("sequences.fasta", "fasta"))
len(recs), recs[0].id, len(recs[0])            # (377, 'ASV0001', 253)
lens = [len(r) for r in recs]
min(lens), max(lens)                           # (253, 253): every sequence has the same length
round(sum(gc_fraction(r.seq) for r in recs) / len(recs), 3)   # 0.519: mean GC content

Three techniques make large files manageable:

  • Stream and filter: SeqIO.write((r for r in SeqIO.parse(f, "fasta") if len(r) >= 500), out, "fasta") never holds the file in memory.
  • Index: idx = SeqIO.index("sequences.fasta", "fasta") gives random access by identifier without loading the sequences: len(idx) was 377 and idx["ASV0002"] returns that record. Use it when you need to pull a few thousand sequences out of a genome-sized file. Close it with idx.close().
  • Compressed input: open the file with gzip.open(path, "rt") and pass the handle to parse.

For FASTQ, each record carries per-base quality in rec.letter_annotations["phred_quality"], a list of integers. A Phred quality Q is an error probability of 10−Q/10. Trimming by quality, filtering by mean quality and counting bases below Q20 are all list operations on that field; for large datasets do this in a compiled tool (, ) and use Biopython for the analysis that follows.

Sequence statistics

Bio.SeqUtils has the common calculations: gc_fraction(seq), molecular_weight(seq) (for the 14-base example, 4358.8), seq1 and seq3 to convert between one- and three-letter amino-acid codes, and MeltingTemp.Tm_NN for the melting temperature of a primer by the nearest-neighbour method (40.8 °C for ATGCGTACGTTAGC with default salt conditions). Treat a computed Tm as a guide: the value depends on salt and primer concentration, which you must set to your reaction (Na=, dnac1=).

Pairwise alignment

A pairwise alignment finds the arrangement of two sequences that maximises a score (see the alignment chapter of this handbook). PairwiseAligner does global (Needleman–Wunsch) and local (Smith–Waterman) alignment with an affine gap penalty, and for proteins accepts a substitution matrix.

from Bio.Align import PairwiseAligner, substitution_matrices
al = PairwiseAligner(mode="global", match_score=1, mismatch_score=-1, open_gap_score=-2, extend_gap_score=-0.5)
a = al.align("GATTACA", "GCATGCA")[0]
a.score        # 1.0
print(a)
# target            0 GATTACA 7
#                   0 |..|.|| 7
# query             0 GCATGCA 7

bl = PairwiseAligner(substitution_matrix=substitution_matrices.load("BLOSUM62"), open_gap_score=-10, extend_gap_score=-0.5)
bl.align("HEAGAWGHEE", "PAWHEAE")[0].score     # 4.0

Read the output: bars mark identical letters, dots mismatches, and the numbers are positions. The scores are only comparable under the same scoring scheme; a score of 4.0 under BLOSUM62 is not "better" than 1.0 under a match/mismatch scheme. For two protein sequences always use a substitution matrix (BLOSUM62 for moderately distant, BLOSUM80 for close, BLOSUM45 for distant sequences), not match/mismatch scores. For many sequences use a dedicated multiple-alignment program (, MUSCLE) and read the result with AlignIO; Biopython aligns pairs, it is not a multiple-alignment engine.

Motifs

A motif such as a promoter box or a binding site is a family of similar short sequences. Bio.motifs counts the letters at each position, builds a position weight matrix (PWM) and scores new sequences against it.

from Bio import motifs
m = motifs.create([Seq("TATAAT"), Seq("TATGAT"), Seq("TACAAT"), Seq("TATAAT")])
print(m.counts)
#         0      1      2      3      4      5
# A:   0.00   4.00   0.00   3.00   4.00   0.00
# C:   0.00   0.00   1.00   0.00   0.00   0.00
# G:   0.00   0.00   0.00   1.00   0.00   0.00
# T:   4.00   0.00   3.00   0.00   0.00   4.00
m.consensus                                          # Seq('TATAAT')
pwm = m.counts.normalize(pseudocounts=0.5)
round(pwm.log_odds()["T"][0], 2)                     # 1.58 (log2 odds of T at position 0)

Two cautions. Counts from four sites are a very poor estimate: the pseudocount keeps a letter that was never seen from getting probability zero, and with so few examples every score is uncertain. And a PWM scan of a genome finds many matches by chance; a threshold must be justified against the background (pwm.pssm and its distribution), and a hit is a candidate, not a binding site.

Trees

Bio.Phylo reads Newick, NEXUS and PhyloXML trees and offers traversal, pruning, rooting and drawing. On the tree that comes with the MicroRes demonstration data:

from Bio import Phylo
tree = Phylo.read("tree.nwk", "newick")
tree.count_terminals()                    # 377: one leaf per sequence
round(tree.total_branch_length(), 3)      # 19.465
sorted(t.name for t in tree.get_terminals())[:3]   # ['ASV0001', 'ASV0002', 'ASV0003']
tree.root_at_midpoint()                   # root the tree
Phylo.draw_ascii(subtree)                 # a text drawing, useful on a server

Typical tasks: tree.common_ancestor("a", "b") gives the smallest clade that contains two leaves; tree.distance("a", "b") the path length between them; tree.prune(leaf) removes a leaf; Phylo.write(tree, "out.nwk", "newick") saves it. A tree is a hypothesis built from an alignment under a model; do not read branch lengths between distantly related leaves as time without a calibrated model.

Fetching and parsing from outside

Bio.Entrez talks to 's E-utilities: search (esearch), fetch (efetch), link (elink).  asks every user to give an e-mail address and to respect a limit of three requests per second without an API key. A complete, live example, with its output, is in the section on  records below.

For , run the program on your machine or a server with -outfmt 6 (tab-separated) and read the table with pandas (columns: query, subject, percent identity, alignment length, mismatches, gap opens, query start and end, subject start and end, e-value, bit score). Bio.SearchIO parses  XML and many other search formats into query, hit and HSP objects when you need the alignments themselves (an example follows). Calling 's web  from a script is slow and rate-limited; it is not a way to search thousands of sequences.

The same checks in Python: a file, its qualities, and a cross-check of two tools

This section and those that follow continue with the files of chapter P12: the phiX174 genome, the simulated reads S1.fq, the gene table, the  hits. The first question is the one asked in the shell: how long is the genome, what is its GC content, and how good are the reads? gc_fraction is the current name of the function that older books call GC (it returns a fraction between 0 and 1, not a percentage).

from Bio import SeqIO
from Bio.SeqUtils import gc_fraction

rec = SeqIO.read("data/phix174.fa", "fasta")
print(rec.id, len(rec.seq), f"GC {100 * gc_fraction(rec.seq):.2f}%")
print(rec.description)

# output
gi|9626372|ref|NC_001422.1| 5386 GC 44.71%
gi|9626372|ref|NC_001422.1| Coliphage phiX174, complete genome

For a FASTQ file, SeqIO decodes the quality string into a list of integers in letter_annotations["phred_quality"], with the Sanger offset of 33. That makes the arithmetic of chapter P12 a few lines of Python, and gives a way to settle a question left open there: which quantity does a tool report as "average quality"? The program below computes the mean of the read means (the number our awk gave), the share of bases at Q30 or better, and the quality that corresponds to the mean error probability.

from math import log10
from Bio import SeqIO

n = q_sum = bases = q30 = 0
perr = 0.0
for r in SeqIO.parse("data/S1.fq", "fastq"):
    q = r.letter_annotations["phred_quality"]          # decoded with the Sanger offset of 33
    n += 1
    q_sum += sum(q) / len(q)
    bases += len(q)
    q30 += sum(x >= 30 for x in q)
    perr += sum(10 ** (-x / 10) for x in q)
print(f"{n} reads; mean of the read means {q_sum / n:.2f}")
print(f"bases at Q30 or better: {100 * q30 / bases:.2f}%")
print(f"quality from the mean error probability: {-10 * log10(perr / bases):.2f}")

# output
6000 reads; mean of the read means 33.55
bases at Q30 or better: 80.87%
quality from the mean error probability: 31.24

The first value, 33.55, is what awk printed. The last, 31.24, is exactly what  stats -a reports as AvgQual. The two differ because a quality score is a logarithm: a few bad bases (Q10, error 0.1) raise the mean error probability far more than they lower the mean score. The error-probability average is the better summary of how many errors to expect, and it is the one to quote. Seeing two tools disagree and then reproducing both numbers is the quickest way to learn what each of them means.

From a GFF line to a protein: the strand, the table and three checks

A gene table becomes proteins in four steps: convert the 1-based GFF coordinates to Biopython's 0-based half-open ones, build a SeqFeature with a FeatureLocation that carries the strand, extract the sequence (which reverse-complements a minus-strand feature), and translate with the right genetic code. The code below does it for the seven genes of chapter P12 and prints three checks for each: the length in bases, whether the protein ends with a stop, and whether there is a stop inside.

from Bio import SeqIO
from Bio.Seq import Seq
from Bio.SeqFeature import SeqFeature, FeatureLocation

genome = SeqIO.read("data/phix174.fa", "fasta").seq
for line in open("data/genes.gff"):
    if line.startswith("#"):
        continue
    f = line.rstrip("\n").split("\t")
    start, end, strand = int(f[3]), int(f[4]), 1 if f[6] == "+" else -1
    gid = f[8].split(";")[0].replace("ID=", "")
    feat = SeqFeature(FeatureLocation(start - 1, end, strand=strand), type="CDS")   # GFF is 1-based, Biopython 0-based
    dna = feat.extract(genome)
    prot = dna.translate(table=11)
    stop = prot.endswith("*")
    print(gid, f"{len(dna):5d} bp", f"{len(prot) - stop:4d} aa", "ends with stop" if stop else "NO final stop", "internal stops:", str(prot[:-1]).count("*"), str(prot[:8]))

# output
1_1   171 bp   56 aa ends with stop internal stops: 0 MSRKIILI
1_2   459 bp  152 aa ends with stop internal stops: 0 MSQVTEQS
1_3   117 bp   38 aa ends with stop internal stops: 0 MSKGKKRS
1_4  1284 bp  427 aa ends with stop internal stops: 0 MSNIQTGA
1_5   528 bp  175 aa ends with stop internal stops: 0 MFQTFISR
1_6   987 bp  328 aa ends with stop internal stops: 0 MFGAIAGG
1_7  1404 bp  468 aa NO final stop internal stops: 0 MVRSYYPS

Six genes end with a stop and have no internal stop, as a complete gene should. The seventh, 1_7, has no final stop. Chapter P12 showed the same gene with and put it down to a gene that is cut by the end of the linear sequence. The next section, with the  record, shows that this reading was right.

Finding ORFs yourself, and why a prediction is better than a rule

An open reading frame (ORF) is a stretch with no stop codon in one of the six reading frames (three per strand). The simplest gene finder takes every long ORF and starts it at the first methionine after the previous stop. It needs ten lines of code, and it shows both how far a rule goes and where it stops.

from Bio import SeqIO

genome = SeqIO.read("data/phix174.fa", "fasta").seq
found = []
for strand, s in ((1, genome), (-1, genome.reverse_complement())):
    for frame in range(3):
        prot = str(s[frame:len(s) - (len(s) - frame) % 3].translate(table=11))
        pos = 0
        for stretch in prot.split("*"):
            m = stretch.find("M")                        # an ORF starts at the first Met after a stop
            if m >= 0 and len(stretch) - m >= 100:
                aa_start = pos + m
                found.append((strand, frame, frame + 3 * aa_start, len(stretch) - m))
            pos += len(stretch) + 1
print(len(found), "ORFs of at least 100 amino acids, six frames")
for strand, frame, nt, aa in sorted(found, key=lambda x: -x[3])[:5]:
    where = nt + 1 if strand == 1 else len(genome) - nt
    print(f"strand {strand:+d} frame {frame}  starts at {where}  {aa} aa")

# output
10 ORFs of at least 100 amino acids, six frames
strand +1 frame 2  starts at 3954  477 aa
strand +1 frame 1  starts at 1001  427 aa
strand +1 frame 2  starts at 2931  328 aa
strand +1 frame 0  starts at 3076  202 aa
strand -1 frame 0  starts at 1651  185 aa

The longest ORFs coincide with real genes (the ORFs at 1,001 and 2,931 are genes 1_4 and 1_6), but the rule starts the one at 3,954 at the first methionine after the stop, 27 bases before the start that  chose (3,981). A rule cannot tell which of several in-frame ATGs is the true start; a gene finder uses the ribosome-binding site and the codon statistics of the organism to decide. The rule also reports ORFs that are only chance stretches without stops, such as the one on the minus strand. Use the rule to understand; use a predictor for results, and a curated annotation when there is one.

Protein properties with ProteinAnalysis

Bio.SeqUtils.ProtParam.ProteinAnalysis computes properties from a protein sequence: molecular weight, isoelectric point (the pH at which the protein carries no net charge), the GRAVY score (the mean hydropathy of Kyte and Doolittle: positive for hydrophobic, negative for hydrophilic) and the instability index (Guruprasad et al.: above 40 predicts an unstable protein in a test tube). The values are predictions from sequence, useful for choosing a purification pH or spotting a membrane protein, not measurements.

from Bio import SeqIO
from Bio.SeqFeature import SeqFeature, FeatureLocation
from Bio.SeqUtils.ProtParam import ProteinAnalysis

genome = SeqIO.read("data/phix174.fa", "fasta").seq
print("gene    aa      MW(kDa)  pI    GRAVY  instability")
for line in open("data/genes.gff"):
    if line.startswith("#"):
        continue
    f = line.split("\t")
    gid = f[8].split(";")[0].replace("ID=", "")
    cds = SeqFeature(FeatureLocation(int(f[3]) - 1, int(f[4]), strand=1), type="CDS").extract(genome)
    prot = str(cds.translate(table=11)).rstrip("*").replace("*", "")
    pa = ProteinAnalysis(prot)
    print(f"{gid:7s} {len(prot):4d}  {pa.molecular_weight() / 1000:7.1f}  {pa.isoelectric_point():5.2f} {pa.gravy():6.2f}  {pa.instability_index():6.1f}")

# output
gene    aa      MW(kDa)  pI    GRAVY  instability
1_1       56      6.4   7.89   0.07    53.1
1_2      152     16.9   4.91   0.11    50.3
1_3       38      4.2  12.00  -1.40    65.6
1_4      427     48.5   6.39  -0.42    41.4
1_5      175     19.0   6.24   0.14    29.9
1_6      328     34.4   5.14  -0.43    14.2
1_7      468     53.8   9.14  -0.45    37.8

Gene 1_3 is only 38 residues, has an isoelectric point of 12.00 and the most negative GRAVY: a short, very basic, hydrophilic protein, which fits a DNA-binding protein (the  record names it a DNA condensation protein). The instability index is not reliable for very short sequences; read it for the larger proteins only.

Restriction enzymes

Bio.Restriction holds the enzymes of the REBASE database as objects. enzyme.search(seq) returns the cut positions, and Analysis answers questions about a whole set: which enzymes cut once, which not at all, for a linear or a circular molecule. For cloning, the enzymes that cut a vector exactly once are the interesting ones.

from Bio import SeqIO
from Bio.Restriction import AllEnzymes, Analysis, CommOnly, EcoRI, PstI

genome = SeqIO.read("data/phix174.fa", "fasta").seq
print("EcoRI sites:", EcoRI.search(genome), "| PstI sites:", PstI.search(genome))
an = Analysis(CommOnly, genome, linear=False)              # commercially available enzymes, circular genome
once = sorted(str(e) for e, cuts in an.with_N_sites(1).items())
print(len(once), "commercial enzymes cut the circular genome exactly once, for example:", ", ".join(once[:8]))
print(len(an.without_site()), "commercial enzymes do not cut it at all")

# output
EcoRI sites: [] | PstI sites: []
81 commercial enzymes cut the circular genome exactly once, for example: AanI, AasI, AatII, Acc16I, AccBSI, AdeI, AhdI, AleI
235 commercial enzymes do not cut it at all

This agrees with chapter P12, where grep found no EcoRI, BamHI, HindIII or PstI sites in this genome. Eighty-one commercially available enzymes cut the circular genome once, and 235 do not cut it at all. The two methods are independent, and their agreement is the check.

 results as objects: SearchIO

The tabular  output of P12 can be read as a table with pandas. Bio.SearchIO reads it (format blast-tab), the XML output (blast-xml) and many other search formats into three levels: a query result, its hits, and for each hit its high-scoring pairs (HSPs), the actual local alignments. Use it when you need the alignment itself or the coordinates in a consistent, 0-based form: note that query_start is 0-based, so we add one to compare with the tabular output.

from Bio import SearchIO

for qres in SearchIO.parse("results/hits.tsv", "blast-tab"):
    best = qres[0][0]                                   # best hit, best HSP
    print(f"{qres.id:14s} -> {best.hit_id.split('|')[3]}  identity {best.ident_pct:.1f}% over {best.aln_span} bp  "
          f"query {best.query_start + 1}-{best.query_end}  subject {best.hit_start + 1}-{best.hit_end}  e={best.evalue:.1e}")

# output
q1_from_301    -> NC_001422.1  identity 97.3% over 150 bp  query 1-150  subject 301-450  e=6.6e-72
q2_from_1501   -> NC_001422.1  identity 97.3% over 150 bp  query 1-150  subject 1501-1650  e=6.6e-72
q3_from_2801   -> NC_001422.1  identity 98.0% over 150 bp  query 1-150  subject 2801-2950  e=1.4e-73
q4_from_3901   -> NC_001422.1  identity 97.3% over 150 bp  query 1-150  subject 3901-4050  e=6.6e-72
q5_from_4801   -> NC_001422.1  identity 98.0% over 150 bp  query 1-150  subject 4801-4950  e=1.4e-73

The identities, the aligned lengths and the subject coordinates agree with the table of chapter P12. XML output (-outfmt 5) carries the alignment text and is larger; tabular output with chosen columns is the right default for large searches.

 records, versions, and what an annotation really contains

Bio.Entrez sends requests to 's E-utilities. efetch returns a record in the format you ask for, and SeqIO parses it. The code below was run live: it downloads the  record of phiX174, counts its coding sequences and prints each with its location.  asks for a contact address with every request (use your own) and limits requests to three per second without an API key.

from Bio import Entrez, SeqIO

Entrez.email = "[email protected]"                       #  asks for a contact address; use your own
with Entrez.efetch(db="nuccore", id="NC_001422.1", rettype="gb", retmode="text") as h:
    rec = SeqIO.read(h, "")
cds = [f for f in rec.features if f.type == "CDS"]
print(rec.id, rec.annotations["organism"], len(rec.seq), "bases;", len(cds), "annotated CDS")
for f in cds:
    q = f.qualifiers
    print(f"  {q['locus_tag'][0]:>9s}  {str(f.location):<28s} {len(q['translation'][0]):4d} aa  {q['product'][0]}")
SeqIO.write(rec, "results/phix_ncbi.gb", "")

# output
NC_001422.1 Escherichia phage phiX174 5386 bases; 11 annotated CDS
  phiX174p01  join{[3980:5386](+), [0:136](+)}  513 aa  DNA replication initiation
  phiX174p02  join{[4496:5386](+), [0:136](+)}  341 aa  DNA replication initiation
  phiX174p03  join{[5074:5386](+), [0:51](+)}  120 aa  head morphogenesis
  phiX174p04  [50:221](+)                    56 aa  K
  phiX174p05  [132:393](+)                   86 aa  terminase
  phiX174p07  [389:848](+)                  152 aa  head morphogenesis
  phiX174p08  [567:843](+)                   91 aa  endolysin
  phiX174p09  [847:964](+)                   38 aa  DNA condensation
  phiX174p06  [1000:2284](+)                427 aa  major head protein
  phiX174p10  [2394:2922](+)                175 aa  major spike protein
  phiX174p11  [2930:3917](+)                328 aa  pilot protein for DNA ejection

Three things stand out. First, the record has eleven coding sequences, against the seven predicted in P12: the predictor, as P12 warned, reports one gene per region and misses the genes that lie inside others, such as phiX174p08, the 91-residue endolysin at 568 to 843, which lies entirely inside phiX174p07 at 390 to 848. Second, the first three genes have a compound location, join{[3980:5386](+), [0:136](+)}: gene A (513 amino acids) starts at base 3,981, runs to the end of the sequence and carries on from the start, because the genome is circular. This is exactly why  reported 1_7 as partial and why its protein had no stop: the stop lies on the other side of the origin. A feature extracted with feature.extract(seq) handles a join correctly; slicing by start and end does not. Third, the record has no gene qualifiers; the locus_tag and product carry the names. Read a record before you write code that assumes its fields.

The record also lets us test a statement from P12: the BBTools copy of the genome differs from the current  record.

from Bio import SeqIO
from Bio.Align import PairwiseAligner

local = SeqIO.read("data/phix174.fa", "fasta").seq
ncbi = SeqIO.read("results/phix_ncbi.gb", "").seq
diff = [i + 1 for i, (a, b) in enumerate(zip(local, ncbi)) if a != b]
print(len(local), len(ncbi), "bases; positions that differ:", diff)
for i in diff:
    print(f"  {i}: local {local[i - 1]},  {ncbi[i - 1]}")

# output
5386 5386 bases; positions that differ: [587, 833, 2731, 2793, 2811]
  587: local A,  G
  833: local A,  G
  2731: local G,  A
  2793: local T,  C
  2811: local T,  C

Five bases differ, so two things that carry the same accession are not the same sequence. Always record the source, the version and the date of a reference, and if a result depends on single bases (variant calling, primer design), use the sequence that your reads actually come from.

Several genomes: alignment, distance and a tree

Biopython reads and writes alignments with Bio.AlignIO (FASTA, Clustal, Stockholm, PHYLIP, Nexus, MAF and more) and treats an alignment as a table: aln[:, 100] is column 100, aln[0] is a row, len(aln) counts the sequences. It does not align sequences with more than two sequences itself; for that use , MUSCLE or Clustal Omega (chapter A2) and read the result. The example aligns the genomes of three related microviruses (phiX174, G4 and alpha3, downloaded from ), measures the share of identical columns, builds a distance matrix and a neighbour-joining tree.

import subprocess
from Bio import AlignIO, Entrez, SeqIO
from Bio.Phylo.TreeConstruction import DistanceCalculator, DistanceTreeConstructor
from Bio import Phylo

Entrez.email = "[email protected]"
ids = ["NC_001422.1", "NC_001420.2", "NC_001330.1"]       # phiX174, G4, alpha3: three related phages
with Entrez.efetch(db="nuccore", id=",".join(ids), rettype="fasta", retmode="text") as h:
    recs = list(SeqIO.parse(h, "fasta"))
SeqIO.write(recs, "results/microvirus.fa", "fasta")
out = subprocess.run(["", "--quiet", "results/microvirus.fa"], capture_output=True, text=True).stdout
open("results/microvirus.aln.fa", "w").write(out)
aln = AlignIO.read("results/microvirus.aln.fa", "fasta")
print(len(aln), "sequences,", aln.get_alignment_length(), "alignment columns;", [f"{r.id}:{len(str(r.seq).replace('-', ''))}" for r in aln])
ident = sum(1 for i in range(aln.get_alignment_length()) if len(set(aln[:, i])) == 1 and "-" not in aln[:, i])
print(f"columns identical in all three and gap-free: {ident} ({100 * ident / aln.get_alignment_length():.1f}%)")
dm = DistanceCalculator("identity").get_distance(aln)
print(dm)
tree = DistanceTreeConstructor().nj(dm)
Phylo.draw_ascii(tree)

# output
3 sequences, 7915 alignment columns; ['NC_001422.1:5386', 'NC_001420.2:5577', 'NC_001330.1:6087']
columns identical in all three and gap-free: 1876 (23.7%)
NC_001422.1 0.000000
NC_001420.2 0.585850    0.000000
NC_001330.1 0.677448    0.398231    0.000000
    NC_001422.1 NC_001420.2 NC_001330.1
  _________________________________________________________________ NC_001422.1
 |
_|______________________ NC_001420.2
 |
 |____________________________________ NC_001330.1

The three genomes (5,386, 5,577 and 6,087 bases) align into 7,915 columns, of which 23.7 % are identical in all three, and the genome of G4 is closer to alpha3 than either is to phiX174 (an identity distance of 0.40 against 0.59 and 0.68). Read this as a sketch. Whole-genome alignments of rearranged or very divergent genomes are unreliable, an identity distance counts every gap column, and neighbour joining on three sequences cannot be rooted. A real tree needs a protein or codon alignment of conserved genes, a substitution model and support values (chapter A2). The point is the mechanics: AlignIO.read, DistanceCalculator, DistanceTreeConstructor and Phylo.draw_ascii take four lines.

A protein structure with Bio.PDB

Bio.PDB parses structure files into the hierarchy structure, model, chain, residue, atom, with coordinates as NumPy arrays, so geometry is ordinary array arithmetic. The example downloads crambin (PDB entry 1CRN, a 46-residue plant protein), counts its parts, builds the sequence from the atoms, and computes distances between the Cα atoms. Consecutive Cα atoms are about 3.8 Å apart in every protein, which is a useful test that the coordinates were read correctly; the number of residue pairs that are close in space but far apart in sequence is a crude measure of how compact the fold is.

import urllib.request
from Bio.PDB import PDBParser, PPBuilder
import numpy as np

urllib.request.urlretrieve("https://files.rcsb.org/download/1CRN.pdb", "results/1crn.pdb")
s = PDBParser(QUIET=True).get_structure("crambin", "results/1crn.pdb")
model = s[0]
chain = next(model.get_chains())
res = [r for r in chain if r.id[0] == " "]
print("chains:", len(list(model.get_chains())), "| residues:", len(res), "| atoms:", sum(1 for _ in model.get_atoms()))
print("sequence:", str(PPBuilder().build_peptides(chain)[0].get_sequence()))
ca = np.array([r["CA"].coord for r in res])
d = np.linalg.norm(ca[:, None] - ca[None], axis=-1)
print(f"consecutive CA-CA distance: {np.mean([d[i, i + 1] for i in range(len(res) - 1)]):.2f} A")
print("residue pairs closer than 8 A but more than 4 apart in sequence:", int(sum(1 for i in range(len(res)) for j in range(i + 5, len(res)) if d[i, j] < 8)))
rg = np.sqrt(((ca - ca.mean(axis=0)) ** 2).sum(axis=1).mean())
print(f"radius of gyration of the CA atoms: {rg:.1f} A")

# output
chains: 1 | residues: 46 | atoms: 327
sequence: TTCCPSIVARSNFNVCRLPGTPEAICATYTGCIIIPGATCPGDYAN
consecutive CA-CA distance: 3.82 A
residue pairs closer than 8 A but more than 4 apart in sequence: 67
radius of gyration of the CA atoms: 9.7 A

The consecutive distance of 3.82 Å is the expected value. Sixty-seven pairs of residues more than four apart in sequence lie within 8 Å of each other, and the radius of gyration of the Cα atoms is 9.7 Å, small for 46 residues: a compact protein. Secondary-structure assignment needs the external program DSSP (Bio.PDB.DSSP calls it); superposition of two structures uses Bio.PDB.Superimposer; for models predicted by AlphaFold the same parser reads the mmCIF or PDB file, and the per-residue confidence is stored in the B-factor column.

Version traps: why old tutorials fail

Biopython has changed more than most libraries, and code in books and answers written before 2021 often fails or warns. The changes below are the ones you will meet; the first two we checked on the version used for this chapter (1.88).

  • Alphabets are gone. Seq("ATG", IUPAC.unambiguous_dna) raised an error from version 1.78: delete the alphabet argument. A Seq prints as plain text and does not remember whether it is DNA or protein; you keep track of that.
  • GC is now gc_fraction and returns a fraction; from Bio.SeqUtils import GC fails with an ImportError.
  • Bio.pairwise2 is deprecated and warns on import; use Bio.Align.PairwiseAligner (see above).
  • Seq is now close to str: it can be compared, hashed and searched; seq.tostring() and seq.tomutable() no longer exist.
  • Entrez needs a contact address and, for heavy use, an API key; the old pattern of calling it in a tight loop is blocked.

For the rest, the Biopython Tutorial and Cookbook is the manual, and it is kept current with every release: sequence objects (Ch. 3), sequence annotation objects (Ch. 4), Sequence Input/Output (Ch. 5), multiple alignments (Ch. 6), search tools (Ch. 8), Entrez (Ch. 9), Bio.PDB (Ch. 11), Bio.Phylo (Ch. 13), motifs (Ch. 14), clustering and supervised learning (Ch. 15 and 16), graphics including GenomeDiagram (Ch. 17), KEGG (Ch. 18) and the Cookbook (Ch. 20), which holds worked examples such as finding the longest ORFs, plotting GC along a genome and working with the file formats that SeqIO does not cover.

What Biopython is not

It is not fast for bulk work: counting reads, trimming adapters or sorting alignments belongs to compiled tools (, , ). It does not read BAM or VCF files well; use pysam for those. It is not a statistics library; hand the data to pandas, SciPy or R. Its strength is correctness on biological objects: coordinates, strands, genetic codes, ambiguity codes, file-format quirks. Use it for exactly that.

Check your understanding

The same 9 bases, ATGGCCTGA, are translated with table 11 and with table 4. What do you get, and why?

Under table 11, MA followed by a stop (TGA). Under table 4 (Mycoplasma), TGA codes for tryptophan: MAW. Always state the table.

A GFF line gives a gene at 101 to 200 on the minus strand. How do you extract it from a record r?

SeqFeature(FeatureLocation(100, 200, strand=-1), type="gene").extract(r.seq). Convert the start to 0-based; the end stays; extract reverse-complements for the minus strand.

Why does Seq("ACGTRYN").reverse_complement() give NRYACGT?

The sequence is reversed and each letter complemented, including the ambiguity codes: R (A or G) becomes Y (T or C), and N stays N.

You need 5,000 sequences out of a 4 GB FASTA. What do you use?

SeqIO.index for random access by identifier, or a streaming filter with SeqIO.parse; or  grep -f ids.txt on the command line. Not list(SeqIO.parse(...)).

A PWM built from four sites finds 20,000 hits in a genome. Is that a result?

No. Four sites are a poor estimate and a motif is short, so many matches arise by chance. Set a threshold against a background model, and treat hits as candidates to test.

Key references

  1. Cock PJA, Antao T, Chang JT, et al. Biopython: freely available Python tools for computational molecular biology and bioinformatics. Bioinformatics 25, 1422–1423 (2009).
  2. Danecek P, Bonfield JK, Liddle J, et al. Twelve years of  and . GigaScience 10, giab008 (2021).
  3. Shen W, Le S, Li Y, Hu F. : a cross-platform and ultrafast toolkit for FASTA/Q file manipulation. PLoS ONE 11, e0163962 (2016).
  4. Hendrix D. Applied Bioinformatics. Read: Ch. 1, introduction to biological sequences, Biopython and GNU/Linux.
  5. Buffalo V. Bioinformatics Data Skills. O’Reilly, 2015. Read: Ch. 10, working with sequence data; ch. 11, working with range data.
  6. Cock PJA, Fields CJ, Goto N, Heuer ML, Rice PM. The Sanger FASTQ file format for sequences with quality scores, and the Solexa/Illumina FASTQ variants. Nucleic Acids Research 38, 1767–1771 (2010).
  7. Katoh K, Standley DM.  multiple sequence alignment software version 7: improvements in performance and usability. Molecular Biology and Evolution 30, 772–780 (2013).
  8. Hamelryck T, Manderick B. PDB file parser and structure class implemented in Python. Bioinformatics 19, 2308–2310 (2003).
  9. Kyte J, Doolittle RF. A simple method for displaying the hydropathic character of a protein. Journal of Molecular Biology 157, 105–132 (1982).
  10. Guruprasad K, Reddy BVB, Pandit MW. Correlation between stability of a protein and its dipeptide composition: a novel approach for predicting in vivo stability of a protein from its primary sequence. Protein Engineering 4, 155–161 (1990).
  11. Roberts RJ, Vincze T, Posfai J, Macelis D. REBASE: a database for DNA restriction and modification: enzymes, genes and genomes. Nucleic Acids Research 38, D234–D236 (2010).
  12. Saitou N, Nei M. The neighbor-joining method: a new method for reconstructing phylogenetic trees. Molecular Biology and Evolution 4, 406–425 (1987).
  13. Sanger F, Air GM, Barrell BG, et al. Nucleotide sequence of bacteriophage φX174 DNA. Nature 265, 687–695 (1977).
  14. Chang J, Chapman B, Friedberg I, et al. Biopython Tutorial and Cookbook (version 1.81). Read: Ch. 3 to 6 (sequences, annotation, SeqIO, alignments), Ch. 8 (search tools), Ch. 9 (Entrez), Ch. 11 (Bio.PDB), Ch. 13 (Bio.Phylo), Ch. 14 (motifs), Ch. 17 (GenomeDiagram), Ch. 20 (Cookbook).
Core skills · Programming and data

Data science with Python: from a count table to a tested result

This chapter is one complete analysis in Python, from the raw count table to a figure and a statement you can defend. It uses the demonstration study that ships with MicroRes: 72 stool samples from three diets (Control, Fibre, Antibiotic), 4 negative-control blanks, 377 features, and a table of measurements per sample. Every number below was produced by the code shown, run with pandas, NumPy, SciPy, statsmodels, scikit-learn and Matplotlib. The same analysis is repeated in R in the next chapter, and the two agree.

1. Load and check

import numpy as np, pandas as pd
from scipy import stats
counts = pd.read_csv("counts.tsv", sep="\t", index_col=0)     # features x samples
meta   = pd.read_csv("metadata.tsv", sep="\t", index_col=0)   # samples x variables
counts.shape, meta.shape                       # ((377, 76), (76, 14))
assert list(counts.columns) == list(meta.index)   # the same samples in the same order
meta["Sample_type"].value_counts().to_dict()   # {'Sample': 72, 'Blank': 4}
meta["Diet"].value_counts().to_dict()          # {'Control': 24, 'Fibre': 24, 'Antibiotic': 24, 'Blank': 4}

Before any statistics, look at what you have: the shape, the types of the columns (meta.dtypes), missing values (meta.isna().sum()), the design (three groups of 24, balanced), and what is not a sample (four blanks, which are controls and must not enter the group comparison). A balanced design is easy to analyse; with unbalanced groups, many methods need extra care.

bio = meta.index[meta["Sample_type"] == "Sample"]
c, m = counts[bio], meta.loc[bio]
c.shape, int(c.values.sum())                   # ((377, 72), 2109697)
depth = c.sum(axis=0)
int(depth.min()), int(depth.median()), int(depth.max())    # (950, 25055, 80357)

The sequencing depth varies by a factor of 85 between samples (950 to 80,357 reads). That is the first thing to deal with: a feature that has 10 reads in a deep sample and 10 in a shallow one is not equally abundant. Counts must be put on a common scale before samples are compared.

2. Filter

Features seen in very few samples carry little information and add noise and tests. A prevalence filter keeps those present in at least 10% of the samples:

prev = (c > 0).mean(axis=1)
c = c[prev >= 0.10]
c.shape[0]                                      # 300 of the 377 features kept

Choose the threshold before looking at the results, and report it. A filter that depends on the groups (keep features that are present in at least 10% of one group) is a legitimate choice, but it must be applied to the pooled data without looking at the outcome, otherwise it biases the test.

3. Normalise and transform

Two common starting points, each with a purpose:

  • Relative abundance (divide by the sample total): easy to read, and the basis of Bray–Curtis distances and stacked bar plots.
  • Centred log-ratio (CLR): take the logarithm of the counts (with a pseudocount for zeros) and subtract the sample mean. It treats the data as compositional, as they are, and puts the features on a scale where standard statistics behave better.
rel = c / c.sum(axis=0)
clr = np.log(c + 0.5)
clr = clr - clr.mean(axis=0)                    # centre every sample

The pseudocount (0.5 here) is a choice that matters for rare features. State it, and check that the main conclusion does not depend on it by repeating the analysis with 0.1 and 1. Sequencing data are compositional: they tell you the share of each feature, not its absolute amount, so a feature that rises makes all the others fall (Gloor et al., 2017).

4. Diversity and a group comparison

The Shannon index summarises how many features there are and how evenly they share the reads:

p = rel
shannon = -(p * np.log(p.where(p > 0))).sum(axis=0)
shannon.groupby(m["Diet"]).median().round(3).to_dict()
# {'Antibiotic': 3.771, 'Control': 4.252, 'Fibre': 4.143}

Is the difference between the three diets larger than chance? The values are not normally distributed and the groups are small, so use rank-based tests: Kruskal–Wallis for all three groups, then Mann–Whitney for a pair.

g = [shannon[m["Diet"] == d].values for d in ["Control", "Fibre", "Antibiotic"]]
stats.kruskal(*g)             # statistic 22.67, p = 1.2e-05
a, b = shannon[m["Diet"] == "Antibiotic"], shannon[m["Diet"] == "Control"]
stats.mannwhitneyu(a, b)      # U = 85, p = 3.0e-05

Report an effect size with the p value: the antibiotic group has a median Shannon of 3.77 against 4.25 in the control group, a drop of 0.48, which is the number a reader can judge. A very small p value with a tiny difference is not interesting; a large difference with p = 0.06 in six samples per group may be. Both numbers belong in the sentence.

5. Thousands of tests: control the false-discovery rate

Which features differ between antibiotic and control? Test each of the 300 features, on the CLR values, and correct for the number of tests:

from statsmodels.stats.multitest import multipletests
x = clr.loc[:, m["Diet"] == "Antibiotic"]
y = clr.loc[:, m["Diet"] == "Control"]
pv = [stats.mannwhitneyu(x.loc[i], y.loc[i]).pvalue for i in clr.index]
q = multipletests(pv, method="fdr_bh")[1]
(np.array(pv) < 0.05).sum(), (q < 0.05).sum()      # (119, 72)

119 features have a raw p below 0.05, and 72 survive the Benjamini–Hochberg correction at a false-discovery rate of 5%. With 300 tests, about 15 features would have p < 0.05 by chance even if nothing differed, so the raw count overstates the finding by a lot. The corrected value says: of the features reported at this threshold, about 5% are expected to be false. Always report the corrected number and name the method (Benjamini and Hochberg, 1995). Microbiome methods built for compositional counts (ANCOM-BC, ALDEx2, -type models) are better than a rank test on a transformed table when effects are small; use more than one and report what several agree on.

6. A regression, to adjust for something

A group difference might be explained by something else that differs between the groups. A linear model estimates the diet effect while holding other variables fixed:

import statsmodels.formula.api as smf
df = pd.DataFrame({"shannon": shannon, "fibre": m["Fibre_g_day"], "diet": m["Diet"], "bmi": m["BMI"]})
fit = smf.ols("shannon ~ fibre + C(diet, Treatment('Control')) + bmi", data=df).fit()
fit.params.round(3)
# Intercept 4.222 | diet[Antibiotic] -0.480 | diet[Fibre] -0.067 | fibre -0.006 | bmi 0.005
round(fit.rsquared, 3)                 # 0.374

Read the coefficients against the reference level, Control (set explicitly with Treatment('Control')): antibiotic lowers Shannon by 0.48 units relative to control after adjusting for fibre intake and BMI, while fibre intake and BMI do not matter once diet is in the model. R2 = 0.37 says the model explains 37% of the variation in Shannon. Look at the residuals (fit.resid) before you trust the standard errors; a model for repeated measures from the same person (this study has three time points per subject) needs a random effect or a robust standard error, and the p values above ignore that.

7. Ordination: see the samples

Principal component analysis on the CLR values (which gives an Aitchison-distance view) projects the samples onto the directions of greatest variation:

from sklearn.decomposition import PCA
pca = PCA(n_components=2).fit(clr.T.values)
pca.explained_variance_ratio_.round(3)      # [0.135 0.037]

The first two axes explain 13.5% and 3.7% of the variation: small shares, as is usual for microbiome data with hundreds of features, so a separation of groups on these two axes is a strong signal, and the absence of one is not proof of no difference. A PERMANOVA on a distance matrix (skbio.stats.distance.permanova) is the test that goes with the picture; see the R chapter for its output on the same data.

8. Prediction, honestly estimated

Can the community tell whether a sample is from the antibiotic group? A random forest is a reasonable first model. The danger is overfitting: a model with hundreds of features and 72 samples can memorise them. The only honest estimate of performance is on samples the model has not seen, by cross-validation:

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score
y_ = (m["Diet"] == "Antibiotic").astype(int)
cv = StratifiedKFold(5, shuffle=True, random_state=1)
auc = cross_val_score(RandomForestClassifier(300, random_state=1), clr.T.values, y_, cv=cv, scoring="roc_auc")
auc.round(2), round(auc.mean(), 2)          # [0.7 0.92 0.98 0.94 1.0] 0.91

The area under the ROC curve is 0.91 on average, but it ranges from 0.70 to 1.0 across the five folds: with 72 samples, the estimate is noisy, and a single number hides that. Two more rules. Everything that learns from the data (a filter on prevalence, a normalisation fitted to the data, a feature selection) must happen inside each fold, or the test samples leak into the training. And these samples come from 24 subjects with three time points each; if the three samples of a subject fall in both training and test sets, the model can recognise the person, not the diet, and the AUC is inflated. Group the folds by subject (GroupKFold) for such data.

9. A figure that shows the data

import matplotlib; matplotlib.use("Agg")
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(5, 3.5))
for i, (d, col) in enumerate(zip(["Control", "Fibre", "Antibiotic"], ["#4477aa", "#228833", "#cc3311"])):
    v = shannon[m["Diet"] == d]
    ax.scatter(np.random.default_rng(0).normal(i, 0.06, len(v)), v, s=14, color=col)   # jittered points
ax.set_xticks([0, 1, 2], ["Control", "Fibre", "Antibiotic"]); ax.set_ylabel("Shannon index")
fig.tight_layout(); fig.savefig("shannon.png", dpi=300)

Show every sample as a point, so that the reader sees the spread and the sample size; add a median line if you wish; use a colour-blind-safe palette (the three colours above are from Paul Tol's set); label the axes with units; and save at 300 dpi for print or as SVG or PDF for vector output. A bar of means with error bars hides everything that matters in 24 values.

10. The pandas toolbox

The first nine sections used pandas without explaining it. Most of a data scientist's day is spent here, so the main operations are collected in one place. A DataFrame is a table with named columns and a labelled row index; a Series is one column. Every example below ran on the demonstration study.

Selecting rows and columns

meta.loc["P01_Day0", ["Diet", "Age"]].to_dict()    # by label: {'Diet': 'Control', 'Age': 22.0}
meta.iloc[0:2, 3:5]                                  # by position: the first two rows, columns 3 and 4
old = meta[(meta.Sample_type == "Sample") & (meta.Age > 40)]   # a boolean mask; each condition in brackets, joined by & or |
len(old)                                             # 48 samples from people older than 40
meta.query("Diet == 'Fibre' and BMI > 28").shape[0]  # the same idea, written as a string: 9 rows

loc uses labels and includes both ends of a slice; iloc uses positions and excludes the end, like a Python list. Writing and instead of & in a mask is the most common error message a beginner meets: pandas needs the element-wise operator.

Group, aggregate, summarise

samples = meta[meta.Sample_type == "Sample"]
samples.groupby("Diet").agg(n=("Subject", "size"), subjects=("Subject", "nunique"),
                            age=("Age", "mean"), butyrate=("Butyrate_mM", "median")).round(2)
#              n  subjects    age  butyrate
# Antibiotic  24         8  43.12      8.74
# Control     24         8  46.38     20.26
# Fibre       24         8  44.00     33.38

This table already contains a design fact that matters later: 24 samples per diet, but only 8 subjects per diet, because every person was sampled three times (days 0, 14 and 28). The samples are not independent, and sections 13 and 14 return to this. groupby follows one pattern in all its uses: split the table by a key, apply a function to each piece, combine the results.

Categories: set the order, save memory

meta["Diet"] = pd.Categorical(meta["Diet"], categories=["Control", "Fibre", "Antibiotic", "Blank"])
meta["Diet"].cat.categories.tolist()     # ['Control', 'Fibre', 'Antibiotic', 'Blank']  (the order of plots and tables)
meta["Diet"].memory_usage(deep=True)     # 6947 bytes, against 10748 as plain strings

A categorical column stores each label once and every row as a small integer. It sorts and plots in the order you gave, not alphabetically, and in a model the first category is the reference level.

Strings: parsing a taxonomy

tax = pd.read_csv("taxonomy.tsv", sep="\t", index_col=0, header=None)    # one column: the lineage
t = tax[1].str.split(";", expand=True)
t.columns = ["kingdom", "phylum", "class", "order", "family", "genus", "species"]
t = t.apply(lambda col: col.str[3:])         # drop the 'k__', 'p__' prefixes
t["phylum"].value_counts().head(4).to_dict()
# {'Firmicutes': 196, 'Bacteroidota': 92, 'Actinobacteriota': 31, 'Proteobacteria': 29}

The .str accessor applies a string method to a whole column, and expand=True turns the pieces into columns. For patterns use .str.extract(r"(Day\d+)") and .str.contains; they take regular expressions (the earlier chapters on the shell and on Python explain them).

Joining tables

reads = counts.sum(axis=1).rename("reads")          # total reads per feature
t.join(reads).groupby("phylum")["reads"].sum().sort_values(ascending=False).head(4).to_dict()
# {'Bacteroidota': 928866, 'Firmicutes': 844813, 'Proteobacteria': 147642, 'Actinobacteriota': 128908}

join aligns on the index; merge(a, b, on="key", how="left") aligns on columns and chooses what happens to unmatched rows: inner keeps only matches, left keeps every row of the left table (the safe default when the left table is your samples), outer keeps everything. After every merge, check the number of rows: an unexpected increase means duplicated keys, an unexpected decrease means keys that did not match (often a stray space or a different case).

Wide and long

small = counts.iloc[:3, :3].reset_index()
long = small.melt(id_vars="ASV", var_name="sample", value_name="count")     # one row per feature and sample
#        ASV     sample  count
# 0  ASV0001   P01_Day0   4049
# 1  ASV0002   P01_Day0   5851  ...  (9 rows)
long.pivot(index="ASV", columns="sample", values="count")                    # back to wide

Wide tables (features by samples) suit matrix mathematics; long tables (one measurement per row) suit plotting with seaborn and models with a formula. Wickham's rule of "tidy data" is that each variable is a column, each observation a row, each kind of unit a table (Wickham, 2014). Moving between the two with melt and pivot is a daily task.

Missing values

x = pd.Series([1.0, np.nan, 3.0])
x.isna().sum(), x.mean(), x.fillna(x.mean()).tolist()    # (1, 2.0, [1.0, 2.0, 3.0])
meta.isna().sum().sum()                                   # 32 missing cells in the metadata

pandas skips NaN when it averages, which is convenient and sometimes dangerous. Find where the missing cells are before deciding what to do (here they are the metadata of the four blanks, which is expected). Filling with the mean hides uncertainty and shrinks the variance; deleting rows loses samples; a model that handles missingness explicitly is better when much is missing. Whatever you choose, count what you removed and say so.

11. NumPy: arrays, broadcasting, and what PCA is

pandas is built on NumPy arrays: blocks of numbers of one type, stored together, and operated on all at once (Harris et al., 2020).

a = np.arange(12).reshape(3, 4)
a.shape, a.sum(axis=0).tolist(), a.sum(axis=1).tolist()    # ((3, 4), [12, 15, 18, 21], [6, 22, 38])

axis=0 collapses the rows (one result per column); axis=1 collapses the columns (one per row). The rule to remember: the axis you name is the one that disappears.

v = counts.to_numpy(float)                 # 377 features x 76 samples
rel = v / v.sum(axis=0)                    # broadcasting: a (377, 76) array divided by a (76,) vector, column by column
rel.shape, np.allclose(rel.sum(axis=0), 1)   # ((377, 76), True)

Broadcasting stretches the smaller array across the larger one when their shapes are compatible, comparing shapes from the last dimension backwards. It replaces loops, and it is also the main source of silent mistakes: dividing a (377, 76) array by a vector of length 377 would fail, but dividing a square matrix by the wrong vector would run and give nonsense. Check shapes after every step.

Why vectorise

import time
t0 = time.perf_counter(); out  = [sum(v[i, j] for j in range(v.shape[1])) for i in range(v.shape[0])]
t1 = time.perf_counter(); out2 = v.sum(axis=1);  t2 = time.perf_counter()
np.allclose(out, out2), round((t1 - t0) / (t2 - t1))     # (True, 189)  the array call was 189 times faster

The loop and the array call give the same numbers; the array call runs in compiled code. The ratio depends on the machine (this one measured 189), but the order of magnitude does not. Write the clear version first, measure, and vectorise only the part that is slow.

PCA written with a decomposition

X = clr.T.to_numpy();  X = X - X.mean(axis=0)         # samples x features, centred
U, S, Vt = np.linalg.svd(X, full_matrices=False)
(S**2 / (S**2).sum())[:3].round(4).tolist()            # [0.1346, 0.0366, 0.0353]
from sklearn.decomposition import PCA
PCA(3).fit(X).explained_variance_ratio_.round(4).tolist()   # [0.1346, 0.0366, 0.0353]: the same
scores = U[:, :2] * S[:2]                               # the coordinates of the samples on the first two axes

PCA is not magic: it is the singular value decomposition of the centred table. The squared singular values give the share of variance of each axis, Vt holds the loadings (which features define an axis) and U * S the sample coordinates. Knowing this makes it clear why PCA on centred log-ratios (the Aitchison distance) is the standard for compositional data, and why scaling every feature to unit variance gives rare features as much weight as dominant ones.

Random numbers belong here too: rng = np.random.default_rng(0) creates a generator whose sequence is fixed by the seed (rng.integers(0, 10, 5) gave [8, 6, 5, 2, 3]), and every function that shuffles, samples or initialises should take it or a seed, so that the analysis can be repeated exactly.

12. Statistics you can write yourself: permutation, bootstrap, effect size, power

A test is a recipe, not a ritual. Writing it from the definition once removes much of the mystery, and it gives you a method that works when no ready-made test fits. The question here: is the Shannon index of the Antibiotic group (n = 24) lower than that of the Control group (n = 24)?

from scipy import stats
a = shannon[m.Diet == "Control"].to_numpy();  b = shannon[m.Diet == "Antibiotic"].to_numpy()
stats.ttest_ind(a, b, equal_var=False)       # Welch: t = 5.01, df = 27.1, p = 2.9e-05
stats.mannwhitneyu(a, b)                     # rank-sum: U = 491 (85 for the other group), p = 3.0e-05

The Welch test compares means without assuming equal variances (the antibiotic group's standard deviation, 0.45, is three times the control group's, 0.13, so this matters; use it as the default instead of the classical t-test). The rank-sum test uses only the order of the values. Both say the groups differ. Now the same question without any distribution assumption:

rng = np.random.default_rng(1)
obs = a.mean() - b.mean();  pool = np.r_[a, b];  n = len(a)
perm = np.empty(10000)
for i in range(10000):
    rng.shuffle(pool)                                  # if the diet did not matter, the labels are exchangeable
    perm[i] = pool[:n].mean() - pool[n:].mean()
obs.round(3), (np.sum(np.abs(perm) >= abs(obs)) + 1) / 10001     # (0.478, 0.0001)

The permutation test asks how often shuffling the labels produces a difference as large as the observed 0.478. With 10,000 shuffles none did, so the p value is reported as its smallest possible value, 1/10,001 (the "+1" counts the observed arrangement as one of the permutations; a p value of exactly 0 is never correct). The same logic, with the test statistic replaced, is what PERMANOVA does for whole communities.

bs = np.array([rng.choice(a, len(a)).mean() - rng.choice(b, len(b)).mean() for _ in range(5000)])
np.percentile(bs, [2.5, 97.5]).round(3)               # [0.305, 0.667]: a 95% bootstrap interval for the difference

The bootstrap resamples each group with replacement to see how much the estimate would vary from study to study. A p value says "not zero"; the interval says how large the difference plausibly is, which is what a reader needs. An interval from 0.31 to 0.67 Shannon units is a clear loss of diversity.

Effect size

sp = np.sqrt((a.var(ddof=1) + b.var(ddof=1)) / 2)
round((a.mean() - b.mean()) / sp, 2)                  # Cohen's d = 1.45: a large standardised difference
(np.sign(a[:, None] - b[None, :])).sum() / (len(a) * len(b))      # Cliff's delta = 0.70

Cohen's d expresses the difference in units of the pooled standard deviation. Cliff's delta is the probability that a random control sample is higher than a random antibiotic sample, minus the probability of the opposite: 0.70 means that in about 85% of random pairs the control is higher. Report an effect size next to every p value, because with enough samples a trivial difference is "significant" and with too few a large one is not.

Power: how many samples?

from statsmodels.stats.power import TTestIndPower
TTestIndPower().solve_power(effect_size=0.8, alpha=0.05, power=0.8)   # 25.5 per group
TTestIndPower().solve_power(effect_size=0.5, alpha=0.05, power=0.8)   # 63.8 per group

To detect a large effect (d = 0.8) with 80% probability at the 5% level, you need about 26 samples per group; for a medium effect (d = 0.5) about 64. The effect observed here (d = 1.45) is larger than either, which is why 24 per group was enough. Power must be calculated before the study from an effect size you can justify (a pilot, the literature, or the smallest difference that matters), never from the result afterwards. For microbiome-specific designs, where the unit is a community and the outcome a distance or a taxon proportion, see the power chapter of Xia and Sun's book (Read: ch. 5).

13. Models in more detail: diagnostics, counts, repeated measures, binary outcomes

Check the assumptions of the regression

import statsmodels.api as sm, statsmodels.formula.api as smf
m["shannon"] = shannon                     # from section 4, now a column of the sample table
mod = smf.ols("shannon ~ C(Diet, Treatment('Control')) + Age + C(Sex)", data=m).fit()
mod.params.round(3)       # Intercept 4.052; Antibiotic -0.466; Fibre -0.104; Sex[M] 0.007; Age 0.004
round(mod.rsquared, 3)    # 0.376
stats.shapiro(mod.resid).pvalue.round(3)                                  # 0.003: residuals are not normal
sm.stats.diagnostic.het_breuschpagan(mod.resid, mod.model.exog)[1]        # p = 0.002: the spread differs between groups

The coefficients are differences from the reference (Control): the antibiotic group is 0.47 Shannon units lower, adjusted for age and sex, while age and sex add nothing here. But both diagnostic tests fail. The cause is the one seen above: the antibiotic group is far more variable. The estimates remain unbiased, but the standard errors are not trustworthy. Three honest responses: use heteroscedasticity-robust standard errors (.fit(cov_type="HC3")), use a model with a separate variance per group, or use a permutation or rank-based method as in section 12. Plot the residuals against the fitted values first; a plot shows how the assumption fails, which a test does not.

Counts need a count model

dd = m.assign(y=c.loc["ASV0001"].values, depth=c.sum(axis=0).values)
pg = smf.glm("y ~ C(Diet, Treatment('Control'))", data=dd, family=sm.families.Poisson(),
             offset=np.log(dd.depth)).fit()
round(pg.pearson_chi2 / pg.df_resid)        # 1290: far above 1, so the Poisson model is wrong for these data
nb = smf.negativebinomial("y ~ C(Diet, Treatment('Control'))", data=dd, offset=np.log(dd.depth)).fit(disp=0)
np.exp(nb.params).round(3)    # Intercept 0.046; Antibiotic 2.730; Fibre 0.377  (rate ratios against Control)
1 / nb.params["alpha"]        # theta = 0.94  

This feature is the first of the table, and it is a good example of why counts are not analysed with a plain regression. The Poisson model assumes that the variance equals the mean; the Pearson dispersion of 1,290 says the variance is more than a thousand times larger. The negative binomial model adds a dispersion parameter. Its coefficients, exponentiated, are rate ratios: after accounting for sequencing depth (the offset, which makes the model about reads per read sequenced), the feature is 2.73 times as abundant in the antibiotic group (p = 0.0007) and 0.38 times as abundant in the fibre group (p = 0.001) as in the controls. The same model in R (MASS::glm.nb, next chapter) gave the same rate ratios and p values to three decimals, which is the check that matters: two implementations agreeing means the code and the model are understood. Dedicated tools for sequencing counts are built on this idea:  and  fit the negative binomial model and shrink the dispersion across features, which stabilises the estimates when each feature has few samples, and - takes another route, a linear model on log counts with a weight for each observation (Love et al., 2014; Law et al., 2014).

Repeated measures: the same person three times

mm = smf.mixedlm("shannon ~ C(Diet, Treatment('Control')) * Day", data=m, groups=m["Subject"]).fit()
mm.summary().tables[1]
#                                                    Coef.  Std.Err.      z   P>|z|
# Intercept                                          4.230     0.087  48.60  0.000
# Antibiotic                                        -0.255     0.123  -2.07  0.038
# Fibre                                             -0.002     0.123  -0.02  0.987
# Day                                                0.000     0.004   0.06  0.951
# Antibiotic:Day                                    -0.016     0.006  -2.82  0.005
# Fibre:Day                                         -0.008     0.006  -1.39  0.165
mm.cov_re.iloc[0, 0], mm.scale        # variance between subjects 0.019, residual variance 0.050

Each of the 24 subjects contributes three samples, so the 72 rows are not 72 independent observations. A mixed model gives each subject its own baseline (the random intercept, groups=m["Subject"]) and estimates the effects of diet and time across subjects. The result tells a story the single-time comparison could not: at day 0 the antibiotic group is already 0.26 lower, and its Shannon index then falls by 0.016 per day relative to the controls (p = 0.005), while the fibre group does not differ detectably. About 27% of the variance (0.019 of 0.069) lies between people, the rest within a person over time.

Treating the 72 samples as independent, as in section 6, gives a standard error that is too small, because the information in 24 people is less than in 72 strangers. Always ask: what is the unit that was randomised or sampled independently? Here it is the subject. The R version (nlme::lme) gave the same estimates; it gave p = 0.051 for the antibiotic main effect where statsmodels gave 0.038, because R uses a t distribution with 21 degrees of freedom for a between-subject effect and statsmodels uses a normal approximation. With 8 subjects per diet the t distribution is the careful choice, and the difference between 0.038 and 0.051 is a reminder that p values near 0.05 do not deserve a verdict, only an interval.

A binary outcome: logistic regression

dl = m[m.Diet.isin(["Control", "Antibiotic"])].copy();  dl["ab"] = (dl.Diet == "Antibiotic").astype(int)
lg = smf.logit("ab ~ shannon", data=dl).fit(disp=0)
lg.params.round(3).to_dict(), np.exp(lg.params["shannon"]).round(3)    # slope -6.158; odds ratio per Shannon unit 0.002
from sklearn.metrics import roc_auc_score
roc_auc_score(dl.ab, lg.predict(dl)).round(3)                           # 0.852

Logistic regression models the log-odds of the outcome. The slope of -6.16 means that each additional Shannon unit multiplies the odds of being an antibiotic sample by 0.002. This number looks extreme because one Shannon unit is a huge change in this data (the standard deviation is about 0.3), so express such effects per standard deviation. The area under the ROC curve, 0.85, is the probability that a random antibiotic sample has a lower diversity than a random control sample. Note that it was computed on the same data the model was fitted to; section 14 shows what it becomes when it is estimated honestly.

Faraway's Extending the Linear Model with R (Read: ch. 2 binary response, ch. 5 count regression, ch. 10 random effects, ch. 11 repeated measures) and ISLP (ch. 3 and 4) treat the same models with more mathematics.

14. Machine learning without fooling yourself

Machine learning in bioinformatics fails far more often through the evaluation than through the algorithm. This section predicts, from the 300 CLR features, whether a sample belongs to the antibiotic group (24 positives of 72), and spends most of its length on how to know how well that works (James et al., 2023; Pedregosa et al., 2011).

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, StratifiedGroupKFold, cross_val_score
X = clr.T.to_numpy();  y = (m.Diet == "Antibiotic").to_numpy().astype(int);  groups = m.Subject.to_numpy()
lr = make_pipeline(StandardScaler(), LogisticRegression(C=0.1, max_iter=5000))   # scaling is inside the pipeline

The split must follow the independence of the data

cross_val_score(lr, X, y, cv=StratifiedKFold(5, shuffle=True, random_state=1), scoring="roc_auc").round(3)
# [0.98  0.92  0.925 0.978 0.978]  mean 0.956
sg = StratifiedGroupKFold(5, shuffle=True, random_state=1)
cross_val_score(lr, X, y, cv=sg, groups=groups, scoring="roc_auc").round(3)
# [0.852 0.889 0.833 0.889 0.806]  mean 0.854

The two lines differ in one thing: in the first, the three samples of one person can be split between the training and the test folds; in the second, a subject is entirely in one or the other. The first gives AUC 0.96, the second 0.85. The model in the first case is probably recognising people in part (each person has a stable microbiome), and the better score is an illusion. When the model will be applied to new patients, only the second number estimates what you will get. Any time a data set has repeated samples, families, batches, hospitals or sequencing runs, the unit of splitting must be that group.

Leakage: the other way to fool yourself

from sklearn.feature_selection import SelectKBest, f_classif
yperm = np.random.default_rng(0).permutation(y)                    # the labels are now pure noise
Xl = SelectKBest(f_classif, k=20).fit_transform(X, yperm)           # choose the 20 best features using ALL samples...
cross_val_score(LogisticRegression(max_iter=5000), Xl, yperm, cv=StratifiedKFold(5, shuffle=True, random_state=1), scoring="roc_auc").mean()   # 0.885
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=5000))   # ...or inside each training fold
cross_val_score(pipe, X, yperm, cv=StratifiedKFold(5, shuffle=True, random_state=1), scoring="roc_auc").mean()                  # 0.659

With random labels, selecting the features first and cross-validating afterwards yields an AUC of 0.89, a model with a spectacular and entirely false performance, because the test samples helped choose the features. Selecting inside the pipeline gives 0.66, which is what noise looks like on 72 samples (the true value of a useless model is 0.5; with so few samples the estimate wanders, and error bars on a cross-validated score are wide (Varoquaux, 2018)). Every step that learns something from the data (scaling, imputation, feature selection, dimension reduction, tuning) belongs inside the cross-validation loop. A Pipeline object makes this automatic (Ambroise and McLachlan, 2002).

Tuning needs its own loop

from sklearn.model_selection import GridSearchCV
inner = GridSearchCV(make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)),
                     {"logisticregression__C": [0.001, 0.01, 0.1, 1]},
                     cv=StratifiedKFold(3, shuffle=True, random_state=2), scoring="roc_auc")
nested = cross_val_score(inner, X, y, cv=sg, groups=groups, scoring="roc_auc")
nested.round(3), nested.mean().round(3)        # [0.852 0.87 0.926 0.889 0.806]  0.869

If you tune a setting (here the regularisation strength C) and report the best cross-validated score, that score is optimistic, because the setting was chosen to maximise it. Nested cross-validation tunes inside each training fold and scores on a fold that tuning never saw: the honest estimate is 0.87. Regularisation is the central idea of ISLP's chapter 6: with 300 features and 72 samples, an unpenalised model would fit the training data perfectly and predict poorly, and the penalty C controls the trade-off.

Other models, and what they say

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(500, random_state=1, n_jobs=2)
cross_val_score(rf, X, y, cv=sg, groups=groups, scoring="roc_auc").mean().round(3)            # 0.865  (sg: the grouped splitter above)
l1 = make_pipeline(StandardScaler(), LogisticRegression(penalty="l1", solver="liblinear", C=0.1))
cross_val_score(l1, X, y, cv=sg, groups=groups, scoring="roc_auc").mean().round(3)            # 0.641
l1.fit(X, y);  int((l1[-1].coef_[0] != 0).sum())                                               # 10 of 300 features kept

The random forest (an ensemble of decision trees that vote) reaches 0.87 and the L1-penalised model, which also selects features by setting coefficients to zero, only 0.64 with this value of C. The lasso was not tuned here, and a stronger or weaker penalty would change the result: a lesson in why every method must be tuned in a nested loop before it is compared with another. Simple models are strong baselines; try the logistic regression before the forest, and report both.

The cost of a threshold, and importance

from sklearn.model_selection import cross_val_predict
from sklearn.metrics import confusion_matrix, balanced_accuracy_score
pred = cross_val_predict(lr, X, y, cv=sg, groups=groups, method="predict_proba")[:, 1]
confusion_matrix(y, pred > 0.5).tolist(), balanced_accuracy_score(y, pred > 0.5).round(3)   # [[45, 3], [8, 16]]  0.802

At the 0.5 threshold the model finds 16 of the 24 antibiotic samples (sensitivity 67%) and wrongly flags 3 of the 48 others (specificity 94%). Accuracy alone would be misleading when the classes are unbalanced; report sensitivity, specificity, the AUC, and the precision if the outcome is rare. Which features matter can be asked from a held-out set with permutation_importance: shuffle one feature and see how much the held-out AUC drops. Here the top feature was ASV0155 (AUC drop 0.066), but the held-out AUC of that forest was only 0.73 and the set small, so a ranking of features from 24 test samples is a hint for follow-up, not a finding.

y3 = m.Diet.to_numpy()      # three classes
cross_val_score(lr, X, y3, cv=sg, groups=groups, scoring="balanced_accuracy").mean().round(3)     # 0.611

Telling the three diets apart, with a chance level of 33%, gives a balanced accuracy of 0.61: the antibiotic samples are recognisable, but Control and Fibre are not easily separated by this feature set, in line with the small difference in Shannon seen earlier. State what the model cannot do as well as what it can.

Clustering: finding groups nobody labelled

from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score
Z = PCA(5).fit_transform(X - X.mean(0))
adjusted_rand_score(m.Diet, KMeans(3, n_init=10, random_state=1).fit(Z).labels_).round(3)     # 0.438
from scipy.cluster.hierarchy import linkage, fcluster
from scipy.spatial.distance import pdist
adjusted_rand_score(m.Diet, fcluster(linkage(pdist(X), "ward"), 3, "maxclust")).round(3)       # 0.502

The adjusted Rand index compares two partitions: 1 means identical, 0 means what chance would give. Both methods recover the diets only moderately (0.44, 0.50). Clustering always returns clusters, even in noise, so judge them against something (labels, replicates, stability on resampling), and never name the groups it found as if they were biological categories.

15. Figures with matplotlib and seaborn

Matplotlib draws figures from objects: a Figure holds one or more Axes, and everything is a method of an Axes. Write figures this way, not with the stateful plt.plot shortcuts, and they stay controllable when they grow (Hunter, 2007; Waskom, 2021).

import matplotlib.pyplot as plt, seaborn as sns
order = ["Control", "Fibre", "Antibiotic"]
pal = {"Control": "#4477aa", "Fibre": "#228833", "Antibiotic": "#cc3311"}      # a colour-blind-safe set
m["shannon"] = shannon
fig, ax = plt.subplots(1, 2, figsize=(8, 3.4), constrained_layout=True)
sns.boxplot(data=m, x="Diet", y="shannon", order=order, palette=pal, showfliers=False, ax=ax[0])
sns.stripplot(data=m, x="Diet", y="shannon", order=order, color="k", size=3, ax=ax[0])    # every sample, on top
ax[0].set(xlabel="", ylabel="Shannon index")
for d in order:                                                      # change over time, one line per diet
    s = m[m.Diet == d].groupby("Day")["shannon"].mean()
    ax[1].plot(s.index, s.values, marker="o", color=pal[d], label=d)
ax[1].set(xlabel="Day", ylabel="mean Shannon");  ax[1].legend(frameon=False)
fig.savefig("shannon.pdf")                       # a vector file for a paper
fig.savefig("shannon.png", dpi=300)              # 2400 x 1020 pixels at 8 x 3.4 inches

This is the figure of the section 13 story: left, the distribution of every sample by diet, with the points shown so that nobody has to trust the box; right, the group means over time, where the falling antibiotic line is the interaction. Rules that make a figure honest: show the individual points when there are few; label axes with units; use colour for the variable that matters and keep it the same across figures; choose a colour-blind-safe palette; never let the y axis start at a value that exaggerates a small difference in a bar chart; save as PDF or SVG for print and at 300 dpi or more for bitmaps. sns.clustermap(np.log10(top + 1), cmap="viridis") draws a clustered heat map of the most abundant features, and sns.pairplot a matrix of all pairs of variables.

On a server without a screen, call matplotlib.use("Agg") before importing pyplot, and the figure is written to a file without trying to open a window.

16. Larger data and faster code

The demonstration table has 28,000 cells; a real shotgun study can have a hundred million. Four ideas cover most of the problem.

  • Sparse storage. A count table is mostly zeros: here 55.6%. scipy.sparse.csr_matrix(x) stores only the non-zeros: 146,220 bytes against 217,152 for the dense 64-bit array here, a saving of a third. The saving grows quickly with sparsity (a table that is 95% zeros needs less than a tenth of the dense memory) and pays off for shotgun and single-cell data. Smaller number types help too: x.astype(np.int32) halves the memory of a table of counts.
  • Read in chunks when the file does not fit in memory: for chunk in pd.read_csv(f, chunksize=50000): total += chunk["a"].sum() reads 200,000 rows 50,000 at a time and gives the same sum as reading it whole.
  • Binary formats. A text table must be parsed on every read. df.to_parquet("x.parquet") stores types and compresses; the 200,000-row test table was smaller than the CSV. Use HDF5 or Zarr for large arrays, and keep CSV or TSV for tables that people will open in a spreadsheet.
  • Vectorise, then measure. big.apply(lambda r: r["a"] + r["b"], axis=1) loops in Python over 200,000 rows and was more than fifty times slower than big["a"] + big["b"] with identical results. cProfile (pr = cProfile.Profile(); pr.enable(); ...; pr.disable()) shows where the time goes; optimise only what the profile points to.

When one machine is not enough, the same code moves to a cluster: Dask and Polars give a pandas-like interface over data larger than memory, and a workflow manager (, ) runs one script per sample in parallel. The scientific logic does not change; only where it runs.

17. Write code that you and others can trust

A script that only you can run is a draft. Three habits turn analysis code into something reusable (Wilson et al., 2017).

Functions with a contract

# diversity.py
import numpy as np

def shannon(counts):
    """Shannon index (natural log) of a vector of counts; zeros are ignored."""
    x = np.asarray(counts, dtype=float)
    if x.ndim != 1 or (x < 0).any():
        raise ValueError("counts must be a one-dimensional, non-negative vector")
    total = x.sum()
    if total == 0:
        return float("nan")
    p = x[x > 0] / total
    return float(-(p * np.log(p)).sum())

A function that states what it expects, refuses what it cannot handle and returns something definite for the edge case (an empty sample) is a function you can reuse in the next project without rereading it.

Tests

# tests/test_diversity.py
import math, pytest
from diversity import shannon

def test_one_feature_has_zero_diversity():
    assert shannon([10, 0, 0]) == 0.0

def test_even_community_is_log_richness():
    assert shannon([5, 5, 5, 5]) == pytest.approx(math.log(4))

def test_scaling_does_not_change_it():
    assert shannon([1, 2, 3]) == pytest.approx(shannon([10, 20, 30]))

def test_empty_sample_is_nan():
    assert math.isnan(shannon([0, 0]))

def test_negative_counts_are_refused():
    with pytest.raises(ValueError):
        shannon([1, -1])
$ python -m pytest -q
.....                                                                    [100%]
5 passed in 0.01s

Each test states a fact you know independently of the code: a community with one feature has zero diversity; four equally abundant features have a Shannon index of ln 4; multiplying all counts by ten changes nothing. If the code is later "improved" and one fact stops being true, the test says so at once. Test the edges (empty, one feature, negative numbers, missing values), because that is where analysis code breaks.

A command-line interface

# summarise.py
import argparse
import pandas as pd
from diversity import shannon

def main():
    ap = argparse.ArgumentParser(description="Shannon index of every sample in a count table (features x samples).")
    ap.add_argument("counts", help="tab-separated count table, features in rows")
    ap.add_argument("-o", "--out", default="shannon.tsv")
    a = ap.parse_args()
    t = pd.read_csv(a.counts, sep="\t", index_col=0)
    t.apply(shannon).rename("shannon").to_csv(a.out, sep="\t")
    print(f"{t.shape[1]} samples written to {a.out}")

if __name__ == "__main__":
    main()
$ python summarise.py counts.tsv -o shannon.tsv
76 samples written to shannon.tsv
$ python summarise.py -h        # prints the usage, the arguments and the description

argparse gives a script a documented interface, so it can be called from a shell loop, a Makefile or a workflow manager with different inputs, which is how a one-off analysis becomes a step of a pipeline. Keep the logic in importable functions and only the argument handling in main(); the if __name__ == "__main__" guard lets the tests import the module without running it. Add git for history and a requirements.txt with the versions, and the project can be reproduced.

18. Make it rerun

Put the code in a script, not in notebook cells run out of order. Fix random seeds (random_state=1, default_rng(0)). Record the versions (pip freeze). Read inputs from data/, write to results/, and keep the raw table untouched. At the top, assert the things that must be true (assert list(counts.columns) == list(meta.index)), so that a wrong file fails loudly instead of producing a plausible wrong answer.

19. Why the statistics work: learning from simulation

You can check every claim in the statistics sections by generating data where you know the truth. This is the best way to understand p values, power and error rates, and the way to test an analysis before trusting it on real data. Everything below ran with a fixed seed (rng = np.random.default_rng(7)): 10,000 simulated experiments of two groups of 24 measurements each.

def experiment(delta, n=24, B=10000):
    a = rng.normal(0, 1, (B, n));  b = rng.normal(delta, 1, (B, n))
    return stats.ttest_ind(a, b, axis=1).pvalue                 # B p values, one per experiment
(experiment(0.0) < 0.05).mean()     # 0.050: with no real difference, 5% of experiments are "significant"
(experiment(0.5) < 0.05).mean()     # 0.394: a medium effect (d = 0.5) is found in only 39% of experiments
(experiment(0.8) < 0.05).mean()     # 0.776: a large effect (d = 0.8) in 78%

The first line is the meaning of a 5% significance level: it is the false-positive rate when nothing is going on. The next two are statistical power. A study of 24 per group that looks for a medium effect misses it six times in ten, and that is a property of the design and not of bad luck, which is why section 12 calculates the sample size beforehand. A "negative" result from an underpowered study says little.

Many tests: where false discoveries come from

a = rng.normal(0, 1, (300, 24));  b = rng.normal(0, 1, (300, 24));  b[:60] += 1.0     # 60 features truly differ by one SD
p = stats.ttest_ind(a, b, axis=1).pvalue
(p < 0.05).sum(), (p[60:] < 0.05).sum()                    # 61 "hits", 9 of them false (the 240 null features)
q = multipletests(p, method="fdr_bh")[1]
(q < 0.05).sum(), (q[60:] < 0.05).sum()                    # 48 hits, 2 false: a false-discovery proportion of 4%
(multipletests(p, method="bonferroni")[1] < 0.05).sum()     # 16 hits, none false: safe but it finds few of the 60 real ones

This is section 5's problem with the answer known. Without correction, 9 of the 61 hits are false (15%, close to the 12 that 240 null tests × 5% would predict). The Benjamini–Hochberg procedure keeps the proportion of false discoveries near the 5% it promises (2 of 48), while Bonferroni, which controls the chance of any false hit, is so strict that it finds only 16 of the 60 real differences. FDR control is the standard for exploratory screens, and Bonferroni for a handful of pre-specified tests (Benjamini and Hochberg, 1995).

What a confidence interval promises

x = rng.normal(5, 2, (10000, 24));  mean = x.mean(1);  se = x.std(1, ddof=1) / np.sqrt(24);  t = stats.t.ppf(.975, 23)
((mean - t * se <= 5) & (5 <= mean + t * se)).mean()      # 0.954

A 95% interval is a procedure that captures the true value in 95% of repetitions: 0.954 here. It does not mean that the true value lies in this particular interval with 95% probability, and it does not mean that 95% of the data are inside it. The correct reading, "the data are compatible with true values in this range", is what makes an interval more informative than a p value.

The winner's curse and regression to the mean

true = rng.normal(0, 1, 2000);  o1 = true + rng.normal(0, 1, 2000);  o2 = true + rng.normal(0, 1, 2000)
top = o1 > np.quantile(o1, 0.9)
o1[top].mean().round(2), o2[top].mean().round(2)          # 2.40, then 1.13

Take the top 10% on a noisy first measurement and measure them again: the mean falls from 2.40 to 1.13 with no intervention at all, because part of what put them on top was luck. The features with the most extreme effects in a discovery study are, on average, overestimated, and a replication shows smaller effects. Whoever picks "the best" biomarkers from a small study must expect that, and must replicate in new samples.

The central limit theorem, as it affects you

stats.skew(rng.negative_binomial(1, 0.05, 100000)).round(2)                      # 2.0: raw counts are strongly skewed
stats.skew(rng.negative_binomial(1, 0.05, (10000, 24)).mean(1)).round(2)         # 0.40: means of 24 of them much less so

Means of many values are close to normal even when the values themselves are not, which is why t-tests on means of reasonable samples are often acceptable, and why the same test applied to a single heavily skewed variable with 5 observations per group is not. The skewness falls from 2.0 for single counts to 0.4 for means of 24. This is not a licence to ignore the data: the previous sections found that the variances differed between groups, and the central limit theorem does nothing for that.

The practical habit is to simulate before the experiment (what would my design find if the effect were as I hope?), and after the analysis (does my method return the answer when I plant it?). Rosner's Fundamentals of Biostatistics (Read: the chapters on estimation, hypothesis testing and multiple comparisons) and Petrie and Watson's Statistics for Veterinary and Animal Science explain the theory behind these experiments.

20. Exercises

Do these on the demonstration files (the Data menu of MicroRes has them) before opening the answers. Each one uses only what this chapter showed.

  1. How many features are present in all 72 biological samples, and how many in at least half of them?
  2. What is the median sequencing depth in each diet group? Could a difference in depth by diet distort the comparison?
  3. Which genus has the most reads over the whole study? (Parse the taxonomy first.)
  4. How many reads do the four blanks have, and what would you do with that number?
  5. Compare the median number of observed features (richness) of the three diets, and test it in the way section 12 tests Shannon.
  6. Write a function pielou(counts) (Shannon divided by the logarithm of the number of observed features) with a test for an even community and for a community of one feature.
Answers

1. One feature is present in all 72 samples, and 166 are present in at least 36: (c > 0).all(axis=1).sum() and ((c > 0).mean(axis=1) >= 0.5).sum(). Almost no feature is universal, which is why prevalence filters are chosen by the question and not by a single "core" rule.

2. c.sum(axis=0).groupby(m.Diet).median(): Antibiotic 27,718, Control 22,098, Fibre 28,377. The groups are within 30% of each other, much narrower than the 85-fold range between samples, so depth is not confounded with diet; the normalisation of section 3 still has to be done, because it is the spread between samples that matters.

3. Bacteroides with 490,232 reads, then Faecalibacterium (203,095) and Phocaeicola (160,443). Join the taxonomy table to the row sums and group by the genus column.

4. 1,999, 1,267, 1,468 and 1,227 reads, about 5% to 8% of a typical sample (the median depth is 25,055). Reads in a blank are contamination from reagents or from the laboratory; features that are as abundant in the blanks as in the samples are suspect, and tools such as decontam use exactly this comparison. Never subtract blank counts by hand without recording it.

5. (c > 0).sum(axis=0).groupby(m.Diet).median(): Antibiotic 141, Fibre 172.5, Control 185.5. The antibiotic group has the fewest features, in line with its lower Shannon index. Use stats.kruskal on the three arrays, and a permutation test for the pair, and state the effect size in features.

6. def pielou(x): h = shannon(x); n = (np.asarray(x) > 0).sum(); return float("nan") if n < 2 else h / math.log(n). Tests: pielou([5, 5, 5]) == pytest.approx(1.0) (perfectly even), math.isnan(pielou([10, 0, 0])) (undefined for one feature, as the logarithm of 1 is zero), and pielou([1, 2, 3]) == pytest.approx(pielou([10, 20, 30])).

Check your understanding

Sequencing depth ranges from 950 to 80,357 reads. Why not compare raw counts between samples?

A feature has more reads in a deep sample simply because more was sequenced. Convert to relative abundance or a log-ratio scale, or use a model that estimates size factors, before comparing.

119 of 300 features have p < 0.05. How many do you report as different?

The number that survives false-discovery correction, 72 at a 5% FDR. With 300 tests about 15 would pass 0.05 by chance.

A cross-validated AUC is 0.91 with folds from 0.70 to 1.0. What do you write?

That the mean AUC was 0.91 (range 0.70–1.0 over five folds), and that with 72 samples the estimate is imprecise. Also say whether subjects were kept within one fold.

Why is the first principal axis only 13.5% of the variation yet useful?

With hundreds of features the variation is spread across many axes. A group separation on the first two axes is a strong signal even though they explain little in total.

Key references

  1. Gloor GB, Macklaim JM, Pawlowsky-Glahn V, Egozcue JJ. Microbiome datasets are compositional: and this is not optional. Frontiers in Microbiology 8, 2224 (2017).
  2. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B 57, 289–300 (1995).
  3. McKinney W. Data structures for statistical computing in Python. Proceedings of the 9th Python in Science Conference, 56–61 (2010).
  4. Harris CR, Millman KJ, van der Walt SJ, et al. Array programming with NumPy. Nature 585, 357–362 (2020).
  5. Wilson G, Aruliah DA, Brown CT, et al. Best practices for scientific computing. PLoS Biology 12, e1001745 (2014).
  6. McKinney W. Python for Data Analysis. O’Reilly. Read: Ch. 5, pandas; ch. 7, data cleaning; ch. 8, joining and reshaping; ch. 10, group operations.
  7. Wickham H. Tidy data. Journal of Statistical Software 59, 1–23 (2014).
  8. Love MI, Huber W, Anders S. Moderated estimation of fold change and dispersion for RNA-seq data with . Genome Biology 15, 550 (2014).
  9. Law CW, Chen Y, Shi W, Smyth GK. : precision weights unlock linear model analysis tools for RNA-seq read counts. Genome Biology 15, R29 (2014).
  10. Ambroise C, McLachlan GJ. Selection bias in gene extraction on the basis of microarray gene-expression data. PNAS 99, 6562–6566 (2002).
  11. Varoquaux G. Cross-validation failure: small sample sizes lead to large error bars. NeuroImage 180, 68–77 (2018).
  12. Hunter JD. Matplotlib: a 2D graphics environment. Computing in Science & Engineering 9, 90–95 (2007).
  13. Waskom ML. seaborn: statistical data visualization. Journal of Open Source Software 6, 3021 (2021).
  14. Wilson G, Bryan J, Cranston K, et al. Good enough practices in scientific computing. PLoS Computational Biology 13, e1005510 (2017).
  15. Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011).
  16. James G, Witten D, Hastie T, Tibshirani R, Taylor J. An Introduction to Statistical Learning with Applications in Python. Springer (2023). Read: ch. 2 statistical learning, ch. 3 linear regression, ch. 4 classification, ch. 5 resampling, ch. 6 regularisation, ch. 8 trees.
  17. Faraway JJ. Extending the Linear Model with R. Chapman & Hall/CRC. Read: ch. 2 binary response, ch. 5 count regression, ch. 10 random effects, ch. 11 repeated measures.
  18. Xia Y, Sun J, Chen D-G. Statistical Analysis of Microbiome Data with R. Springer (2018). Read: ch. 5 power and sample size, ch. 6 diversity, ch. 7 ordination, ch. 8 univariate community analysis.
Core skills · Programming and data

Data science with R: the same analysis, with vegan and ggplot2

R is where statistics is done first, and where ecological and genomic methods are published. This chapter repeats the analysis of the previous chapter in R, on the same files, so that you can compare the two languages line by line, and adds what R does best: formal community statistics (PERMANOVA and dispersion), tidy modelling and graphics. Everything shown was run in R 4.6 with the packages vegan and ggplot2; the packages of the tidyverse and Bioconductor are described but not run here, and marked as such.

1. Load and check

suppressPackageStartupMessages({library(vegan); library(ggplot2)})
counts <- read.delim("counts.tsv", row.names = 1, check.names = FALSE)   # features x samples
meta   <- read.delim("metadata.tsv", row.names = 1, stringsAsFactors = FALSE)
stopifnot(all(colnames(counts) == rownames(meta)))          # same samples, same order
dim(counts)                    # 377  76
table(meta$Sample_type)        # Blank 4, Sample 72
table(meta$Diet)               # Antibiotic 24, Blank 4, Control 24, Fibre 24

Two R-specific cautions. check.names = FALSE stops R from "repairing" sample names with symbols, which would make them differ from the metadata and fail the check. And factor levels are alphabetical by default, so set the reference level yourself: m$Diet <- factor(m$Diet, levels = c("Control", "Fibre", "Antibiotic")). In every model, the first level is the reference and the other coefficients are differences from it.

bio <- rownames(meta)[meta$Sample_type == "Sample"]
cnt <- counts[, bio]; m <- meta[bio, ]
m$Diet <- factor(m$Diet, levels = c("Control", "Fibre", "Antibiotic"))
c(nrow(cnt), ncol(cnt), sum(cnt))                  # 377 72 2109697
summary(colSums(cnt))        # depth per sample: min 950, median about 25,000, max about 80,000

The depth varies about 85-fold between samples, so counts must be put on a common scale before samples are compared, exactly as in the Python chapter.

2. Filter, normalise, transform

prev <- rowMeans(cnt > 0)
cnt  <- cnt[prev >= 0.10, ]; nrow(cnt)          # 300 features kept
rel  <- sweep(cnt, 2, colSums(cnt), "/")          # relative abundance
clr  <- log(as.matrix(cnt) + 0.5)
clr  <- sweep(clr, 2, colMeans(clr))              # centred log-ratio

sweep(x, 2, v, "/") applies an operation between every column of x and the vector v: here, dividing each sample by its total. The result is identical to the NumPy version: the same 300 features, the same depth range (949 to 80,278 after filtering), the same transformation.

3. Diversity and a group comparison

sh <- diversity(t(cnt), "shannon")                 # vegan wants samples in rows
round(tapply(sh, m$Diet, median), 3)
#    Control      Fibre Antibiotic
#      4.252      4.143      3.771
kruskal.test(sh ~ m$Diet)      # chi-squared = 22.666, df = 2, p = 1.197e-05
pairwise.wilcox.test(sh, m$Diet, p.adjust.method = "BH")
#            Control Fibre
# Fibre      0.0045  -
# Antibiotic 3e-05   0.0028

The medians (4.252, 4.143, 3.771) and the Kruskal–Wallis test are the same as in Python (22.67, p = 1.2e-05), which is what should happen: two correct implementations of the same method agree. The pairwise table gives Benjamini–Hochberg adjusted p values for every pair, so all three diets differ in Shannon diversity at the 5% level, with the antibiotic group lowest. vegan expects samples in rows (hence t(cnt));  and most expression tools expect genes in rows. Check the orientation of every function you use.

4. Thousands of tests

x <- clr[, m$Diet == "Antibiotic"]; y <- clr[, m$Diet == "Control"]
pv <- sapply(rownames(clr), function(i) wilcox.test(x[i, ], y[i, ])$p.value)
q  <- p.adjust(pv, "BH")
c(sum(pv < 0.05), sum(q < 0.05))          # 119  74

119 raw p values below 0.05, as in Python; 74 after correction in R against 72 in Python. The difference is real and instructive: the two programs handle tied values in the Wilcoxon / Mann–Whitney test differently (an exact test or a normal approximation with a continuity correction), which moves a few borderline p values across the line. Neither is wrong. It is a reminder that a count of "significant features" is a property of the method as well as the data, so report the method and look at the effect sizes, not only the count.

5. A linear model

fit <- lm(sh ~ Diet + Fibre_g_day + BMI, data = m)
round(coef(summary(fit)), 3)
#                Estimate Std. Error t value Pr(>|t|)
# (Intercept)       4.222      0.251  16.845    0.000
# DietFibre        -0.067      0.106  -0.626    0.534
# DietAntibiotic   -0.480      0.081  -5.915    0.000
# Fibre_g_day      -0.006      0.005  -1.228    0.224
# BMI               0.005      0.009   0.575    0.567

The coefficients match the Python model (antibiotic −0.480 against control). Read the table by row: Estimate is the change in the outcome per unit (or against the reference level), Std. Error its uncertainty, t value the ratio, and Pr(>|t|) the p value of the null hypothesis that the coefficient is zero. Plot the residuals with plot(fit) before trusting the table. For repeated measures on the same subject, use a mixed model (nlme::lme(sh ~ Diet, random = ~ 1 | Subject), run in section 12), which models the correlation between samples of one person that this simple model ignores.

6. Ordination, PERMANOVA and dispersion

Principal components on the CLR values, and a PERMANOVA on Bray–Curtis distances of the relative abundances:

pc <- prcomp(t(clr))
round(summary(pc)$importance[2, 1:2], 3)            # PC1 0.135  PC2 0.037  (as in Python)

d <- vegdist(t(rel), "bray"); set.seed(1)
adonis2(d ~ Diet, data = m, permutations = 999)
#          Df SumOfSqs      R2      F Pr(>F)
# Model     2    2.580 0.14936 6.0576  0.001 ***
# Residual 69   14.694 0.85064
# Total    71   17.274 1.00000
bd <- betadisper(d, m$Diet)
round(anova(bd)[1, "Pr(>F)"], 4)                    # 0.1187

Read it. Diet explains 14.9% of the variation in community composition (R2 = 0.149), a moderate effect by microbiome standards, and it is significant: only 1 of 999 random relabellings gave an F as large (the smallest possible p value). The test of dispersion is not significant (p = 0.12), so the groups do not differ detectably in how spread out they are, which means the PERMANOVA result reflects a difference in the centre of the groups and not only in their spread. If the dispersion test had been significant, you would have to say that the groups differ in variability and treat the PERMANOVA with caution.

7. A figure

df <- data.frame(PC1 = pc$x[, 1], PC2 = pc$x[, 2], Diet = m$Diet)
p <- ggplot(df, aes(PC1, PC2, colour = Diet)) +
  geom_point(size = 2.5) +
  scale_colour_manual(values = c("#4477aa", "#228833", "#cc3311")) +
  labs(x = "PC1 (13.5%)", y = "PC2 (3.7%)") +
  theme_minimal()
ggsave("pca.pdf", p, width = 5, height = 4)       # a vector file; PDF worked on the test machine

ggplot2 builds a figure in layers on a "grammar": data (df), aesthetic mappings (aes: which column is x, y, colour), geometries (geom_point), scales (scale_colour_manual), labels and a theme. This is why a change such as "colour by sex instead of diet" is one word. Facets (facet_wrap(~ Batch)) split a figure by a variable, and geom_boxplot() with geom_jitter() shows summary and points together. Put the axis percentages in the labels, as above, because they tell the reader how much the picture explains.

8. The tidyverse and Bioconductor versions

This is how the same table work is written in the tidyverse, which many groups prefer for readability. Section 10 runs it and shows the results; the code here is the reshaping in one chain:

library(dplyr); library(tidyr)
long <- counts |>
  tibble::rownames_to_column("feature") |>
  pivot_longer(-feature, names_to = "sample", values_to = "count") |>
  left_join(tibble::rownames_to_column(meta, "sample"), by = "sample") |>
  filter(Sample_type == "Sample")
long |> group_by(Diet) |> summarise(reads = sum(count), features = n_distinct(feature[count > 0]))

The pipe |> passes the result of one step to the next, so the code reads as a sequence of verbs: reshape, join, filter, group, summarise. For omics data, Bioconductor adds objects that keep the count matrix, the feature table and the sample table together (SummarizedExperiment, phyloseq) and packages for models of counts (, , ) (Huber et al., 2015). Base R, tidyverse and Bioconductor combine well; use the one that makes the step clearest.

9. The R language you need

The analysis above used R as a calculator with packages. To read other people's scripts and to write your own functions you need the language itself. Gentleman's R Programming for Bioinformatics is the reference for the details (Read: ch. 2, language fundamentals; ch. 3, object-oriented programming; ch. 9, debugging and profiling). Every output below was produced by running the code.

Vectors: the basic object

x <- c(4.2, 3.8, NA, 5.1)         # a numeric vector with one missing value
typeof(x)                          # "double"
mean(x)                            # NA: a missing value makes the mean missing
mean(x, na.rm = TRUE)              # 4.366667: you must ask to drop it
is.na(x)                           # FALSE FALSE TRUE FALSE
1:10 %% 3 == 0                     # a test on every element: FALSE FALSE TRUE FALSE FALSE TRUE ...
seq(0, 1, by = 0.25)               # 0.00 0.25 0.50 0.75 1.00
rep(c("a", "b"), times = c(2, 3))  # "a" "a" "b" "b" "b"
v <- c(a = 1, b = 2, c = 3)
v["b"]; v[-1]; v[v > 1]            # by name; all but the first; by a condition

In R there is no single number: everything is a vector, and operations apply to every element at once (this is R's "vectorisation", the counterpart of NumPy's). A vector has one type; mixing types coerces to the most general one: c(1, "a", TRUE) gives "1" "a" "TRUE", all text. NA is contagious, deliberately, so that a missing value cannot disappear silently. Indexing starts at 1, a negative index drops elements, and a logical vector selects the TRUE ones.

Two numeric facts that catch everyone: integer division is 5 %/% 2 (2) and the remainder 5 %% 2 (1); and decimals are stored approximately, so 0.1 + 0.2 == 0.3 is FALSE. Compare with isTRUE(all.equal(0.1 + 0.2, 0.3)), which is TRUE.

Factors, lists and data frames

d <- factor(c("Fibre", "Control", "Antibiotic", "Control"))
levels(d)                          # "Antibiotic" "Control" "Fibre": alphabetical by default
as.integer(d)                      # 3 2 1 2: the codes underneath
levels(relevel(d, ref = "Control"))   # "Control" "Antibiotic" "Fibre": Control is now the reference
l <- list(name = "run1", reads = c(10, 20), ok = TRUE)
l$reads[2]                         # 20: a list holds objects of any type; $ takes one out by name
str(l)                             # a compact description of any object: use it constantly

A factor is a categorical variable stored as integer codes with labels. It is what models and plots use for groups, and the first level is the reference in every model. A list holds anything. A data frame is a list of equal-length columns, which is why meta$Diet and meta[["Diet"]] both work.

meta[meta$Sample_type == "Sample" & meta$Age > 40, "Subject"][1:3]      # rows by condition, one column: "P02" "P02" "P02"
sapply(meta[, c("Age", "BMI", "Butyrate_mM")], function(z) round(mean(z, na.rm = TRUE), 2))
#         Age         BMI Butyrate_mM
#       44.50       24.65       21.57
tapply(meta$Butyrate_mM, meta$Diet, median, na.rm = TRUE)       # a summary per group
# Antibiotic      Blank    Control      Fibre
#      8.735         NA     20.265     33.375
aggregate(Age ~ Diet, data = meta, FUN = mean)                  # the same with a formula, returns a data frame
vapply(split(meta$BMI, meta$Diet), median, numeric(1), na.rm = TRUE)   # split into pieces, apply, and check the result type

The apply family replaces loops: sapply applies a function to each element and simplifies the result, tapply and aggregate do it by group, vapply is the safe form because you declare the type of the result. The NA printed for the blanks is correct: the four blanks have no butyrate measurement, and na.rm = TRUE removes missing values within a group but cannot invent a value for a group that has none.

Write functions

shannon <- function(x) { p <- x[x > 0] / sum(x); -sum(p * log(p)) }
sh_all <- apply(counts[, bio], 2, shannon)          # apply by column (2); rows would be 1
round(head(sh_all, 3), 3)                           # P01_Day0 4.291  P01_Day14 4.274  P01_Day28 4.313
safe_log <- function(x, pc = 0.5) { stopifnot(is.numeric(x)); log(x + pc) }    # a check, and a default argument
tryCatch(safe_log("a"), error = function(e) conditionMessage(e))                # "is.numeric(x) is not TRUE"
rarefy_to <- function(depth) function(x) min(x, depth)      # a function that returns a function (a closure)
rarefy_to(100)(250)                                          # 100

A function returns the value of its last expression, takes arguments with defaults, and can refuse bad input with stopifnot or stop. tryCatch turns an error into a value you can handle, which is how a script processing a hundred samples survives one broken file and says which one. A function that returns a function is a closure; the inner function remembers depth. This is the idea behind the functions you pass to sapply and to the packages' options.

Objects and methods (S3)

summary.runinfo <- function(object, ...) cat("Run", object$name, "with", length(object$reads), "samples\n")
r <- structure(list(name = "run1", reads = 1:3), class = "runinfo")
summary(r)                          # Run run1 with 3 samples

This is why summary(fit), plot(fit) and print(fit) do something different for every kind of object: the generic function looks at the object's class and calls the matching method (summary.lm, summary.runinfo). Bioconductor uses the stricter S4 system for objects such as SummarizedExperiment, but reading S3 is enough to understand most code.

10. The tidyverse in practice

This section's code was run with dplyr, tidyr, stringr, purrr and readr (R 4.4.3). The tidyverse is a family of packages that share one design: functions take a data frame first, return a data frame, and chain with the pipe |> (Wickham, 2014).

library(dplyr); library(tidyr); library(stringr); library(purrr); library(tibble)
tax <- read.delim("taxonomy.tsv", header = FALSE, row.names = 1)
lin <- str_split_fixed(tax$V2, ";", 7)                        # split the lineage into 7 columns
colnames(lin) <- c("kingdom","phylum","class","order","family","genus","species")
lin <- as_tibble(lin) |> mutate(across(everything(), ~ str_sub(.x, 4))) |> mutate(feature = rownames(tax), .before = 1)
lin |> count(phylum, sort = TRUE) |> head(4)
#   phylum               n
# 1 Firmicutes         196
# 2 Bacteroidota        92
# 3 Actinobacteriota    31
# 4 Proteobacteria      29

The same parsing as in the Python chapter, written as a sequence of verbs. mutate adds or changes columns, across(everything(), ...) applies a function to all columns, count tabulates. Now the table most analyses need, the share of each phylum by diet:

cnt  <- counts[, bio]
long <- cnt |> rownames_to_column("feature") |>
  pivot_longer(-feature, names_to = "sample", values_to = "count") |>                  # wide to long: 27,144 rows
  left_join(rownames_to_column(meta, "sample")[, c("sample", "Diet")], by = "sample") |>
  left_join(lin[, c("feature", "phylum")], by = "feature")
res <- long |> group_by(Diet, phylum) |> summarise(reads = sum(count), .groups = "drop") |>
  group_by(Diet) |> mutate(share = round(100 * reads / sum(reads), 1)) |>
  filter(phylum %in% c("Firmicutes", "Bacteroidota"))
res |> select(Diet, phylum, share) |> pivot_wider(names_from = phylum, values_from = share) |>
  mutate(FB = round(Firmicutes / Bacteroidota, 2))
#   Diet       Bacteroidota Firmicutes    FB
# 1 Antibiotic         59.5       21.6  0.36
# 2 Control            41.4       44.1  1.07
# 3 Fibre              31.1       54.2  1.74

Read the result as biology: the Firmicutes-to-Bacteroidota ratio is 1.07 in the controls, rises to 1.74 on the fibre diet and collapses to 0.36 on antibiotics, where Bacteroidota make up 60% of the reads. The code made one thing explicit that is easy to miss: pivot_longer turns the 377 × 72 table into 27,144 rows, one per feature and sample, which is the shape that group_by and ggplot2 want. The Python equivalent is melt and groupby (Python chapter, section 10). purrr::map_dbl(split(x, g), mean) applies a function to each group and returns numbers; stringr has str_extract("P01_Day14", "Day[0-9]+") (gives "Day14"), str_detect and str_replace for patterns.

Reading files carefully

readr::read_csv(file)          # guesses each column's type and says so; a column of "1", "NA", "3" becomes numeric
sapply(read.csv(file), class)  # base R on the same file: id "character", val "integer"

Always check the types after reading: a stray text value turns a numeric column into text, a sample called 1E5 can be read as a number, and Excel has famously turned gene names into dates. stringsAsFactors, check.names, na.strings and colClasses are the arguments that control this.

11. Classical tests, power, permutation and bootstrap

m$Diet <- factor(m$Diet, levels = c("Control", "Fibre", "Antibiotic"))
a <- sh[m$Diet == "Control"]; b <- sh[m$Diet == "Antibiotic"]
t.test(a, b)                         # Welch: t = 5.0138, df = 27.069, p = 2.915e-05; mean difference 0.478, 95% CI 0.282 to 0.674
t.test(sh[m$Diet == "Fibre"], a)     # Fibre against Control alone: p = 0.0061, difference -0.112, 95% CI -0.190 to -0.034
wilcox.test(a, b, exact = FALSE)     # W = 491, p = 2.974e-05
summary(aov(sh ~ Diet, data = m))    # F = 19.02 on 2 and 69 df, p = 2.63e-07
TukeyHSD(aov(sh ~ Diet, data = m))
#                        diff        lwr        upr     p adj
# Antibiotic-Control -0.4781 -0.6723 -0.2839 0.0000004
# Fibre-Control      -0.1118 -0.3061  0.0824 0.3572431
# Fibre-Antibiotic    0.3662  0.1720  0.5605 0.0000743
var.test(a, b)$p.value               # 1.9e-07: the variances differ (R's t.test does Welch by default for this reason)
power.t.test(delta = 0.5, sd = 1, power = 0.8)       # n = 63.8 per group
chisq.test(table(m$Sex, m$Diet))     # X-squared = 3.2, p = 0.20: no imbalance of sex between diets
cor.test(m$Butyrate_mM, m$Fibre_g_day, method = "spearman", exact = FALSE)    # rho = 0.507, p = 5.5e-06

The results are identical to those of the Python chapter (t = 5.01, p = 2.9e-05; U/W = 491; power n = 63.8; rho = 0.507), which is the best test of a data scientist's own work. The Tukey table is the right follow-up to a one-way analysis of variance: it compares all pairs and controls the error over the family. Read alone, it says that Antibiotic differs from both other diets and that Fibre does not differ from Control (adjusted p = 0.36). That last statement does not survive a second look: the pairwise Wilcoxon test of section 3 gave an adjusted p of 0.0045 for Fibre against Control, and a Welch test on that pair alone gives p = 0.006 (difference −0.11, 95% CI −0.19 to −0.03). The cause is variance. Tukey's method uses one pooled error from all three groups, and the antibiotic group, whose standard deviation is three times as large as the others, inflates it, hiding a small and consistent difference between two tight groups. Tests that assume equal variance are not neutral when the variances differ. Choose the test for the data, and when methods disagree, say so and show the plot. The chi-squared test checks the design: sex is balanced across the diets, so it cannot explain a diet effect. The Spearman correlation says that butyrate rises with fibre intake across samples, which is the biology the study was built around.

set.seed(1)
obs  <- mean(a) - mean(b);  pool <- c(a, b)
perm <- replicate(10000, { s <- sample(pool); mean(s[seq_along(a)]) - mean(s[-seq_along(a)]) })
(sum(abs(perm) >= abs(obs)) + 1) / (10000 + 1)                # 9.999e-05
bs <- replicate(5000, mean(sample(a, replace = TRUE)) - mean(sample(b, replace = TRUE)))
round(quantile(bs, c(.025, .975)), 3)                           # 0.297 0.673

set.seed(1) makes the random draws repeat; without it nobody, including you next week, can reproduce the p value. replicate(n, expr) runs an expression n times and collects the results, the compact way to write a simulation. The permutation p value cannot be smaller than 1/(n+1) (here 0.0001); report it as p < 0.0001, never as 0. The bootstrap interval, 0.30 to 0.67 Shannon units, agrees with the t-test interval (0.28 to 0.67): the assumptions of the t-test were good enough.

12. Models, in more detail

A linear model, its diagnostics and its contrasts

fit <- lm(sh ~ Diet + Age + Sex, data = m)
round(coef(fit), 3)       # (Intercept) 4.052  DietFibre -0.104  DietAntibiotic -0.466  Age 0.004  SexM 0.007
round(summary(fit)$r.squared, 3)                    # 0.376
anova(fit)                # sequential tests: Diet F = 19.09, p = 2.8e-07; Age p = 0.14; Sex p = 0.93
shapiro.test(residuals(fit))$p.value                # 0.0033: the residuals are not normal
library(emmeans)
emmeans(fit, pairwise ~ Diet, adjust = "tukey")$contrasts
#  contrast             estimate    SE df t.ratio p.value
#  Control - Fibre         0.104 0.0837 67   1.246  0.4307
#  Control - Antibiotic    0.466 0.0823 67   5.667  <.0001
#  Fibre - Antibiotic      0.362 0.0814 67   4.446  0.0001

The coefficient for the antibiotic diet, −0.466, is the Python result (statsmodels gave −0.466 for the same formula). emmeans turns a model into the comparisons a reader wants, adjusted for the other variables and for the number of comparisons, with the groups averaged over the levels of the covariates (these tests share the pooled-variance weakness described above, so the Control–Fibre line, p = 0.43, should not be read as "no difference"); every result in a paper that says "adjusted for age and sex" should come from something like this. The normality of the residuals fails (p = 0.003), driven by the very variable antibiotic group; with 24 samples per group the estimates of the means are still sound, the standard errors are the weak point, which is why the permutation and rank-based results above matter. The sequential anova adds the terms in order, so the answer for Age depends on Diet having been entered first; with orthogonal groups, as here, it hardly matters, but in unbalanced data use the marginal tests (car::Anova(fit, type = 2) or drop1).

Counts

library(MASS)
dd <- data.frame(y = unlist(cnt["ASV0001", ]), Diet = m$Diet, depth = colSums(cnt))
pg <- glm(y ~ Diet + offset(log(depth)), family = poisson, data = dd)
sum(residuals(pg, "pearson")^2) / pg$df.residual            # 1290: far above 1; Poisson is wrong
nb <- glm.nb(y ~ Diet + offset(log(depth)), data = dd)
round(exp(coef(nb)), 3)       # (Intercept) 0.046  DietFibre 0.377  DietAntibiotic 2.730   (rate ratios against Control)
round(nb$theta, 2)            # 0.94
round(summary(nb)$coefficients[, 4], 5)                     # p values: Fibre 0.00106, Antibiotic 0.00074

The negative binomial model, with sequencing depth as an offset, gives exactly the rate ratios and p values that the Python chapter got with statsmodels (2.730 and 0.377; p = 0.0007 and 0.001). When two languages agree to three decimals you can be confident that you understood the model and not only the function. glm(..., family = binomial) is the logistic regression: glm(ab ~ shannon, family = binomial, data = dl) gives the intercept 25.07, the slope −6.158 and the odds ratio exp(−6.158) = 0.002 per Shannon unit.

Repeated measures with a mixed model

library(nlme)
mm <- lme(sh ~ Diet * Day, random = ~ 1 | Subject, data = m)        # a random intercept for every person
round(summary(mm)$tTable, 3)
#                    Value Std.Error DF t-value p-value
# (Intercept)        4.230     0.087 45  48.601   0.000
# DietFibre         -0.002     0.123 21  -0.017   0.987
# DietAntibiotic    -0.255     0.123 21  -2.074   0.051
# Day                0.000     0.004 45   0.061   0.952
# DietFibre:Day     -0.008     0.006 45  -1.388   0.172
# DietAntibiotic:Day -0.016    0.006 45  -2.818   0.007
anova(mm)                   # Diet F = 14.06 (p = 0.0001); Day p = 0.0017; Diet:Day F = 3.97 (p = 0.026)
intervals(mm, which = "fixed")$fixed["DietAntibiotic:Day", ]        # -0.027  -0.016  -0.005
VarCorr(mm)                 # between subjects 0.0189 (sd 0.137), residual 0.0500 (sd 0.224)

Each person gets their own baseline, and the degrees of freedom show what the model understands: 21 for a diet effect (24 subjects minus the three group means) and 45 for the effects of time, not 69 as a model that treated the samples as independent would assume. The Diet:Day interaction (F = 3.97, p = 0.026) says that the groups change differently over time, and the interval for the antibiotic slope, −0.027 to −0.005 per day, quantifies it. nlme ships with R; the lme4 package (lmer(sh ~ Diet * Day + (1 | Subject))) is the usual alternative and the one in most tutorials (Bates et al., 2015). Faraway's Extending the Linear Model with R (Read: ch. 10 random effects, ch. 11 repeated measures) explains the models and their assumptions.

13. Community analysis in depth with vegan

The PERMANOVA of section 6 is the starting point. Four questions decide whether it is valid and what it means, and they are the questions a reviewer asks (Anderson, 2001).

What are the units that can be permuted?

d <- vegdist(t(rel), "bray")
adonis2(d ~ Diet, data = m, permutations = 999)                    # R2 = 0.149, F = 6.06, p = 0.001
h <- how(plots = Plots(strata = m$Subject, type = "free"), within = Within(type = "none"), nperm = 999)
a2 <- adonis2(d ~ Diet, data = m, permutations = h)               # R2 = 0.1494, p = 0.001
adonis2(d ~ Diet, data = m, permutations = how(blocks = m$Subject, nperm = 999))$`Pr(>F)`[1]    # 1

The three lines answer the question of the repeated measures. The first treats the 72 samples as exchangeable, which they are not (three come from one person). The second permutes whole subjects as units, keeping each person's three samples together, which is the valid test for a diet that was given to a person and not to a sample; it gives the same result, R2 = 0.149 and p = 0.001 (999 permutations cannot give less). The third is a trap: blocks = Subject permutes samples within each person, but diet does not vary within a person, so every permutation reproduces the observed data and the p value is exactly 1. A p value of 1 here does not mean "no effect"; it means the test was set up with the wrong structure. When a result is surprising, check that the permutation scheme matches the design.

Is something else the explanation?

adonis2(d ~ Diet + Sex + Age, data = m, by = "margin", permutations = 999)
#      Df SumOfSqs    R2     F Pr(>F)
# Diet  2    2.545 0.147 6.183  0.001
# Sex   1    0.414 0.024 2.011  0.010
# Age   1    0.465 0.027 2.261  0.009
table(m$Batch, m$Diet)
#        Control Fibre Antibiotic
#   Run1       9    12         15
#   Run2      15    12          9

by = "margin" tests each term after all the others, so the diet effect (R2 = 0.147) is adjusted for sex and age, and it hardly changes. Sex and age have small but nominally significant effects (about 2% of the variation each); these p values come from free permutations and are therefore too optimistic for the same reason as above. The table is the more important result: the two sequencing runs are not balanced across the diets (15 antibiotic samples in Run 1 but only 9 in Run 2, the reverse for controls). A batch effect would therefore be partly confounded with diet. A good analysis tests the batch (adonis2(d ~ Batch + Diet, ...)), plots the samples coloured by batch, and, if the batch matters, includes it in the model as a covariate or a stratum. Confounding between a technical variable and the question cannot be repaired afterwards by statistics alone, only reduced; it is prevented by randomising samples to runs.

Which measured variables go with the ordination?

pc <- rda(t(log(as.matrix(cnt) + 0.5)))
ef <- envfit(pc, m[, c("Butyrate_mM", "Fibre_g_day", "BMI")], permutations = 999)
round(ef$vectors$r, 3)      # Butyrate_mM 0.792  Fibre_g_day 0.400  BMI 0.147

envfit regresses each variable on the ordination axes. Butyrate explains 79% of the variation along its best direction in the plot, much more than fibre intake (40%) or BMI (15%): the butyrate gradient is a strong organising axis of this community (the p values, 0.001, 0.001 and 0.002, again assume exchangeable samples). Use it to describe a plot, and confirm anything important with a model.

Rarefaction, and why it is debated

set.seed(1)
rc <- rrarefy(t(cnt), 900)                       # subsample every sample to 900 reads (the smallest sample has 949)
tapply(rowSums(rc > 0), m$Diet, median)          # Control 119.5  Fibre 114.0  Antibiotic 95.0
tapply(rarefy(t(cnt), 900), m$Diet, median)      # the expected value, without random sampling: 122.0  112.2  93.5

Rarefying cuts every sample down to the same number of reads so that richness can be compared. It throws data away (here all but 900 reads of samples with up to 80,000), which is why it is controversial for testing differences between groups; for diversity indices and richness, the expected value computed by rarefy() is better than one random draw, as it has no sampling noise. The conclusion does not change: richness is lowest in the antibiotic group. Asking rarefy(t(cnt), 2000) fails with a warning, because the smallest sample (949 reads) cannot supply 2,000: the function does not silently do something else, and you should read warnings.

Groups that were not given in advance

hc <- hclust(d, "average"); table(cutree(hc, 3), m$Diet)
#    Control Fibre Antibiotic
# 1       24    18          7
# 2        0     6          0
# 3        0     0         17

Hierarchical clustering into three groups isolates 17 of the 24 antibiotic samples in one cluster; all 24 controls fall in the large cluster, together with 18 fibre samples and 7 antibiotic samples. It agrees with the PERMANOVA: the antibiotic effect is the strongest, and the fibre effect is subtler.

14. Counts at scale: -, and what repeated measures do

Testing 300 features one by one with a Wilcoxon test is a start; testing them with a model that shares information across features is better, and it is what the dedicated packages do. with converts counts to log counts per million, estimates how the variance depends on the mean and gives every observation a weight, then fits a linear model per feature with moderated statistics (Law et al., 2014; Ritchie et al., 2015).

library()
design <- model.matrix(~ Diet, data = m)                 # Control is the reference
v   <- (cnt, design)                                   # counts to weighted log-CPM
fit <- eBayes(lmFit(v, design))
sum(topTable(fit, coef = "DietAntibiotic", number = Inf)$adj.P.Val < 0.05)     # 87 features at FDR 5%
sum(topTable(fit, coef = "DietFibre", number = Inf)$adj.P.Val < 0.05)          # 14

dup  <- duplicateCorrelation(v, design, block = m$Subject)     # 0.195: samples of one person are correlated
fit2 <- eBayes(lmFit(v, design, block = m$Subject, correlation = dup$consensus.correlation))
sum(topTable(fit2, coef = "DietAntibiotic", number = Inf)$adj.P.Val < 0.05)    # 52
sum(topTable(fit2, coef = "DietFibre", number = Inf)$adj.P.Val < 0.05)         # 1

This is the most important result of the chapter. The first model treats the 72 samples as independent: 87 features differ with antibiotics and 14 with fibre. Once the model knows that three samples come from one person (the within-person correlation is 0.195), the numbers fall to 52 and 1. The count for antibiotics fell by 40% and the count for fibre from 14 to 1: a large part of the first answer came from counting the same 24 people three times. This is not a technicality: an analysis that ignores repeated measures reports too many findings, and here would have overstated the effect of fibre.

Two further remarks. The Wilcoxon test on CLR values (section 4) found 74 features and -; they share only 39 features. Differential-abundance methods disagree strongly in published benchmarks as well (Nearing et al., 2022), so a careful report uses the method that matches the design, states it, and trusts the features that several methods agree on (this is the idea behind the consensus in the MicroRes tool). Second,  and  are the standard negative binomial tools for count data (Love et al., 2014); they were not installed on the machine used for this chapter, so they were not run, and the design formula ~ Diet with a subject block, or a model that includes the subject as a term for paired designs, is how they take the same structure into account.

15. When the code does not work: errors, speed, and R's own tools

f <- function(x) { if (x < 0) stop("x must not be negative: ", x); sqrt(x) }
tryCatch(f(-1), error = function(e) paste("caught:", conditionMessage(e)))      # "caught: x must not be negative: -1"
withCallingHandlers(as.integer("a"), warning = function(w) { message("warning seen: ", conditionMessage(w)); invokeRestart("muffleWarning") })
g <- function() { on.exit(cat("cleaned up\n")); stop("failed") }; try(g(), silent = TRUE)     # cleaned up

Reading an error is the first skill: the message names the function and often the cause, and traceback() after a failure shows the chain of calls. stop raises an error, warning a warning, message information; tryCatch handles them, and on.exit guarantees cleanup (closing a file, removing a temporary folder) even when the function fails. To look inside a function while it runs, put browser() in it, or call debugonce(f), and step through line by line.

x <- runif(2e6); y <- numeric(length(x))
system.time(for (i in seq_along(x)) y[i] <- x[i] * 2)    # 0.119 s: a loop of 2 million steps
system.time(y2 <- x * 2)                                  # 0.007 s: the same result, 17 times faster
v <- c(); system.time(for (i in 1:50000) v <- c(v, i))   # 1.8 s: growing a vector copies it every time
v2 <- numeric(50000); system.time(for (i in 1:50000) v2[i] <- i)   # 0.003 s: allocate once, fill in
Rprof(tf <- tempfile()); invisible(sapply(1:200, function(i) sort(runif(1e4)))); Rprof(NULL); summaryRprof(tf)$by.total

The two rules of fast R are to vectorise and to allocate once: growing a result inside a loop with c() is the classic mistake, 600 times slower here at 50,000 elements, and the cost grows with the square of the length. A modern R compiles simple loops, so the plain loop is only 17 times slower than the vectorised call and not hundreds. Measure before optimising (system.time, and Rprof for a profile of where the time goes), because the slow part is rarely where you expect.

16. Scripts, reproducibility and packages

#!/usr/bin/env Rscript
# summarise.R: Shannon index of every sample in a count table (features x samples)
args <- commandArgs(trailingOnly = TRUE)
if (length(args) < 1) stop("usage: summarise.R counts.tsv [out.tsv]")
out <- if (length(args) >= 2) args[2] else "shannon.tsv"
shannon <- function(x) { p <- x[x > 0] / sum(x); -sum(p * log(p)) }
t   <- read.delim(args[1], row.names = 1, check.names = FALSE)
res <- data.frame(sample = colnames(t), shannon = apply(t, 2, shannon))
write.table(res, out, sep = "\t", quote = FALSE, row.names = FALSE)
cat(ncol(t), "samples written to", out, "\n")
$ Rscript summarise.R counts.tsv shannon.tsv
76 samples written to shannon.tsv
$ Rscript summarise.R
Error: usage: summarise.R counts.tsv [out.tsv]

A script that takes its inputs from the command line and stops with a usage message when they are missing can be run in a loop, on a cluster, or from a workflow manager. The output file is identical to that of the Python script in the Python chapter (the same 76 Shannon values, 4.2913 for the first sample).

  • Record the environment: end every analysis with sessionInfo() (R, platform and the version of every loaded package) and save it with the results. The renv package gives a project its own library and a lock file, so that the same versions can be restored later; a container does the same for the whole system. Package behaviour changes between versions, and several results in this chapter would not be reproducible without knowing them.
  • Seeds: set.seed() at the top of any script that permutes, samples or starts an algorithm at random. set.seed(42); sample(1:10, 3) returns 1 5 10 every time.
  • Literate documents: R Markdown or Quarto combine text, code and results in one file that is rerun to produce a report, which avoids the error of pasting yesterday's figure next to today's data.
  • Packages: when you reuse a function in a second project, put it in a package (usethis::create_package, a DESCRIPTION file, roxygen comments and testthat tests). Gentleman's book (ch. 7) explains the structure, and it is the way Bioconductor software is built (Huber et al., 2015).
  • Libraries can fail: on the machine used to write this chapter, several packages (dplyr, lme4,  among them) would not load in the system R because they had evidently been built for an earlier R version, and the tidyverse code was run in a second R installation. Library mismatches are normal on shared servers; the remedies are to rebuild the packages, to use renv or a container, and to note which R was used.

17. Bioconductor objects: sequences, ranges and experiments

Bioconductor is the collection of R packages for genomic data. Its strength is a small number of shared objects, so that a tool written by one group works on the output of another (Huber et al., 2015). Three of them cover most uses. Everything here was run with Biostrings 2.72 and the GenomicRanges and SummarizedExperiment packages on the demonstration data.

Biostrings: sequences

library(Biostrings)
seqs <- readDNAStringSet("sequences.fasta")
length(seqs); summary(width(seqs))             # 377 sequences, every one 253 bases (the demonstration data are simulated)
subseq(seqs[[1]], 1, 30)                       # AGTGACTTAAGCACTATAGCCTAGTGCCAG
summary(as.numeric(letterFrequency(seqs, "GC", as.prob = TRUE)))     # GC fraction: 0.482 to 0.561, median 0.518
alphabetFrequency(seqs[1:3])[, 1:4]            # counts of A, C, G, T per sequence
reverseComplement(subseq(seqs[[1]], 1, 12))    # GCTTAAGTCACT
sum(vcountPattern("GTGCCAGC", seqs) > 0)       # 1 sequence contains the 8-base pattern exactly
start(matchPattern("GTGCCAGC", seqs[[1]], max.mismatch = 1))   # 24: found at position 24 when one mismatch is allowed
round(consensusMatrix(seqs[1:20], as.prob = TRUE)[c("A","C","G","T"), 1:5], 2)   # base frequencies at the first 5 positions
#   [,1] [,2] [,3] [,4] [,5]
# A 0.15 0.00 0.00 0.05    1
# G 0.00 0.90 0.00 0.95    0
# T 0.80 0.05 0.95 0.00    0

A DNAStringSet is a set of sequences that behaves like a vector: you subset it, measure widths, count letters and search for patterns in all of them at once, quickly, because the work is done in compiled code. consensusMatrix shows the first columns of an alignment-like set: position 2 is G in 90% of the first 20 sequences, position 3 is T in 95%, and position 5 is A in all of them (a conserved start, which is what primer sites look like). Pairwise and multiple alignment in Bioconductor lives in the pwalign and msa packages, which were not installed here, so no alignment is shown; the Biopython chapter showed alignment in Python. Gentleman's book (Read: ch. 5, working with character data) and the Bioinformatics with R Cookbook (Read: ch. 3, sequence analysis with R) cover these tasks.

GenomicRanges: intervals on a genome

library(GenomicRanges)
gr    <- GRanges("chr1", IRanges(c(100, 400, 800), width = c(200, 300, 150)), strand = c("+", "-", "+"), gene = c("g1", "g2", "g3"))
reads <- GRanges("chr1", IRanges(c(150, 350, 420, 900), width = 50))
countOverlaps(gr, reads)             # 1 1 1: reads falling in each gene
findOverlaps(reads, gr)              # read 1 with gene 1, read 3 with gene 2, read 4 with gene 3 (read 2 hits none)
start(flank(gr, 50, start = TRUE))   # 50 700 750: the 50 bases upstream of each gene, on the correct side of each strand

(A toy example with invented coordinates; no real genome is involved.) The last line is the reason this object exists: for a gene on the minus strand, "upstream" is at the higher coordinate, and flank knows that, whereas hand-written interval arithmetic often does not. Counting reads in genes, finding which variants fall in an exon, and extracting promoters are overlap problems, and findOverlaps solves them efficiently for millions of ranges.

SummarizedExperiment: the data and what describes it, together

library(SummarizedExperiment)
se <- SummarizedExperiment(assays  = list(counts = counts),                 # features x samples
                          colData = DataFrame(meta),                        # one row per sample
                          rowData = DataFrame(lineage = tax[rownames(counts), "V2"]))   # one row per feature
dim(se)                                   # 377 76
dim(se[, se$Diet == "Fibre"])             # 377 24: choosing samples by a metadata column subsets the counts and the metadata together
assay(se, "rel") <- sweep(assay(se, "counts"), 2, colSums(assay(se, "counts")), "/")     # a second assay
assayNames(se)                            # "counts" "rel"
keep <- rowSums(assay(se, "counts") > 0) / ncol(se) >= 0.10
dim(se[keep, ])                           # 300 76: the filter of section 2, applied to every part of the object at once

The count table, the sample table and the feature table sit in one object whose dimensions are tied together. Subsetting the samples subsets the counts and the metadata at once, so the mismatch of "sample order in the counts is not the order in the metadata" cannot happen. That error has produced wrong published results; avoiding it by construction is the main reason to use these objects. ,  and  accept a SummarizedExperiment, as do many single-cell tools, and phyloseq is the equivalent for microbiome data.

18. Why the statistics work: learning from simulation

The same simulations as in the Python chapter (section 19), written with replicate. R's random numbers differ from NumPy's, so the values differ slightly in the third decimal; the conclusions are the same.

set.seed(7); n <- 24; B <- 10000
tp <- function(delta) replicate(B, t.test(rnorm(n), rnorm(n, delta))$p.value)
mean(tp(0)  < 0.05)      # 0.0513: with no difference, about 5% of experiments are "significant"
mean(tp(.5) < 0.05)      # 0.3945: power of 24 per group for a medium effect, about 39%
p <- sapply(1:300, function(i) t.test(rnorm(n), rnorm(n, if (i <= 60) 1 else 0))$p.value)    # 60 real differences, 240 null
c(sum(p < 0.05), sum(p[61:300] < 0.05))                 # 73 hits, 18 of them false
q <- p.adjust(p, "BH");  c(sum(q < 0.05), sum(q[61:300] < 0.05))    # 50 hits, 3 false

Without correction, 18 of 73 hits are false (25% in this run; the Python run gave 9 of 61, and the figure varies from run to run); with the Benjamini–Hochberg correction, 3 of 50 (6%). The point is the same as in Python: power is limited at this sample size, and without multiplicity control a screen of many features reports many false discoveries. The design in section 11 and the model in section 14 are the tools to keep that under control.

19. Python or R?

On this analysis they give the same answers where the methods are the same. Choose by the question and the team: R has the larger collection of statistical methods for ecology and omics, and its model formulas (y ~ a + b) and graphics are hard to beat; Python is better for software that others will run, for machine learning pipelines and for gluing tools together. A good bioinformatician reads both, and a good analysis can be rerun by someone who uses only one: give the data, the script, the versions and the seed.

20. Exercises

Use the demonstration files. The answers are below each list; do the exercise first.

  1. What is the mean fibre intake (Fibre_g_day) in each diet group, and what does it tell you about the design?
  2. Write a function richness(x) that counts the features present in a sample, apply it to every sample of the unfiltered table and give the median per diet.
  3. With dplyr, reproduce the ratio of Firmicutes to Bacteroidota reads for each diet.
  4. Compare Fibre and Control in Shannon diversity with a Welch test. Why does the result differ from the Tukey comparison in section 11?
  5. Compute the mean and the median butyrate by diet. Why do they differ for the antibiotic group?
  6. Explain why how(blocks = m$Subject) gave a PERMANOVA p value of 1 in section 13, and what you would use instead.
Answers

1. tapply(m$Fibre_g_day, m$Diet, mean): Antibiotic 18.53, Control 18.54, Fibre 29.42 g per day. The fibre group eats about 11 g more, so the intervention did what it was meant to do, while antibiotic and control groups have the same intake: fibre intake cannot explain the antibiotic effect.

2. richness <- function(x) sum(x > 0); tapply(apply(counts[, bio], 2, richness), m$Diet, median) gives Antibiotic 141, Control 185.5, Fibre 172.5.

3. Group by diet and phylum, sum the reads, divide by the diet total, then divide the two phyla: 0.36 for antibiotics, 1.07 for controls and 1.74 for fibre (section 10).

4. t.test(sh[m$Diet == "Fibre"], sh[m$Diet == "Control"]): p = 0.0061, difference −0.112 (95% CI −0.190 to −0.034). Tukey's 0.36 uses a pooled error dominated by the very variable antibiotic group, so it is too conservative for this pair. The rank-based test of section 3 (adjusted p 0.0045) agrees with the Welch test.

5. Means: Antibiotic 11.67, Control 21.50, Fibre 31.53 mM; medians: 8.74, 20.26, 33.38. The antibiotic mean is above its median because a few samples have a high butyrate value (the distribution is skewed to the right); the median describes the typical sample better there. Always look at the distribution before choosing the summary.

6. blocks permutes the labels within each person. Diet is the same for all three samples of a person, so every permutation reproduces the original table and gives the same F; the observed F is never exceeded by a larger one, but is always matched, so p = 1. For a variable that differs between people, permute whole subjects: how(plots = Plots(strata = m$Subject, type = "free"), within = Within(type = "none")).

Check your understanding

PERMANOVA gives R² = 0.149 and p = 0.001, and the dispersion test p = 0.12. What do you conclude?

Diet explains about 15% of the variation in community composition, a moderate and significant effect. The groups do not differ detectably in spread, so the result reflects a shift in the centre of the groups.

R and Python give 74 and 72 significant features from the same data and the same correction. Is one wrong?

No. The two programs treat tied values in the rank test differently, which moves a few borderline p values. Report the method and the effect sizes, not only the count.

Why does diversity(t(cnt), "shannon") need the transpose?

vegan expects samples in rows and features in columns; the count table has features in rows.

A model has a coefficient DietAntibiotic = -0.48. What is it relative to?

The reference level, the first level of the factor: Control. Antibiotic samples have a Shannon index 0.48 lower than control samples, holding the other variables fixed.

Key references

  1. Anderson MJ. A new method for non-parametric multivariate analysis of variance. Austral Ecology 26, 32–46 (2001).
  2. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B 57, 289–300 (1995).
  3. Gloor GB, Macklaim JM, Pawlowsky-Glahn V, Egozcue JJ. Microbiome datasets are compositional: and this is not optional. Frontiers in Microbiology 8, 2224 (2017).
  4. Wickham H. Tidy data. Journal of Statistical Software 59, 1–23 (2014).
  5. Huber W, Carey VJ, Gentleman R, et al. Orchestrating high-throughput genomic analysis with Bioconductor. Nature Methods 12, 115–121 (2015).
  6. Irizarry RA. Introduction to Data Science: Data Analysis and Prediction Algorithms with R. Read: the R basics, data wrangling and visualisation parts.
  7. Wickham H. Tidy data. Journal of Statistical Software 59, 1–23 (2014).
  8. Anderson MJ. A new method for non-parametric multivariate analysis of variance. Austral Ecology 26, 32–46 (2001).
  9. Bates D, Mächler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. Journal of Statistical Software 67, 1–48 (2015).
  10. Law CW, Chen Y, Shi W, Smyth GK. : precision weights unlock linear model analysis tools for RNA-seq read counts. Genome Biology 15, R29 (2014).
  11. Ritchie ME, Phipson B, Wu D, et al.  powers differential expression analyses for RNA-sequencing and microarray studies. Nucleic Acids Research 43, e47 (2015).
  12. Love MI, Huber W, Anders S. Moderated estimation of fold change and dispersion for RNA-seq data with . Genome Biology 15, 550 (2014).
  13. Nearing JT, Douglas GM, Hayes MG, et al. Microbiome differential abundance methods produce different results across 38 datasets. Nature Communications 13, 342 (2022).
Core skills · Programming and data

Bash for bioinformatics: formats, labs and pipelines on one real genome

Bioinformatics files are text, and the shell is the fastest way to look inside them. This chapter works through nine short labs, in the order a newcomer meets the formats: FASTA, FASTQ and Phred quality, motifs,  on the command line, annotation files, alignments, variants, read counts, and finally a pipeline script that ties them together. Everything runs on one real genome, the 5,386-base phage phiX174, and every output shown is the real output of the command above it.

What this chapter adds

Chapter P1 introduced the shell, P3 the text tools (grep, awk, sed, join) and P4 the language of scripts. This chapter uses all three on the file formats of sequence analysis, and it is organised the way a practical course is: one lab per format, each ending with a result you can check. The labs follow the order of Hendrix, Applied Bioinformatics (Lab 1 FASTA files and GNU/Linux, Lab 2 FASTQ and quality scores, Lab 3 motifs, Lab 4  on the command line, Lab 6 annotation data, Lab 7 RNA-seq), and the habits behind them come from Buffalo, Bioinformatics Data Skills (Chapters 3, 7, 9, 10, 11 and 12). The examples, data and numbers are our own.

The tools beyond the standard shell are the ones every bioinformatician installs first: (sequence files), +, , , , , and for gene prediction. All of them are on Bioconda. awk examples that use and() need gawk; the plain awk on some systems (mawk) lacks it.

The lab: one real genome and simulated reads

The reference is the Illumina PhiX control genome, phiX174 (accession NC_001422.1), 5,386 bases. It is a good teaching genome: tiny, completely known, and the first DNA genome ever sequenced (Sanger et al. 1977). It is also the control that is spiked into many Illumina runs, so you will meet its reads in real data. The copy used here is the one in the BBTools resources (phix174_ill.ref.fa.gz); the current  record of the same accession differs from it at five positions, which is a reminder that a reference has versions and that you should record which one you used.

The reads are simulated by a small seeded script (shown at the end of this section) so that we know the truth. S1 has 6,000 reads of 100 bases from the unchanged genome. S2 has 6,000 reads from a copy in which we planted six single-base changes, one 3-base deletion and one 2-base insertion. Quality values follow a typical Illumina pattern (high at the start, falling along the read), and a base is changed with exactly the probability its quality states. queries.fa holds five 150-base pieces of phiX174 with 2 to 5 % differences and one random decoy. Because the truth is known, each lab can end with a check.

Set up a folder with data/ for inputs and results/ for everything we make. Raw data go in data/ and are never edited; every command writes its output to results/.

$ ls -l data | awk 'NR>1 {print $5, $9}'
1360723 S1.fq
1360750 S2.fq
1951 genes.gff
1825 phix174.fa.gz
5496 phix_mutated.fa
88 planted_variants.tsv
985 queries.fa
$ gunzip -kf data/phix174.fa.gz && head -c 200 data/phix174.fa
>gi|9626372|ref|NC_001422.1| Coliphage phiX174, complete genome
GAGTTTTATCGCTTCCATGACGCAGAAGTTAACACTTTCGGATATTTCTGATGAGTCGAAAAATTATCTT
GATAAAGCAGGAATTACTACTGCTTGTTTACGAATTAAATCGAAGTGGACTGCTGGCGGAAAATG
Make the lab data yourself (the simulation script)

Save the script as simulate.py. It needs only Python 3. Run python3 simulate.py data/phix174.fa.gz data with the gzipped reference in data/; the random seed is fixed, so you get the same reads. The genes were predicted with prodigal -i data/phix174.fa -f gff -o data/genes.gff -p meta.

"""Simulate the lab data for the bash chapter: single-end Illumina-like reads with position-dependent Phred qualities, from the
phiX174 reference (wild type, sample S1) and from a copy with planted variants (sample S2). Seeded, so the chapter can be rerun."""
import gzip
import random
import sys

random.seed(174)
ref_gz, out = sys.argv[1], sys.argv[2]
seq = "".join(l.strip() for l in gzip.open(ref_gz, "rt") if not l.startswith(">")).upper()
L = len(seq)

# planted variants (1-based positions on the reference): six SNPs, one 3-bp deletion, one 2-bp insertion
snps = {500: "A", 1200: "C", 2100: "T", 3000: "G", 4100: "A", 5000: "C"}
mut = list(seq)
planted = []
for pos, alt in snps.items():
    if mut[pos - 1] == alt:
        alt = "T" if alt != "T" else "G"
    planted.append((pos, mut[pos - 1], alt))
    mut[pos - 1] = alt
del_at = 2600                                        # delete 3 bases after position 2600
deleted = "".join(mut[del_at:del_at + 3])
mut[del_at:del_at + 3] = ["", "", ""]
ins_at = 3500                                        # insert 'GG' after position 3500
mut[ins_at - 1] = mut[ins_at - 1] + "GG"
mutated = "".join(mut)
open(f"{out}/planted_variants.tsv", "w").write("pos\tref\talt\n" + "".join(f"{p}\t{r}\t{a}\n" for p, r, a in planted)
                                               + f"{del_at}\t{seq[del_at-1]+deleted}\t{seq[del_at-1]}\n{ins_at}\t{seq[ins_at-1]}\t{seq[ins_at-1]}GG\n")
open(f"{out}/phix_mutated.fa", "w").write(">phiX174_mutated planted variants\n" + "\n".join(mutated[i:i + 70] for i in range(0, len(mutated), 70)) + "\n")

comp = str.maketrans("ACGT", "TGCA")


def qual_profile(n):
    q = []
    for i in range(n):
        base = 38 - 12 * (i / n) ** 2                      # quality decays along the read
        q.append(max(2, min(41, int(random.gauss(base, 2.5)))))
    return q


def reads(genome, n, length, name):
    G = len(genome)
    with open(f"{out}/{name}.fq", "w") as fh:
        for i in range(n):
            start = random.randrange(0, G - length)
            frag = genome[start:start + length]
            if random.random() < 0.5:                         # the fragment is read from the other strand
                frag = frag.translate(comp)[::-1]
            r = list(frag)
            q = qual_profile(length)                          # quality belongs to the sequencing cycle, not to the genome strand
            for j, qq in enumerate(q):
                if random.random() < 10 ** (-qq / 10):        # an error with exactly the probability the quality states
                    r[j] = random.choice([b for b in "ACGT" if b != r[j]])
            s = "".join(r)
            fh.write(f"@{name}_read{i + 1:05d} sim:{start + 1}\n{s}\n+\n{''.join(chr(33 + x) for x in q)}\n")


reads(seq, 6000, 100, "S1")
reads(mutated, 6000, 100, "S2")
#  queries: 5 pieces of phiX with 2-5% differences, and one decoy that is not phiX
with open(f"{out}/queries.fa", "w") as fh:
    for k, st in enumerate((300, 1500, 2800, 3900, 4800), 1):
        piece = list(seq[st:st + 150])
        for _ in range(int(150 * random.choice((0.02, 0.03, 0.05)))):
            j = random.randrange(150)
            piece[j] = random.choice([b for b in "ACGT" if b != piece[j]])
        fh.write(f">q{k}_from_{st + 1}\n{''.join(piece)}\n")
    fh.write(">q6_decoy\n" + "".join(random.choice("ACGT") for _ in range(150)) + "\n")
print("planted:", planted, "del", del_at, deleted, "ins", ins_at)

Lab 1: FASTA at the shell

A FASTA file is a header line that starts with >, followed by the sequence, usually wrapped at a fixed width. Two facts decide how you handle it in the shell: the sequence of one record spans many lines, and > is the only mark that separates records. Counting records is therefore a count of headers, and measuring a sequence means removing the line breaks first.

$ grep -c '>' data/phix174.fa
1
$ grep -v '>' data/phix174.fa | tr -d '\n' | wc -c
5386

The length, 5,386, agrees with the genome's published size. Base composition comes next: fold -w1 puts one base on each line, and sort | uniq -c counts them (P3). Then a single awk command gives the GC content, with gsub doing the counting (it returns the number of replacements).

$ grep -v '>' data/phix174.fa | tr -d '\n' | fold -w1 | sort | uniq -c
   1292 A
   1155 C
   1253 G
   1686 T
$ grep -v '>' data/phix174.fa | tr -d '\n' | awk '{n=length($0); gc=gsub(/[GC]/,""); printf "GC = %.2f%% of %d bases\n", 100*gc/n, n}'
GC = 44.71% of 5386 bases

GC content of 44.71 % is moderate and is one of the numbers that identify an organism, a contig or a contaminant. A single global value hides local differences, and the shell can slide a window along the genome: fold -w 1000 cuts the sequence into 1,000-base pieces, and awk prints the GC of each one. A window that departs from its neighbours points to a foreign segment, a repeat or a gene island.

$ grep -v '>' data/phix174.fa | tr -d '\n' | fold -w 1000 | awk '{n=length($0); gc=gsub(/[GC]/,""); printf "%d\t%.1f\n", (NR-1)*1000+1, 100*gc/n}'
1	44.7
1001	45.0
2001	43.2
3001	44.7
4001	45.8
5001	45.1

The reverse complement matters in every analysis, because DNA has two strands and a gene may lie on either. For a short sequence the shell does it in one line: rev reverses the characters and tr ACGT TGCA swaps each base for its partner. (For IUPAC codes and for whole files use  seq -r -p.) Compare the first thirty bases with their reverse complement:

$ sed -n 2p data/phix174.fa | cut -c1-30; sed -n 2p data/phix174.fa | cut -c1-30 | rev | tr ACGT TGCA
GAGTTTTATCGCTTCCATGACGCAGAAGTT
AACTTCTGCGTCATGGAAGCGATAAAACTC

The two most common problems with FASTA files are invisible. Windows line endings: a file edited on Windows ends every line with a carriage return (\r), which becomes part of the sequence, so lengths are off by one per line and grep patterns at the end of a line never match. file names the problem, grep -c $'\r' counts the lines it affects, and tr -d '\r' removes it. The second command below is the fix; zero lines are left with a carriage return.

$ sed 's/$/\r/' data/phix174.fa > results/windows.fa; file results/windows.fa | cut -d: -f2; grep -c $'\r' results/windows.fa; tr -d '\r' < results/windows.fa | grep -c $'\r' || true
 ASCII text, with CRLF line terminators
78
0

The other is awkward headers. The header of this record, gi|9626372|ref|NC_001422.1| Coliphage phiX174, complete genome, contains vertical bars and spaces, which many tools treat as separators. sed replaces the bars and cuts everything after the first space. When you need one record out of many, awk can switch a flag on at the header you want and print while it is on;  grep -p NAME does the same for lists of names. Finally, check your own arithmetic against a tool:  stats reports the same 5,386 bases.

$ grep '>' data/queries.fa | head -3; grep '>' data/phix174.fa | sed 's/|/_/g; s/ .*//'
>q1_from_301
>q2_from_1501
>q3_from_2801
>gi_9626372_ref_NC_001422.1_
$ awk '/^>/ {keep = ($1 == ">q3_from_2801")} keep' data/queries.fa
>q3_from_2801
CGAGCAATAATGTTTATGCTGGTTTCATGGTTTGGTCTAACTTTACCGCTACTAAATGCCGCGGATTGGTTTCGCTGAATCAGGTTATTAAAGAGATTGTTTGTCTCCAGCCACTTAAGTGAGGTGATTTATGTTTGGTGCTATTGCTGG
$  stats data/phix174.fa
file             format  type  num_seqs  sum_len  min_len  avg_len  max_len
data/phix174.fa  FASTA   DNA          1    5,386    5,386    5,386    5,386

Lab 2: FASTQ and the Phred scale

A FASTQ record has four lines: an identifier that starts with @, the sequence, a separator line that starts with +, and one quality character per base. The quality of a base is a Phred score, Q = −10 log10 P, where P is the probability that the base is wrong. Q10 is a 1-in-10 chance of error, Q20 one in 100, Q30 one in 1,000. In the modern (Sanger or Illumina 1.8) encoding the character is the score plus 33 turned into an ASCII character: I is Q40, 5 is Q20, + is Q10 (Cock et al. 2010). Older Illumina files used an offset of 64, so a file whose first character is lower case needs checking before you trust it.

$ head -4 data/S1.fq
@S1_read00001 sim:39
TCTCATTTTCCGCCAGCAGTCCACTTCGATTTAATTCGTAAACAAGCAGTAGTAATTCCTGCTTTATCAAGATAATTTTTCGACTCATCAGAAATATCCG
+
IGGHHEEDCEECECJGIHIGFHFECCGFGHGHJFCFGDABDHGDGBIDG@ICIG@@GAB@D@?CBD>J?=DC@@@A@:;==@@B@<A>???>><=9=<<:
$ echo $(( $(wc -l < data/S1.fq) / 4 )) reads
6000 reads
$ awk 'NR%4==2 {print length($0)}' data/S1.fq | sort | uniq -c
   6000 100
$ awk 'BEGIN {for (q=10; q<=40; q+=10) printf "Q%d\terror probability %g\n", q, 10^(-q/10)}'
Q10	error probability 0.1
Q20	error probability 0.01
Q30	error probability 0.001
Q40	error probability 0.0001

Counting reads is a line count divided by four, and the lengths are tallied with awk on every fourth line (here all 100). To use the qualities you have to decode them. awk has no built-in function that turns a character into its number, so the program builds a lookup table once in BEGIN, using sprintf("%c", i), and then adds up the table values character by character. The command below writes the mean quality of every read to a file and shows the first four:

$ awk 'BEGIN {for (i=33; i<127; i++) ord[sprintf("%c", i)] = i-33}
> NR%4==1 {id=substr($1,2)}
> NR%4==0 {n=length($0); s=0; for (i=1; i<=n; i++) s+=ord[substr($0,i,1)]; printf "%s\t%.2f\n", id, s/n}' data/S1.fq > results/S1.meanq.tsv; head -4 results/S1.meanq.tsv
S1_read00001	33.96
S1_read00002	33.56
S1_read00003	33.56
S1_read00004	33.96

One more awk turns that file into a summary. All 6,000 reads have a mean quality of at least 30, which looks excellent, and the mean of the means is 33.55. A mean hides the shape: quality falls along a read, and the per-position view shows where. Positions 1, 34, 67 and 100 give a sketch of the usual curve; a real run is read with a full per-position plot ( and  draw one, P1).

$ awk '{s+=$2; if ($2>=30) k++} END {printf "mean of the read means %.2f; %d of %d reads (%.1f%%) have a mean quality of at least 30\n", s/NR, k, NR, 100*k/NR}' results/S1.meanq.tsv
mean of the read means 33.55; 6000 of 6000 reads (100.0%) have a mean quality of at least 30
$ awk 'BEGIN {for (i=33; i<127; i++) ord[sprintf("%c", i)] = i-33}
> NR%4==0 {for (i=1; i<=length($0); i++) sum[i]+=ord[substr($0,i,1)]; n++}
> END {for (i=1; i<=100; i+=33) printf "position %d\tmean quality %.1f\n", i, sum[i]/n; printf "position 100\tmean quality %.1f\n", sum[100]/n}' data/S1.fq
position 1	mean quality 37.4
position 34	mean quality 36.2
position 67	mean quality 32.3
position 100	mean quality 25.7
position 100	mean quality 25.7

A filter is a program that keeps a record only when a test passes. Because a FASTQ record is four lines, the program collects four lines before it decides. Here every read passes, since the simulated reads are good; on real data you would keep a read only if its mean quality or its worst window meets a threshold. A tool such as  stats -a gives the same kind of summary in one line (Q20 and Q30 shares, average quality, N50), and is the right way to check a hand-made calculation:

$ awk 'BEGIN {for (i=33; i<127; i++) ord[sprintf("%c", i)] = i-33}
> {rec[NR%4]=$0}
> NR%4==0 {n=length($0); s=0; for (i=1; i<=n; i++) s+=ord[substr($0,i,1)]; if (s/n >= 30) print rec[1] "\n" rec[2] "\n" rec[3] "\n" rec[0]}' data/S1.fq > results/S1.q30.fq; echo $(( $(wc -l < results/S1.q30.fq) / 4 )) reads kept
6000 reads kept
$  stats -a data/S1.fq | cut -f1,4,6,8,9,13-17
file        format  type  num_seqs  sum_len  min_len  avg_len  max_len   Q1   Q2   Q3  sum_gap  N50  N50_num  Q20(%)  Q30(%)  AvgQual  GC(%)  sum_n
data/S1.fq  FASTQ   DNA      6,000  600,000      100      100      100  100  100  100        0  100    3,000   99.98   80.87    31.24   44.8      0

Notice the difference: AvgQual in the  table (31.24) is lower than our 33.55. It is not an error. A mean of quality scores and a mean of error probabilities converted back to a score are different quantities, and  reports the second, which is the more honest summary because a few bad bases count for more (chapter P9 reproduces both numbers in Python). Always know which one a tool reports.

Lab 3: motifs

A motif is a short pattern that means something: a restriction site, a ribosome-binding site, a transcription-factor binding site. In the shell, the simplest motif search is grep -o (print only the match) with -b (print the byte offset). Because the sequence has been joined onto one line, the offset is the position, counted from zero.

$ grep -v '>' data/phix174.fa | tr -d '\n' | grep -ob 'AGGAG'
232:AGGAG
835:AGGAG
2382:AGGAG
3540:AGGAG
5062:AGGAG

The pattern AGGAG occurs five times. DNA has two strands, so a motif that is not its own reverse complement has to be searched on both: the other strand reads the reverse complement. GGAGG (the core of the Shine–Dalgarno sequence that helps a ribosome find a start codon) occurs four times as given and six times as CCTCC, which is the same motif on the other strand. A search of one strand would have missed more than half.

$ seq=$(grep -v '>' data/phix174.fa | tr -d '\n')
$ fwd=$(echo "$seq" | grep -o 'GGAGG' | wc -l)
$ rev=$(echo "$seq" | grep -o 'CCTCC' | wc -l)
$ echo "GGAGG on the given strand: $fwd; its reverse complement CCTCC (the motif on the other strand): $rev"
GGAGG on the given strand: 4; its reverse complement CCTCC (the motif on the other strand): 6

Real motifs are rarely exact. The IUPAC code writes R for A or G, Y for C or T, W for A or T, S for C or G and N for any base, and a pattern is a regular expression once these are replaced. A small shell function does the translation. A ribosome-binding site followed by a start codon after a gap can then be written AGGAG[ACGT]{4,10}ATG, and grep -E finds it. The three hits are real: positions 835, 2,382 and 5,062 lie just before genes predicted by , whose own output records an AGGAG motif for the same genes.

$ iupac2re() { echo "$1" | sed 's/R/[AG]/g; s/Y/[CT]/g; s/W/[AT]/g; s/S/[CG]/g; s/N/[ACGT]/g'; }
$ iupac2re AGGAGRNNNNATG
AGGAG[AG][ACGT][ACGT][ACGT][ACGT]ATG
$ grep -v '>' data/phix174.fa | tr -d '\n' | grep -obE 'AGGAG[ACGT]{4,10}ATG'
835:AGGAGTGATGTAATG
2382:AGGAGTTTAATCATG
5062:AGGAGCTAAAGAATG

The most important question about a motif count is: more or fewer than chance? In a random sequence of length L, a pattern of k bases is expected about (L − k + 1) / 4k times. For a six-base site in 5,386 bases that is 1.31, and for the four-base site GATC it is 21.03. The loop below counts five famous sites and prints the expectation next to each:

$ L=$(grep -v '>' data/phix174.fa | tr -d '\n' | wc -c)
$ seq=$(grep -v '>' data/phix174.fa | tr -d '\n')
$ printf 'site\tobserved\texpected\n'
$ for m in GAATTC GGATCC AAGCTT CTGCAG GATC; do
>   obs=$(echo "$seq" | grep -o "$m" | wc -l)
>   awk -v m=$m -v o=$obs -v L=$L 'BEGIN {k=length(m); printf "%s\t%d\t%.2f\n", m, o, (L-k+1)/4^k}'
> done
site	observed	expected
GAATTC	0	1.31
GGATCC	0	1.31
AAGCTT	0	1.31
CTGCAG	0	1.31
GATC	0	21.03

Each of the six-base sites (EcoRI, BamHI, HindIII, PstI) is absent, and absence is unremarkable when about one copy was expected: the chance of none is e−1.31, about 27 %. GATC is different. About 21 copies are expected and there are none. Using the real base composition instead of equal frequencies (the next command) the expectation is 20.2 and the chance of none is about 2 in a billion. A gap like that is selection, not luck. GATC is the site that the E. coli Dam methylase marks, and a plausible reading is that a phage which depends on its host's repair and restriction systems gains by avoiding the sites they recognise. That is a hypothesis the count suggests, not a result it proves. The method is the lesson: an observed count means nothing until you know the expected one.

$ grep -v '>' data/phix174.fa | tr -d '\n' | fold -w1 | sort | uniq -c | awk -v L=5386 '{f[$2]=$1/L} END {p=f["G"]*f["A"]*f["T"]*f["C"]; e=(L-3)*p; printf "P(GATC at a position) = %.5f; expected copies %.1f; chance of none at all = %.1e\n", p, e, exp(-e)}'
P(GATC at a position) = 0.00375; expected copies 20.2; chance of none at all = 1.7e-09

Finally, a motif can be found instead of searched. Take the 12 bases in front of each predicted start codon, align them by the start, and count the bases in each column.  faidx and  getfasta cut the windows out (the coordinate details come in Lab 5); an awk counter builds the count matrix; and the most common base in each column is the consensus.

$  faidx data/phix174.fa
$ cut -f1,2 data/phix174.fa.fai > results/genome.sizes
$ grep -v '^#' data/genes.gff | awk 'BEGIN{OFS="\t"} $4>13 {split($9,a,";"); sub("ID=","",a[1]); print $1, $4-13, $4-1, a[1], 0, $7}' > results/rbs_windows.bed
$ cat results/rbs_windows.bed | cut -f2-4
38	50	1_1
377	389	1_2
835	847	1_3
988	1000	1_4
2382	2394	1_5
2918	2930	1_6
3968	3980	1_7
$  getfasta -fi data/phix174.fa -bed results/rbs_windows.bed -nameOnly -tab | tee results/rbs_windows.tsv
1_1	CGGATATTTCTG
1_2	AGGTAAGAAATC
1_3	AGGAGTGATGTA
1_4	TGAGGATAAATT
1_5	AGGAGTTTAATC
1_6	GTGAGGTGATTT
1_7	GGAGGCTTTTTT
$ cut -f2 results/rbs_windows.tsv | awk '{for (i=1; i<=length($0); i++) c[i, substr($0,i,1)]++; n=NR} END {split("A C G T", b, " "); for (j=1; j<=4; j++) {printf "%s", b[j]; for (i=1; i<=12; i++) printf "\t%d", c[i, b[j]]; printf "\n"}}'
A	3	0	2	4	1	3	0	3	4	3	0	1
C	1	0	0	0	0	1	0	0	0	1	0	2
G	2	6	5	2	5	1	2	1	0	1	0	1
T	1	1	0	1	1	2	5	3	3	2	7	3

The columns are the 12 positions and the rows are A, C, G and T. The first five columns lean to A and G, which is the AGGAG signal, and the eleventh column is T in all seven windows. The consensus read from the largest count per column is AGGAGATAAATT. Treat it as a sketch only: seven short windows are too few, and because the gap between the motif and the start codon varies, aligning at the start smears the motif. Real motif finders, such as MEME, search over the offset (Hendrix, Ch. 2, de novo motif finding). The shell is for the first look.

Lab 4:  on the command line

+ is a pair of steps: makeblastdb turns a FASTA file into a searchable database, and (or , ) searches it with queries (Camacho et al. 2009). The tabular output, -outfmt 6, is the one to use in a pipeline: one hit per line, twelve tab-separated columns, no header.

$ makeblastdb -in data/phix174.fa -dbtype nucl -out results/phix -title phix174 > results/makeblastdb.log
$ blastdbcmd -db results/phix -info | head -3
Database: phix174
	1 sequences; 5,386 total bases
$  -query data/queries.fa -db results/phix -evalue 1e-5 -outfmt 6 | tee results/hits.tsv
q1_from_301	gi|9626372|ref|NC_001422.1|	97.333	150	4	0	1	150	301	450	6.59e-72	255
q2_from_1501	gi|9626372|ref|NC_001422.1|	97.333	150	4	0	1	150	1501	1650	6.59e-72	255
q3_from_2801	gi|9626372|ref|NC_001422.1|	98.000	150	3	0	1	150	2801	2950	1.42e-73	261
q4_from_3901	gi|9626372|ref|NC_001422.1|	97.333	150	4	0	1	150	3901	4050	6.59e-72	255
q5_from_4801	gi|9626372|ref|NC_001422.1|	98.000	150	3	0	1	150	4801	4950	1.42e-73	261

The twelve columns of the default format are the query and subject names, percent identity, alignment length, mismatches, gap openings, the start and end on the query, the start and end on the subject, the E-value and the bit score. All five phiX pieces are found at their true places (query q1_from_301 hits subject positions 301 to 450), with 97–98 % identity, as designed. You can ask for other columns, including the query coverage that the default omits, with a quoted list after the number: -outfmt '6 qseqid sseqid pident length qcovs ...'. The command below adds a header with awk:

$  -query data/queries.fa -db results/phix -evalue 1e-5 -outfmt '6 qseqid sseqid pident length qcovs sstart send evalue bitscore' | sed 's/gi|9626372|ref|//; s/|\t/\t/' | awk 'BEGIN {OFS="\t"; print "query", "subject", "identity", "length", "qcov", "sstart", "send", "evalue", "bits"} {print}'
query	subject	identity	length	qcov	sstart	send	evalue	bits
q1_from_301	NC_001422.1	97.333	150	100	301	450	6.59e-72	255
q2_from_1501	NC_001422.1	97.333	150	100	1501	1650	6.59e-72	255
q3_from_2801	NC_001422.1	98.000	150	100	2801	2950	1.42e-73	261
q4_from_3901	NC_001422.1	97.333	150	100	3901	4050	6.59e-72	255
q5_from_4801	NC_001422.1	98.000	150	100	4801	4950	1.42e-73	261

Filtering is the usual next step, and it is plain awk on the columns: keep a hit if identity is at least 95 and the aligned length at least 140. A best hit per query table sorts by query and by bit score (descending) and then keeps the first line of each query, using the idiom awk '!seen[$1]++', which prints a line only the first time its first field is seen. Last, comm lists the queries that had no hit at all, which is often the question that matters; the random decoy is the only one.

$ awk -F'\t' '$3 >= 95 && $4 >= 140' results/hits.tsv | cut -f1,3,4,7-9 | head
q1_from_301	97.333	150	1	150	301
q2_from_1501	97.333	150	1	150	1501
q3_from_2801	98.000	150	1	150	2801
q4_from_3901	97.333	150	1	150	3901
q5_from_4801	98.000	150	1	150	4801
$ sort -k1,1 -k12,12gr results/hits.tsv | awk '!seen[$1]++' | cut -f1,3,4,9,10,12
q1_from_301	97.333	150	301	450	255
q2_from_1501	97.333	150	1501	1650	255
q3_from_2801	98.000	150	2801	2950	261
q4_from_3901	97.333	150	3901	4050	255
q5_from_4801	98.000	150	4801	4950	261
$ comm -23 <(grep '>' data/queries.fa | tr -d '>' | sort) <(cut -f1 results/hits.tsv | sort -u)
q6_decoy

Lab 5: annotation files, and the coordinate trap

An annotation file says where things are on a genome. GFF3 has nine tab-separated columns: sequence name, source, feature type, start, end, score, strand, phase, and a column of key=value attributes separated by semicolons. Its coordinates are 1-based and inclusive: a feature from 51 to 221 covers bases 51, 52, …, 221, and its length is end − start + 1. BED has at least three columns (sequence, start, end), with optional name, score and strand, and its coordinates are 0-based and half-open: the same feature is written 50 and 221, and its length is end − start. Confusing the two shifts every feature by one base, and nothing warns you.

$ grep -v '^#' data/genes.gff | head -3 | cut -f1-8 | sed 's/gi|9626372|ref|NC_001422.1|/NC_001422.1/'
NC_001422.1	_v2.6.3	CDS	51	221	3.3	+	0
NC_001422.1	_v2.6.3	CDS	390	848	54.5	+	0
NC_001422.1	_v2.6.3	CDS	848	964	15.7	+	0
$ grep -v '^#' data/genes.gff | cut -f3,7 | sort | uniq -c
      7 CDS	+

 predicted seven genes, all on the forward strand. The attribute column holds the identifier; awk splits it on the semicolon and strips ID=. A table of start, end, length and codons gives a first quality check: the length of a protein-coding gene must be a multiple of three. All seven pass.

$ grep -v '^#' data/genes.gff | awk -F'\t' 'BEGIN {OFS="\t"; print "gene", "start", "end", "strand", "length", "codons", "multiple_of_3"} {split($9,a,";"); sub("ID=","",a[1]); L=$5-$4+1; print a[1], $4, $5, $7, L, L/3, (L%3==0 ? "yes" : "no")}'
gene	start	end	strand	length	codons	multiple_of_3
1_1	51	221	+	171	57	yes
1_2	390	848	+	459	153	yes
1_3	848	964	+	117	39	yes
1_4	1001	2284	+	1284	428	yes
1_5	2395	2922	+	528	176	yes
1_6	2931	3917	+	987	329	yes
1_7	3981	5384	+	1404	468	yes

To convert to BED, subtract one from the start and keep the end. Then  getfasta cuts each region out of the genome (-s takes the strand into account and reverse-complements minus-strand features, -nameOnly uses just the name). A good test of a gene table is to look at the first and last codon of every gene:

$ grep -v '^#' data/genes.gff | awk 'BEGIN {OFS="\t"} {split($9,a,";"); sub("ID=","",a[1]); print $1, $4-1, $5, a[1], $6, $7}' > results/genes.bed
$ cut -f2-4,6 results/genes.bed
50	221	1_1	+
389	848	1_2	+
847	964	1_3	+
1000	2284	1_4	+
2394	2922	1_5	+
2930	3917	1_6	+
3980	5384	1_7	+
$  getfasta -fi data/phix174.fa -bed results/genes.bed -s -nameOnly -fo results/genes.fa
$  fx2tab -n -l results/genes.fa | head -3
$ echo 'first codons:';  seq -s results/genes.fa | cut -c1-3 | sort | uniq -c
$ echo 'last codons:';  seq -s results/genes.fa | rev | cut -c1-3 | rev | sort | uniq -c
1_1(+)	171
1_2(+)	459
1_3(+)	117
first codons:
      7 ATG
last codons:
      1 CTG
      3 TAA
      3 TGA

All seven genes start with ATG, as the predictor requires. At the other end three finish with TAA and three with TGA, which are stop codons, and one finishes with CTG, which is not. The last gene, 3,981 to 5,384, runs to the end of the sequence:  marked it partial=01, an incomplete gene: it found no stop codon before the sequence ends. The  record (chapter P9 downloads it) shows why: this is gene A, which starts at base 3,981 and, because the genome is circular, continues across the origin, so its stop codon lies at the start of the sequence. Checking the last codon found what a glance at the table would not.

Now the trap itself. If the GFF start is used as the BED start without subtracting one, every sequence begins one base late. The first codons of the same seven genes now read TGA, TGG or TGT instead of ATG, which would make any later translation wrong:

$ grep -v '^#' data/genes.gff | awk 'BEGIN {OFS="\t"} {split($9,a,";"); sub("ID=","",a[1]); print $1, $4, $5, a[1], $6, $7}' > results/genes_wrong.bed
$  getfasta -fi data/phix174.fa -bed results/genes_wrong.bed -s -nameOnly -tab | cut -f2 | cut -c1-3 | sort | uniq -c
      2 TGA
      1 TGG
      4 TGT

Two more questions that a table of genes answers directly.  complement lists the regions between the genes, given the genome size file made from the FASTA index, and a short awk adds up the coding bases. The predicted genes cover 91.9 % of the genome, which is typical of a compact viral genome. Be careful with the number all the same: phiX174 is known to carry about eleven genes, several of them overlapping, which was one of the surprises of its sequence (Sanger et al. 1977). A gene finder reports one gene per region and misses genes that lie inside others, so the seven predicted genes are a floor, not the gene count. The  record lists eleven (chapter P9).

$  complement -i results/genes.bed -g results/genome.sizes | awk '{print $2+1 "-" $3 "\t" $3-$2 " bp"}'
$ awk '{c+=$3-$2} END {printf "predicted coding: %d of 5386 bases (%.1f%%)\n", c, 100*c/5386}' results/genes.bed
1-50	50 bp
222-389	168 bp
965-1000	36 bp
2285-2394	110 bp
2923-2930	8 bp
3918-3980	63 bp
5385-5386	2 bp
predicted coding: 4950 of 5386 bases (91.9%)

Lab 6: alignments, SAM flags and depth

Mapping places each read on the reference.  -ax sr maps short reads and writes SAM text, which  sort turns into a sorted BAM;  index makes it searchable.  flagstat is the first sanity check of any mapping: how many reads, how many mapped (Li 2018; Danecek et al. 2021).

$  -ax sr data/phix174.fa data/S1.fq 2> results/S1..log |  sort -o results/S1.bam
$  index results/S1.bam
$  flagstat results/S1.bam
6000 + 0 in total (QC-passed reads + QC-failed reads)
6000 + 0 primary
0 + 0 secondary
0 + 0 supplementary
0 + 0 duplicates
0 + 0 primary duplicates
6000 + 0 mapped (100.00% : N/A)
6000 + 0 primary mapped (100.00% : N/A)
…

All 6,000 reads map, as they should: they came from this genome. A SAM record is a line of eleven mandatory columns: read name, FLAG, reference name, position (1-based), mapping quality, CIGAR, then the mate fields, sequence and quality. The CIGAR 100M means 100 aligned bases.

$  view results/S1.bam | head -2 | cut -f1-9 | cut -c1-110
S1_read02138	0	gi|9626372|ref|NC_001422.1|	1	60	100M	*	0	0
S1_read00710	16	gi|9626372|ref|NC_001422.1|	1	60	100M	*	0	0

The FLAG is a set of bits added into one number: 1 paired, 2 proper pair, 4 read unmapped, 8 mate unmapped, 16 read on the reverse strand, 32 mate on the reverse strand, 64 first in pair, 128 second in pair, 256 secondary alignment, 512 failed quality control, 1,024 duplicate, 2,048 supplementary. Single-end reads on the forward strand have flag 0 and on the reverse strand 16, so counting flags gives the strand balance. In gawk, and($2, 16) tests a bit;  view -f 16 and -F 4 do the same filtering without you decoding anything.

$  view results/S1.bam | cut -f2 | sort | uniq -c | sort -k1,1nr
   3052 16
   2948 0
$  view results/S1.bam | gawk '{rev += and($2, 16) ? 1 : 0; fwd += and($2, 16) ? 0 : 1} END {print fwd, "reads on the forward strand;", rev, "on the reverse strand"}'
2948 reads on the forward strand; 3052 on the reverse strand

Mapping quality is the second thing to look at. 60 is the best value  gives; here 5,995 of 6,000 reads have it, and five reads have lower values (they come from places where the genome is locally repetitive or where errors cluster). Depth is the third.  depth -a prints one line per base, -a including bases with no reads. The average is 111.4, and 11 of the 5,386 bases are below 10 reads: the first four and the last seven, at the two ends of the linear sequence, where few reads can start or end.

$  view results/S1.bam | cut -f5 | sort -n | uniq -c | sort -k1,1nr | head -4
   5995 60
      2 50
      1 34
      1 37
$  depth -a results/S1.bam | awk '{s+=$3; if ($3<10) low++; if ($3>m) m=$3} END {printf "mean depth %.1f; maximum %d; positions below 10x: %d of %d\n", s/NR, m, low, NR}'
mean depth 111.4; maximum 152; positions below 10x: 11 of 5386

Finally, depth per gene, using the BED file from Lab 5.  bedcov adds up the depth over each interval, and dividing by the length gives the mean depth of the gene. An even depth across genes is what a random library gives; a gene with a much lower depth would suggest a deletion, a repeat that cannot be mapped, or a gene that is absent from this sample. Here all seven genes are between 96 and 120.

$  bedcov results/genes.bed results/S1.bam | awk 'BEGIN {OFS="\t"; print "gene", "length", "mean_depth"} {len=$3-$2; print $4, len, sprintf("%.1f", $7/len)}'
gene	length	mean_depth
1_1	171	106.3
1_2	459	119.8
1_3	117	95.7
1_4	1284	115.7
1_5	528	112.0
1_6	987	111.9
1_7	1404	110.1

Lab 7: variants

A variant is a difference between a sample and the reference.  mpileup collects, for every position, the bases that the reads show, and  call decides which positions differ; the result is a VCF file (Danecek et al. 2011). For a haploid organism such as a phage or a bacterium use --ploidy 1; -m is the multiallelic caller and -v keeps only variant sites. We map both samples and then call the variants in S2, the one that carries the planted changes.

$ for s in S1 S2; do  -ax sr data/phix174.fa data/$s.fq 2> results/$s..log |  sort -o results/$s.bam;  index results/$s.bam; done
$ ls results/*.bam
results/S1.bam
results/S2.bam
$  mpileup -f data/phix174.fa results/S2.bam 2> results/S2.mpileup.log |  call --ploidy 1 -mv -Ov -o results/S2.vcf 2> results/S2.call.log
$ grep -c '^##' results/S2.vcf; grep -v '^##' results/S2.vcf | cut -f1-8 | cut -c1-150 | sed 's/gi|9626372|ref|NC_001422.1|/NC_001422.1/'
28
#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO
NC_001422.1	500	.	G	A	225.417	.	DP=101;VDB=0.343023;SGB=-0.693147;MQSBZ=-0.951662;MQ0F=0;AC=1;AN=1;DP4=0,0,48,53;MQ=59
NC_001422.1	1200	.	T	C	225.417	.	DP=101;VDB=0.976817;SGB=-0.693147;MQSBZ=-0.623579;MQ0F=0;AC=1;AN=1;DP4=0,0,51,50;MQ=59
NC_001422.1	2100	.	A	T	225.417	.	DP=120;VDB=0.137336;SGB=-0.693147;MQSBZ=-0.0118345;MQ0F=0;AC=1;AN=1;DP4=0,0,60,60;MQ=59
NC_001422.1	2600	.	CTAAT	CT	228.377	.	INDEL;IDV=87;IMF=0.887755;DP=98;VDB=0.216449;SGB=-0.693147;RPBZ=-5.38662;MQBZ=0;MQSBZ=0;BQBZ=-1.
NC_001422.1	3000	.	G	T	225.422	.	DP=118;VDB=0.861561;SGB=-0.693147;MQSBZ=0.592246;MQ0F=0;AC=1;AN=1;DP4=0,0,58,60;MQ=59
NC_001422.1	3500	.	C	CGG	228.396	.	INDEL;IDV=109;IMF=0.931624;DP=117;VDB=0.0161494;SGB=-0.693147;RPBZ=-3.53227;MQBZ=-0.473288;MQSBZ=0.
NC_001422.1	4100	.	G	A	225.417	.	DP=120;VDB=0.45931;SGB=-0.693147;MQSBZ=0.258413;MQ0F=0;AC=1;AN=1;DP4=0,0,54,66;MQ=59
NC_001422.1	5000	.	T	C	225.417	.	DP=128;VDB=0.70269;SGB=-0.693147;MQSBZ=-1.66202;MQ0F=0;AC=1;AN=1;DP4=0,0,74,54;MQ=59

A VCF file starts with ## meta-information lines (28 here: the format, the contigs, the definitions of every INFO field, the commands that made it), then one header line starting with #CHROM, then one line per variant with eight fixed columns: chromosome, position, ID, reference allele, alternate allele, quality, filter and INFO. In INFO, DP is the depth and DP4 the number of reference-forward, reference-reverse, alternate-forward and alternate-reverse reads that support the call. The alternate allele here is supported by every read (DP4=0,0,48,53: none for the reference, 48 and 53 for the alternate), as expected for a haploid sample.

The two rows with INDEL need a word. An insertion or deletion is written with a padding base in front, taken from the reference, because a VCF line has to name a position that exists in the reference. We planted the deletion as CTAA to C; the caller wrote it as CTAAT to CT. Both remove the same three bases, TAA. The same event can be written in more than one way, which is why you compare variant sets after normalising them with  norm, or, as here, by position.

$  view -H results/S2.vcf | awk '{print (length($4)==length($5)) ? "SNP" : "indel"}' | sort | uniq -c
      6 SNP
      2 indel
$ cat data/planted_variants.tsv
pos	ref	alt
500	G	A
1200	T	C
2100	A	T
3000	G	T
4100	G	A
5000	T	C
2600	CTAA	C
3500	C	CGG

The test of any variant caller is whether it finds what is there and nothing else. comm compares two sorted lists. All eight planted positions are found, nothing is missed and nothing extra is called. The unchanged sample, S1, gives no variants at all (the count below is 0; note that grep -vc returns exit status 1 when its count is zero, which would stop a script with set -e, a good reason to read the status of a command before you rely on it).

$ awk 'NR>1 {print $1}' data/planted_variants.tsv | sort -n > results/planted.pos
$  query -f '%POS\n' results/S2.vcf | sort -n > results/called.pos
$ echo 'planted and found:'; comm -12 results/planted.pos results/called.pos | tr '\n' ' '; echo
$ echo 'planted, missed:'; comm -23 results/planted.pos results/called.pos | tr '\n' ' '; echo
$ echo 'called, not planted:'; comm -13 results/planted.pos results/called.pos | tr '\n' ' '; echo
planted and found:
500 1200 2100 2600 3000 3500 4100 5000 
planted, missed:

called, not planted:
$  mpileup -f data/phix174.fa results/S1.bam 2>/dev/null |  call --ploidy 1 -mv -Ov 2>/dev/null | grep -vc '^#'
0

This is a clean case: haploid, 111-fold depth, no repeats and a perfectly simulated error model. Real data are harder, so a call is filtered on its quality and depth before it is trusted.  filter -i 'QUAL>=30 && DP>=10' keeps the variants that pass both tests, and  query -f prints just the fields you ask for. Choose thresholds from the distribution you see, not from a recipe: here every variant has a quality above 225 and a depth above 98, so any reasonable threshold keeps all eight.

$  query -f '%POS\t%REF\t%ALT\t%QUAL\t%DP\n' results/S2.vcf
$ echo '--- QUAL>=30 and DP>=10:'
$  filter -i 'QUAL>=30 && DP>=10' results/S2.vcf |  query -f '%POS\t%REF\t%ALT\n' | wc -l
500	G	A	225.417	101
1200	T	C	225.417	101
2100	A	T	225.417	120
2600	CTAAT	CT	228.377	98
3000	G	T	225.422	118
3500	C	CGG	228.396	117
4100	G	A	225.417	120
5000	T	C	225.417	128
--- QUAL>=30 and DP>=10:
8

Lab 8: from reads to genes, counts and TPM

RNA-seq turns the same machinery into a counting problem: how many reads fall on each gene?  coverage -counts counts the reads of a BAM file that overlap each BED interval. We use it on S1 and S2 as a stand-in (these are DNA reads, not RNA, so the numbers are depths, but the commands are the ones used for RNA-seq counting with a gene BED file). Putting two samples side by side is a paste.

$ for s in S1 S2; do  coverage -counts -a results/genes.bed -b results/$s.bam | cut -f4,7 | sort -k1,1 > results/$s.counts; done
$ paste results/S1.counts results/S2.counts | cut -f1,2,4
1_1	250	257
1_2	657	620
1_3	215	231
1_4	1599	1542
1_5	701	684
1_6	1206	1223
1_7	1599	1691

Raw counts cannot be compared between genes, because a long gene collects more reads than a short one, nor between samples, because one library may be larger. TPM (transcripts per million) corrects for both in two steps: divide each count by the gene length in kilobases to get reads per kilobase (RPK), then divide each RPK by the sum of all RPK values and multiply by one million. TPM values of a sample sum to one million, so a gene's share can be compared between samples. The awk program reads the gene lengths from the BED file in its first pass (NR==FNR, as in P3) and the counts in its second:

$ awk 'BEGIN {OFS="\t"} NR==FNR {len[$4]=$3-$2; next} {r=$2/(len[$1]/1000); id[++n]=$1; rpk[n]=r; tot+=r} END {for (i=1; i<=n; i++) printf "%s\t%.0f\n", id[i], 1e6*rpk[i]/tot}' results/genes.bed results/S1.counts | sort -k2,2nr
1_3	190136
1_1	151271
1_2	148103
1_5	137371
1_4	128853
1_6	126427
1_7	117840

Gene 1_3 is the shortest (117 bases) and has the highest TPM although its raw count is among the lowest: it is dense in reads per base. That is exactly what the correction is for, and why a table of raw counts is never ranked directly. (In real RNA-seq, differential expression uses the raw counts with a count model, not TPM; see chapters 27 and S4.)

Lab 9: the pipeline

The labs so far were interactive. A result you will need again belongs in a script, with the habits of P4: strict mode, checked inputs, a clear failure and a log. The script below is the mapping and variant-calling labs in one file. Three features deserve a name. Checks first: the script tests that the reference exists and that the tools are installed before it starts. Skip what is done: the function step runs a command only when its output file is missing, so a re-run after a crash continues where it stopped, and the cost of fixing the last sample is the last sample. Record the versions: the first line of the log holds the date and the versions of the three programs, which is the cheapest reproducibility measure there is.

#!/usr/bin/env bash
# pipeline.sh: reads -> sorted BAM -> variants -> one summary file per sample (results/SAMPLE.summary.tsv).
# usage: ./pipeline.sh REF.fa SAMPLE [SAMPLE ...]   (expects data/SAMPLE.fq; writes into results/)
set -euo pipefail

ref=${1:?usage: $0 REF.fa SAMPLE [SAMPLE ...]}; shift
[[ -s $ref ]] || { echo "reference not found: $ref" >&2; exit 1; }
for tool in   ; do
  command -v "$tool" >/dev/null || { echo "missing tool: $tool" >&2; exit 1; }
done
mkdir -p results
log=results/pipeline.log
{ echo "# $(date +%F) $( --version)  $( --version | head -1 | cut -d' ' -f2)  $( --version | head -1 | cut -d' ' -f2)"; } > "$log"

# a step runs only if its output is missing: a re-run continues where the last one stopped
step() { local out=$1; shift; if [[ -s $out ]]; then echo "skip   $out" >&2; else echo "make   $out" >&2; "$@"; fi; }

map()  {  -ax sr "$ref" "data/$1.fq" 2>>"$log" |  sort -o "results/$1.bam" -;  index "results/$1.bam"; }
call() {  mpileup -f "$ref" "results/$1.bam" 2>>"$log" |  call --ploidy 1 -mv -Ov -o "results/$1.vcf" 2>>"$log"; }

for s in "$@"; do
  [[ -s data/$s.fq ]] || { echo "no reads for $s" >&2; exit 1; }
  step "results/$s.bam" map "$s"
  step "results/$s.vcf" call "$s"
  reads=$( view -c results/$s.bam)
  mapped=$( view -c -F 4 results/$s.bam)
  depth=$( depth -a results/$s.bam | awk '{t+=$3} END {printf "%.1f", t/NR}')
  snps=$( view -H -v snps "results/$s.vcf" | wc -l)
  indels=$( view -H -v indels "results/$s.vcf" | wc -l)
  printf '%s\t%d\t%.2f\t%s\t%d\t%d\n' "$s" "$reads" "$(echo "100*$mapped/$reads" | bc -l)" "$depth" "$snps" "$indels" > "results/$s.summary.tsv"
done
echo "done: $*" >&2

Run on both samples, the script prints what it makes. The per-sample summary files are put together into one table:

$ chmod +x pipeline.sh
$ ./pipeline.sh data/phix174.fa S1 S2 2>&1
make   results/S1.bam
make   results/S1.vcf
make   results/S2.bam
make   results/S2.vcf
done: S1 S2
$ { printf 'sample\treads\tmapped_pct\tmean_depth\tsnps\tindels\n'; cat results/S1.summary.tsv results/S2.summary.tsv; } | column -t -s $'\t'
sample  reads  mapped_pct  mean_depth  snps  indels
S1      6000   100.00      111.4       0     0
S2      6000   100.00      111.4       6     2

Run again, every step is skipped, because the outputs exist. The first line of the log records the versions. And the script fails loudly, with a non-zero exit status that a scheduler or a workflow engine can read, when the reads or the reference are missing:

$ ./pipeline.sh data/phix174.fa S1 S2 2>&1
skip   results/S1.bam
skip   results/S1.vcf
skip   results/S2.bam
skip   results/S2.vcf
done: S1 S2
$ head -1 results/pipeline.log | cut -c1-110
# 2026-11-22 2.28-r1209  
$ ./pipeline.sh data/phix174.fa S9; echo "exit=$?"
exit=1
$ ./pipeline.sh missing.fa S1; echo "exit=$?"
exit=1

To use several cores, hand the sample names to xargs -P 2, which runs two copies of the script at once. The result is the same, and each job wrote only its own files. That is a rule worth stating: parallel jobs must never write to the same file. The first version of this script wrote one shared summary.tsv; run in parallel, the jobs overwrote each other and only the last sample survived. Writing one file per sample and merging them afterwards in a single serial step fixed it. The shared log file has the same weakness, and a production script gives each job its own log.

$ rm -f results/S[12].bam* results/S[12].vcf results/*.summary.tsv
$ nproc | awk '{print $1 " cores available"}'
$ printf 'S1\nS2\n' | xargs -P 2 -I{} bash -c './pipeline.sh data/phix174.fa {} >/dev/null 2>&1 && echo "finished {}"'
$ cat results/S1.summary.tsv results/S2.summary.tsv
12 cores available
finished S1
finished S2
S1	6000	100.00	111.4	0	0
S2	6000	100.00	111.4	6	2

When a pipeline grows past a script, move to a workflow manager ( or ; chapter 22): they keep the idea of skipping finished work, add parallelism and cluster submission, and make the whole run restartable. The step to take first is the one you have just seen, written by hand.

Habits that cost nothing

  • Never edit raw data. Make it read-only (chmod a-w data/*), write results somewhere else, and store a checksum (md5sum data/* > data/MD5.txt, checked later with md5sum -c) so a silent corruption is caught.
  • Test on a tiny file you can check by eye before running on the real data. Every one-liner in this chapter was first tried on a handful of records.
  • Know your coordinate system: GFF and SAM and VCF count from 1; BED counts from 0 with an end that is not included. Convert once, in one place, and test with a gene whose first codon you know.
  • Sort the way tools expect. join, comm and  need sorted input, and the sort order depends on the locale; set LC_ALL=C for a fixed order, and sort -k1,1 -k2,2n for a table keyed on a name and a number.
  • Compress, but keep access. gzip and bgzip files can be streamed with zcat; a BGZF file indexed with can be read at any region without unpacking it (Buffalo, Ch. 13).
  • Send messages to a log, not into a pipe (the head trap above), and read the exit status of every command that matters.
  • Write down what you ran. A script, a notebook or a Makefile is a record; the history of a terminal session is not. The label of every report in Bioinformaniac carries software versions for the same reason.

Check your understanding

A FASTA file reports a length one larger per line than you expect and grep 'ATG$' finds nothing. What do you suspect, and how do you fix it?

Windows line endings: every line ends with a carriage return that is part of the text. file confirms it, and tr -d '\r' removes it.

Which character is Q30 in the Sanger/Illumina 1.8 encoding, and what error probability does it mean?

The character is the ASCII value 30 + 33 = 63, ?. The error probability is 10−30/10 = 0.001.

You search a genome for GGAGG and find 4 matches. Why might the true count be larger?

The motif can also lie on the other strand, where it appears as its reverse complement, CCTCC. Searching only the given strand misses those (here six more).

GFF says a gene runs from 51 to 221. What are its BED coordinates, and how long is it?

BED is 0-based and half-open: 50 and 221. The length is 221 − 50 = 171 (or 221 − 51 + 1 in GFF terms).

A read has FLAG 16. What does that tell you?

It is mapped and lies on the reverse strand (bit 16). A flag of 4 would mean unmapped.

Your gene finder predicts 7 genes in a phage that is known to have about 11. Is the finder wrong?

Not necessarily: overlapping genes, which phiX174 has, are reported once per region. The prediction is a floor; compare with a curated annotation when one exists.

Two parallel jobs both append to summary.tsv. What can go wrong, and what is the fix?

They overwrite or interleave each other's lines, so results are lost or garbled. Let each job write its own file and merge them afterwards in one serial step.

Why is a raw count table not ranked directly, and what does TPM change?

Long genes collect more reads, and libraries differ in size. TPM divides by gene length and by the total of the length-corrected counts, so values are comparable between genes and between samples.

Key references

  1. Hendrix DA. Applied Bioinformatics (the computational analysis of nucleic acid sequences). Open textbook, 2019, licensed CC BY 4.0. Read: Ch. 1 (sequence formats, Lab 1 GNU/Linux and FASTA, Lab 2 FASTQ and quality scores), Ch. 2 (motifs, Lab 3), Ch. 3 (Lab 4,  on the command line), Ch. 5 (genomic data formats, Lab 6 annotation data), Ch. 6 (Lab 7, RNA-seq).
  2. Buffalo V. Bioinformatics Data Skills. O’Reilly, 2015. Read: Ch. 3 (remedial Unix shell), Ch. 6 (data integrity, compression), Ch. 7 (Unix data tools), Ch. 9 (range data and ), Ch. 10 (sequence data), Ch. 11 (alignment data), Ch. 12 (shell scripting, pipelines, find and xargs, Make), Ch. 13 ().
  3. Cock PJA, Fields CJ, Goto N, Heuer ML, Rice PM. The Sanger FASTQ file format for sequences with quality scores, and the Solexa/Illumina FASTQ variants. Nucleic Acids Research 38, 1767–1771 (2010).
  4. Li H, Handsaker B, Wysoker A, et al. The Sequence Alignment/Map format and . Bioinformatics 25, 2078–2079 (2009).
  5. Danecek P, Auton A, Abecasis G, et al. The variant call format and . Bioinformatics 27, 2156–2158 (2011).
  6. Danecek P, Bonfield JK, Liddle J, et al. Twelve years of  and . GigaScience 10, giab008 (2021).
  7. Quinlan AR, Hall IM. : a flexible suite of utilities for comparing genomic features. Bioinformatics 26, 841–842 (2010).
  8. Camacho C, Coulouris G, Avagyan V, et al. +: architecture and applications. BMC Bioinformatics 10, 421 (2009).
  9. Shen W, Le S, Li Y, Hu F. : a cross-platform and ultrafast toolkit for FASTA/Q file manipulation. PLoS ONE 11, e0163962 (2016).
  10. Li H. : pairwise alignment for nucleotide sequences. Bioinformatics 34, 3094–3100 (2018).
  11. Hyatt D, Chen G-L, LoCascio PF, et al. : prokaryotic gene recognition and translation initiation site identification. BMC Bioinformatics 11, 119 (2010).
  12. Sanger F, Air GM, Barrell BG, et al. Nucleotide sequence of bacteriophage φX174 DNA. Nature 265, 687–695 (1977).
Part II · Microbiome analysis

From reads to features: amplicon sequencing

An amplicon study turns millions of reads into a table of counts: rows are sequence features, columns are samples. How those features are defined decides the resolution of everything that follows.

Choosing the marker and region

The 16S rRNA gene has nine hypervariable regions (V1 to V9) separated by conserved stretches where universal primers bind. Short-read studies sequence one or two regions, commonly V4 or V3–V4. No region separates all taxa equally well: V4 is robust and widely used but resolves few species, and primers differ in which groups they miss. Full-length 16S from long reads resolves many more taxa to species and sometimes strain level (Johnson et al., 2019). For fungi the ITS1 or ITS2 region is standard; its length varies a lot between taxa, which affects both PCR and read merging.

OTUs and ASVs

Early pipelines clustered reads into operational taxonomic units (OTUs) at 97 percent identity to absorb sequencing errors. Clustering merges real biological variants and gives results that depend on the dataset. Denoising methods instead learn the error profile of a run and infer the exact biological sequences present, called amplicon sequence variants (ASVs). DADA2 fits a parametric error model to each run and tests whether each unique sequence can be explained as an error from a more abundant one (Callahan et al., 2016). ASVs are reproducible across studies because they are sequences, not cluster labels, and are now the standard unit (Callahan, McMurdie & Holmes, 2017).

A typical DADA2 workflow

  1. Remove primers; inspect quality profiles; trim reads where quality collapses, leaving enough overlap to merge pairs.
  2. Filter reads with too many expected errors.
  3. Learn error rates per run, then denoise forward and reverse reads separately.
  4. Merge pairs, build the sequence table and remove chimeras, which are PCR artefacts joined from two parent sequences.
  5. Track how many reads survive each step. Large losses at merging usually mean the reads were trimmed too short to overlap.

Xia & Sun devote three chapters to the shift from clustering to denoising: how OTU clustering grew out of numerical taxonomy, why a fixed 97 % threshold merges real variants and splits error clouds unevenly, and how DADA2 and Deblur instead model the error process so that every exact sequence variant is either real or an error (Xia & Sun, Bioinformatic and Statistical Analysis of Microbiome Data, ch. 6–8). Mitra's volume compares the main pipelines side by side (Mitra (ed.), Metagenomic Data Analysis, ch. 4).

Taxonomic assignment

Each ASV is assigned a lineage by comparison with a reference database, typically SILVA ( et al., 2013) or -derived 16S sets for bacteria and archaea, and  for fungi. The naïve Bayesian classifier counts shared k-mers between the query and reference sequences and reports a bootstrap confidence at each rank (Wang et al., 2007). Classifiers trained on the exact amplified region perform better than those trained on full-length genes (Bokulich et al., 2018). Species-level calls from short amplicons should be treated with caution; genus is usually the finest reliable rank.

Phylogeny

A phylogenetic tree of the ASVs is needed for phylogeny-aware measures such as Faith's phylogenetic diversity and UniFrac. Trees are usually built by aligning ASVs with  and inferring a tree with , or by inserting ASVs into a reference tree.

Check your understanding

Two sequences differ at 1 of 250 positions. Are they one OTU or two ASVs?

One OTU at 97 % identity, but two ASVs if the denoiser judges the difference to be real rather than a sequencing error.

Why can a V4 amplicon rarely name a species?

A 250 bp region of a conserved gene often has identical sequences across close species; genus is usually the reliable level.

Key references

  1. Callahan BJ, McMurdie PJ, Rosen MJ, et al. DADA2: high-resolution sample inference from Illumina amplicon data. Nature Methods 13, 581–583 (2016).
  2. Callahan BJ, McMurdie PJ, Holmes SP. Exact sequence variants should replace operational taxonomic units in marker-gene data analysis. The ISME Journal 11, 2639–2643 (2017).
  3. Bolyen E, Rideout JR, Dillon MR, et al. Reproducible, interactive, scalable and extensible microbiome data science using QIIME 2. Nature Biotechnology 37, 852–857 (2019).
  4.  C, Pruesse E, Yilmaz P, et al. The SILVA ribosomal RNA gene database project. Nucleic Acids Research 41, D590–D596 (2013).
  5. Wang Q, Garrity GM, Tiedje JM, Cole JR. Naïve Bayesian classifier for rapid assignment of rRNA sequences into the new bacterial taxonomy. Applied and Environmental Microbiology 73, 5261–5267 (2007).
  6. Bokulich NA, Kaehler BD, Rideout JR, et al. Optimizing taxonomic classification of marker-gene amplicon sequences with QIIME 2's q2-feature-classifier plugin. Microbiome 6, 90 (2018).
  7. Johnson JS, Spakowicz DJ, Hong B-Y, et al. Evaluation of 16S rRNA gene sequencing for species and strain-level microbiome analysis. Nature Communications 10, 5029 (2019).
  8. Xia Y, Sun J. Bioinformatic and Statistical Analysis of Microbiome Data: From Raw Sequences to Advanced Modeling with QIIME 2 and R. Springer, 2023. Read: Ch. 4 Building Feature Table and Feature Representative Sequences from Raw Reads, ch. 5 Assigning Taxonomy, Building Phylogenetic Tree, ch. 6–8 on OTU methods and moving beyond them.
  9. Mitra S (ed.). Metagenomic Data Analysis. Methods in Molecular Biology 2649. Humana Press, 2023. Read: Ch. 4 Amplicon Sequencing Pipelines in Metagenomics and ch. 5 A Practical Guide to 16S rRNA Microbiome Analysis.
  10. Christensen H (ed.). Introduction to Bioinformatics in Microbiology. Learning Materials in Biosciences, Springer, 2018. Read: Ch. 8 16S rRNA Amplicon Sequencing for Metagenomics.
  11. Paradis E. Analysis of Phylogenetics and Evolution with R, 2nd edition. Springer, 2012. Read: Ch. 5 Phylogeny Estimation.
Part II · Microbiome analysis

Taxonomic profiling of shotgun metagenomes

Shotgun sequencing reads every genome in the sample, not one gene. Profilers turn those reads into species abundances in two different ways, and the difference matters when you interpret the numbers.

Read classifiers

 2 assigns each read to the lowest common ancestor of all reference genomes that share its k-mers (Wood, Lu & Langmead, 2019). It is fast and classifies reads from any domain in the database, including host, fungi and viruses. Because many reads stop at genus or family when species share sequence, Bracken redistributes them to species using the expected read distribution of each genome (Lu et al., 2017). The output is a read share: the fraction of sequenced reads that came from each taxon. Larger genomes contribute more reads per cell, so read share is not the same as cell share.

Zhong Wang explains the algorithms behind both families, from k-mer matching and lowest common ancestor assignment to marker-gene and binning approaches (Zhong Wang, Introduction to Computational Metagenomics, ch. 6).

Marker-gene profilers

MetaPhlAn maps reads only to a curated set of clade-specific marker genes, about a million of them in MetaPhlAn 4, and estimates abundances from marker coverage (Blanco-Míguez et al., 2023). Because coverage is normalised by marker length, the result approximates the relative abundance of genomes, closer to a cell share. MetaPhlAn 4 defines species-level genome bins (SGBs) from large collections of metagenome-assembled genomes, so it can profile species that have never been cultured (Pasolli et al., 2019). Reads from organisms without markers, and from host or unknown sources, are simply not counted, so profiles sum to 100 percent of the known fraction.

Property + BrackenMetaPhlAn 4
What is countedReads, assigned by k-mersReads on clade-specific marker genes
UnitsRead fractionGenome (cell-like) fraction
Unknown organismsUnclassified or misassigned to relativesCovered when an SGB exists, otherwise ignored
Main riskFalse positives from database gaps; needs confidence filteringMisses taxa without markers; low-abundance detection threshold

Database choice is a modelling choice

Both approaches see only what is in their database. A taxon missing from the reference set is either invisible or attributed to its nearest relative. When comparing studies, use the same tool and database version; when reporting, give both. For environments far from the human gut, such as soils or fermented foods, database completeness is often the main limit, and assembly-based approaches (binning metagenome-assembled genomes) recover organisms that profilers cannot see (Quince et al., 2017).

Validation with a mock community

Running a mock community through the same pipeline shows whether expected species are recovered, whether unexpected ones appear (false positives) and how far abundances drift from the truth. Recovering all expected species with few false positives is a better test than matching the exact proportions, which extraction bias distorts anyway.

Check your understanding

 assigns 40 % of reads to a genus that MetaPhlAn does not report. Which is wrong?

Possibly neither:  counts reads (biased to large genomes and database content), MetaPhlAn counts marker-gene abundance. Check the database and use a mock community.

Why does a missing species in the database cause false positives, not just missed reads?

Its reads are assigned to the closest relative that is in the database, which then appears present.

Part II · Microbiome analysis

Filtering, quality control and contaminants

A raw feature table contains real taxa, off-target sequences, contaminants, artefacts and samples that failed. Filtering removes what cannot be biology, but every filter is a decision that must be reported.

Off-target and unassigned features

16S primers also amplify chloroplast and mitochondrial rRNA, which are bacterial in origin. In plant, food or host-tissue samples these can dominate. Features assigned to Chloroplast (order) or Mitochondria (family), and features unassigned at kingdom or phylum level, are normally removed before analysis.

Prevalence and abundance filters

Most features in a microbiome table are rare: seen in one or two samples at a few reads. They add noise, inflate the number of tests in differential abundance and destabilise network inference. Common filters keep features present in at least a minimum number or fraction of samples (for example 5 to 10 percent) and above a minimum mean relative abundance. These filters should be applied without looking at the group labels, otherwise they bias downstream tests. Alpha diversity is the exception: richness estimators need the rare features, so compute diversity on the unfiltered table.

Filtering is a modelling decision, not housekeeping. Xia, Sun & Chen show how prevalence filters reduce the number of tests and the share of zeros, which improves power, but also how a filter applied separately to each group creates differences that are not there (Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R, ch. 7). Apply one filter to all samples, decide it before testing, and report it.

Sample-level checks

Samples with very low read counts after processing are usually failed libraries. Plot depth per sample and per group: if one group systematically has lower depth, depth becomes a confounder. Ordinations of all samples, including controls, also reveal outliers and mislabelled samples. Controls should cluster away from real samples.

Contaminant identification

Contaminant DNA comes from reagents, kits and the lab, and it behaves in two recognisable ways (Davis et al., 2018):

  • Frequency. A contaminant is added in roughly constant amounts, so its relative frequency is inversely related to the total DNA concentration of the sample. Real taxa do not show this pattern.
  • Prevalence. A contaminant is more prevalent in negative controls than in real samples.

The decontam method tests both patterns per feature and flags features with a score below a threshold. Flagged features should be inspected, not deleted blindly: common gut taxa can appear in blanks through well-to-well cross-contamination, and removing them from real samples would be wrong. Known reagent contaminant genera, such as Ralstonia, Bradyrhizobium and Sphingomonas, deserve suspicion in low-biomass samples (Salter et al., 2014).

Check your understanding

Why should filtering thresholds be fixed before looking at the test results?

Choosing the filter that gives the most significant result is a hidden form of multiple testing and inflates false discoveries.

decontam flags a taxon whose frequency falls as DNA concentration rises. Why?

Contaminant DNA is roughly constant per extraction, so its share is largest in samples with little real DNA.

Part II · Microbiome analysis

Compositionality, normalisation and transformation

Sequencing counts carry information only about ratios between taxa. Normalisation corrects for unequal sequencing depth; transformation puts the data on a scale where standard statistics behave. They solve different problems and are best chosen separately.

Why the data are compositional

The total number of reads per sample is set by the sequencing run, not by the biology. Counts are therefore a random sample of a composition, and only ratios between parts are meaningful (Gloor et al., 2017). Two consequences follow. First, correlations between relative abundances are biased: when one taxon increases, the others must decrease, creating negative correlations that do not exist in the community. Second, a taxon can appear to increase only because others decreased. Aitchison's log-ratio framework (Aitchison, 1986) is the principled way around this.

The full theory, from Aitchison's simplex to CLR, ILR and ALR transformations and their use in tests, is in Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R, ch. 10, and Xia & Sun, Bioinformatic and Statistical Analysis of Microbiome Data, ch. 14.

Normalisation: dealing with depth

  • Total-sum scaling (TSS) divides by library size to give proportions. Simple, but sensitive to a few dominant taxa.
  • Rarefying subsamples every sample to the same depth. It discards data and adds randomness, and is inadmissible for differential abundance testing (McMurdie & Holmes, 2014). It remains defensible for diversity estimates, where unequal effort biases richness directly (Weiss et al., 2017; Schloss, 2024).
  • Size-factor methods estimate a scaling factor from features assumed not to change: the median-of-ratios (RLE) of  (Love, Huber & Anders, 2014), TMM (Robinson & Oshlack, 2010), upper quartile, and GMPR, designed for zero-inflated microbiome data (Chen et al., 2018). CSS scales by a data-driven quantile (Paulson et al., 2013).
  • Absolute scaling multiplies proportions by measured cell load or uses spike-ins (chapter 2). It is the only approach that recovers real changes in total abundance.

Transformation: getting a usable scale

Counts are skewed and their variance grows with the mean, so most methods based on Euclidean distance or linear models need a transformation first.

  • Centred log-ratio (CLR): the log of each count divided by the geometric mean of the sample. Euclidean distances on CLR values give the Aitchison distance. Zeros must first be replaced, by a pseudocount or better by a Bayesian-multiplicative method.
  • Robust CLR computes the geometric mean over non-zero values only and handles sparse data better (Martino et al., 2019).
  • Hellinger (square root of proportions) makes Euclidean methods such as PCA and RDA suitable for abundance data (Legendre & Gallagher, 2001).
  • Log counts per million and variance-stabilising transformation come from RNA-seq and work well when depth is moderately uniform.
  • Presence/absence keeps only occurrence and is useful for rare biosphere questions.

Check your understanding

Taxon A doubles in absolute abundance while everything else is unchanged. What happens to the relative abundance of taxon B?

It falls, although B did not change. This is the compositional effect: one change moves every proportion.

Why does CLR need zeros to be replaced first?

The log of zero is undefined; a pseudocount or a model-based replacement is required, and the choice affects rare taxa most.

Key references

  1. Gloor GB, Macklaim JM, Pawlowsky-Glahn V, Egozcue JJ. Microbiome datasets are compositional: and this is not optional. Frontiers in Microbiology 8, 2224 (2017).
  2. Aitchison J. The Statistical Analysis of Compositional Data. Chapman and Hall, London (1986).
  3. McMurdie PJ, Holmes S. Waste not, want not: why rarefying microbiome data is inadmissible. PLoS Computational Biology 10, e1003531 (2014).
  4. Weiss S, Xu ZZ, Peddada S, et al. Normalization and microbial differential abundance strategies depend upon data characteristics. Microbiome 5, 27 (2017).
  5. Schloss PD. Rarefaction is currently the best approach to control for uneven sequencing effort in amplicon sequence analyses. mSphere 9, e00354-23 (2024).
  6. Love MI, Huber W, Anders S. Moderated estimation of fold change and dispersion for RNA-seq data with . Genome Biology 15, 550 (2014).
  7. Robinson MD, Oshlack A. A scaling normalization method for differential expression analysis of RNA-seq data. Genome Biology 11, R25 (2010).
  8. Chen L, Reeve J, Zhang L, et al. GMPR: a robust normalization method for zero-inflated count data with application to microbiome sequencing data. PeerJ 6, e4600 (2018).
  9. Paulson JN, Stine OC, Bravo HC, Pop M. Differential abundance analysis for microbial marker-gene surveys. Nature Methods 10, 1200–1202 (2013).
  10. Martino C, Morton JT, Marotz CA, et al. A novel sparse compositional technique reveals microbial perturbations. mSystems 4, e00016-19 (2019).
  11. Legendre P, Gallagher ED. Ecologically meaningful transformations for ordination of species data. Oecologia 129, 271–280 (2001).
  12. Xia Y, Sun J, Chen D-G. Statistical Analysis of Microbiome Data with R. ICSA Book Series in Statistics, Springer, 2018. Read: Ch. 10 Compositional Analysis of Microbiome Data.
  13. Xia Y, Sun J. Bioinformatic and Statistical Analysis of Microbiome Data: From Raw Sequences to Advanced Modeling with QIIME 2 and R. Springer, 2023. Read: Ch. 14 Compositional Analysis of Microbiome Data.
  14. Mitra S (ed.). Metagenomic Data Analysis. Methods in Molecular Biology 2649. Humana Press, 2023. Read: Ch. 2, section Best Practices in Dimensionality Reduction.
Part II · Microbiome analysis

Alpha diversity

Alpha diversity summarises a single community in one number: how many kinds of organisms it holds and how evenly they are spread. It is the most reported microbiome statistic and one of the easiest to get wrong.

Richness, evenness and Hill numbers

Richness counts the taxa present. Shannon entropy and the Simpson index also weight taxa by their abundance, so they reflect evenness as well. These indices sit on different scales, which makes them hard to compare. Hill numbers put them on one scale, the effective number of taxa (Hill, 1973):

qD = ( Σ piq )1/(1−q)

With q = 0 this is richness, q = 1 (as a limit) is the exponential of Shannon entropy, and q = 2 is the inverse Simpson index. Plotting D across q gives a diversity profile: a community with many rare taxa has high D at q = 0 that falls steeply as q rises. Faith's phylogenetic diversity sums the branch lengths of the tree spanned by the taxa present and rewards communities that cover distant lineages (Faith, 1992).

Sampling effort

Observed richness grows with sequencing depth, so samples sequenced more deeply look more diverse. There are three common ways to compare at equal effort. Rarefying once to the smallest depth is simple but adds noise and discards data. Repeated rarefaction averages over many subsamples and reduces that noise (Schloss, 2024). Coverage-based standardisation compares samples at equal sample completeness rather than equal read count, which is fairer when communities differ in how many rare taxa they hold (Chao & Jost, 2012). Richness is the most sensitive to effort; Shannon and Simpson much less (Willis, 2019).

The missing-species problem has a famous history. Fisher was asked how many new butterfly species a collector would find in a second year of trapping in Malaya; his answer, refined by Good and Toulmin, is the alternating series above. Efron & Hastie use it to open their discussion of empirical Bayes (Efron & Hastie, Computer Age Statistical Inference, section 6.2). MicroRes reports these quantities for every sample in step 06 ("How many taxa are still unseen?"), together with the sample size a follow-up study needs (chapter 2).

Testing

Diversity values are one number per sample and can be compared with ordinary tests: Wilcoxon or Kruskal–Wallis for groups, linear models when there are covariates, and mixed models when subjects are measured repeatedly. Report effect sizes, not only p-values: a statistically significant difference of two taxa in a richness of 300 is rarely meaningful.

Check your understanding

Sample A has 200 species, 1 dominant; sample B has 150 species, evenly spread. Which is more diverse?

It depends on the order q: richness (q = 0) favours A, Shannon or Simpson (q = 1, 2) likely favour B. Report more than one.

Why is the number of singletons informative?

Many species seen once means many more are unseen. Chao1 and Good–Turing reasoning use singletons and doubletons to estimate the missing species.

Key references

  1. Hill MO. Diversity and evenness: a unifying notation and its consequences. Ecology 54, 427–432 (1973).
  2. Chao A, Jost L. Coverage-based rarefaction and extrapolation: standardizing samples by completeness rather than size. Ecology 93, 2533–2547 (2012).
  3. Faith DP. Conservation evaluation and phylogenetic diversity. Biological Conservation 61, 1–10 (1992).
  4. Willis AD. Rarefaction, alpha diversity, and statistics. Frontiers in Microbiology 10, 2407 (2019).
  5. Schloss PD. Rarefaction is currently the best approach to control for uneven sequencing effort in amplicon sequence analyses. mSphere 9, e00354-23 (2024).
  6. Xia Y, Sun J. Bioinformatic and Statistical Analysis of Microbiome Data: From Raw Sequences to Advanced Modeling with QIIME 2 and R. Springer, 2023. Read: Ch. 9 Alpha Diversity.
  7. Xia Y, Sun J, Chen D-G. Statistical Analysis of Microbiome Data with R. ICSA Book Series in Statistics, Springer, 2018. Read: Ch. 6 Community Diversity Measures and Calculations and ch. 8 Univariate Community Analysis.
  8. Efron B, Hastie T. Computer Age Statistical Inference: Algorithms, Evidence, and Data Science. Cambridge University Press, 2016. Read: Section 6.2 The Missing-Species Problem: Fisher's butterflies and how many species remain unseen.
  9. Good IJ, Toulmin GH. The number of new species, and the increase in population coverage, when a sample is increased in size. Biometrika 43, 45–63 (1956).
  10. Shen T-J, Chao A, Lin C-F. Predicting the number of new species in further taxonomic sampling. Ecology 84, 798–804 (2003).
Part II · Microbiome analysis

Beta diversity and ordination

Beta diversity asks how different two communities are. A distance between every pair of samples, an ordination to look at it, and a permutation test to judge it form the backbone of most microbiome papers.

Choosing a distance

The distance encodes what you consider "different":

  • Bray–Curtis weights abundant taxa and ignores shared absences. It is the ecological default for abundance data but is affected by depth unless the data are normalised or rarefied.
  • Jaccard uses presence/absence only and emphasises rare membership.
  • Aitchison is the Euclidean distance between CLR-transformed samples. It respects compositionality and is scale-invariant. Robust Aitchison handles sparsity better (Martino et al., 2019).
  • UniFrac measures the fraction of tree branch length unique to one sample. Unweighted UniFrac uses presence/absence; weighted UniFrac uses abundance (Lozupone & Knight, 2005).

Different distances can lead to different conclusions, and that is information. A treatment that shifts unweighted UniFrac but not Bray–Curtis changes which rare lineages are present without changing the dominant taxa.

Ordination is unsupervised learning: PCA, PCoA and NMDS all try to show high-dimensional distances in two dimensions, and the share of variance explained by each axis tells you how much they hide (James et al., An Introduction to Statistical Learning, ch. 12; Xia & Sun, Bioinformatic and Statistical Analysis of Microbiome Data, ch. 10).

Ordination

Principal coordinates analysis (PCoA) places samples in a few dimensions that preserve the chosen distance as well as possible. The share of variation on each axis tells you how much of the structure the plot shows. PCA is PCoA with Euclidean distance and works directly on CLR or Hellinger data. Non-metric multidimensional scaling (NMDS) preserves only the rank order of distances and reports a stress value; stress above about 0.2 means the picture is unreliable. Constrained ordinations (RDA, CCA, distance-based RDA) show only the variation explained by chosen metadata and are useful for asking how much a set of variables explains.

Testing: PERMANOVA and dispersion

PERMANOVA partitions the distance matrix among groups or covariates and assesses significance by permuting labels (Anderson, 2001). Its R² is the fraction of variation explained. In microbiome studies, single factors often explain 1 to 10 percent, which is still meaningful given how variable communities are. Two cautions apply. First, PERMANOVA is sensitive to differences in spread (dispersion) as well as location, so test dispersion with PERMDISP and report both. Second, repeated measures must be handled by restricting permutations within subject (strata); otherwise the test treats correlated samples as independent. Covariates are added sequentially, so their order matters unless marginal tests are requested.

Check your understanding

PERMANOVA gives p = 0.001 but PERMDISP is also significant. What can you conclude?

Groups differ, but possibly only in spread, not in centre. Look at the ordination before claiming a shift in composition.

Why does Jaccard react to rare taxa more than Bray–Curtis?

Jaccard uses presence and absence only, so a rare taxon counts as much as a dominant one; Bray–Curtis weights by abundance.

Part II · Microbiome analysis

Differential abundance

Which taxa differ between groups is the question most microbiome studies end on, and the one where methods disagree most. Using more than one method and reading the agreement is the honest answer.

Why it is hard

Microbiome counts are sparse (most entries are zero), over-dispersed, compositional and measured at unequal depth. A method must decide what "no change" means. Because only ratios are observed, a test on relative data can only say that a taxon changed relative to something: the rest of the community, a reference taxon or an assumed unchanged majority. When many taxa change in the same direction, methods that assume most taxa are unchanged break down.

Microbiome counts are also over-dispersed (the variance far exceeds the mean, unlike the Poisson model) and full of zeros. Faraway's chapter on count regression is the clearest introduction to Poisson, quasi-Poisson and negative binomial models (Faraway, Extending the Linear Model with R, ch. 5); Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R, ch. 11–12, apply them to microbiome data, including zero-inflated and hurdle models. Benchmarks agree that no single method wins everywhere, which is why MicroRes runs several and reports the taxa they agree on.

The main families of methods

  • Count models from RNA-seq.  fits a negative binomial model with size factors and shrinks fold changes (Love, Huber & Anders, 2014). It is powerful but can be anti-conservative on very sparse data; size factors such as poscounts or GMPR help.
  • Bias-corrected log-linear models. ANCOM-BC estimates and removes the sample-specific sampling fraction and reports fold changes with confidence intervals (Lin & Peddada, 2020). LinDA fits linear models to CLR data and corrects the compositional bias afterwards; it is fast and handles mixed models (Zhou et al., 2022).
  • Log-ratio methods. ALDEx2 draws Monte Carlo instances from a Dirichlet distribution, applies CLR and tests each instance, so the uncertainty of low counts is propagated (Fernandes et al., 2014). It tends to be conservative.
  • General linear models. MaAsLin2 fits per-feature models on transformed data with fixed and random effects (Mallick et al., 2021).

A comparison of 14 methods on 38 datasets found that they identified very different sets of significant taxa, and that ALDEx2 and ANCOM-II gave the most consistent results across datasets (Nearing et al., 2022). The practical lesson is to run several methods with different assumptions and give most weight to taxa they agree on.

Multiple testing and effect sizes

With hundreds of taxa tested, control the false discovery rate with the Benjamini–Hochberg procedure (Benjamini & Hochberg, 1995). Report the fold change and its uncertainty, the prevalence of the taxon in each group, and the method. A taxon present in 3 of 40 samples can be "significant" in some methods and is rarely worth a biological claim.

Check your understanding

Why can a Poisson model not be used for microbiome counts?

Poisson forces variance = mean; microbiome counts vary far more between samples (over-dispersion). The negative binomial adds a dispersion parameter.

Two methods give different lists of significant taxa. Is one wrong?

Not necessarily: they test different hypotheses (relative versus log-ratio change) under different assumptions. Taxa found by several methods are the robust core.

Key references

  1. Nearing JT, Douglas GM, Hayes MG, et al. Microbiome differential abundance methods produce different results across 38 datasets. Nature Communications 13, 342 (2022).
  2. Lin H, Peddada SD. Analysis of compositions of microbiomes with bias correction. Nature Communications 11, 3514 (2020).
  3. Fernandes AD, Reid JN, Macklaim JM, et al. Unifying the analysis of high-throughput sequencing datasets: characterizing RNA-seq, 16S rRNA gene sequencing and selective growth experiments by compositional data analysis. Microbiome 2, 15 (2014).
  4. Zhou H, He K, Chen J, Zhang X. LinDA: linear models for differential abundance analysis of microbiome compositional data. Genome Biology 23, 95 (2022).
  5. Love MI, Huber W, Anders S. Moderated estimation of fold change and dispersion for RNA-seq data with . Genome Biology 15, 550 (2014).
  6. Mallick H, Rahnavard A, McIver LJ, et al. Multivariable association discovery in population-scale meta-omics studies. PLoS Computational Biology 17, e1009442 (2021).
  7. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B 57, 289–300 (1995).
  8. Xia Y, Sun J, Chen D-G. Statistical Analysis of Microbiome Data with R. ICSA Book Series in Statistics, Springer, 2018. Read: Ch. 11 Modeling Over-Dispersed Microbiome Data and ch. 12 Modeling Zero-Inflated Microbiome Data.
  9. Xia Y, Sun J. Bioinformatic and Statistical Analysis of Microbiome Data: From Raw Sequences to Advanced Modeling with QIIME 2 and R. Springer, 2023. Read: Ch. 12 Differential Abundance Analysis of Microbiome Data, ch. 13 Zero-Inflated Beta Models, ch. 15–18 mixed models for longitudinal data.
  10. Faraway JJ. Extending the Linear Model with R, 2nd edition. CRC Press, 2016. Read: Ch. 5 Count Regression: Poisson, over-dispersion and the negative binomial.
  11. Efron B, Hastie T. Computer Age Statistical Inference: Algorithms, Evidence, and Data Science. Cambridge University Press, 2016. Read: Ch. 15 Large-Scale Hypothesis Testing and False-Discovery Rates.
Part II · Microbiome analysis

Associations and co-occurrence networks

Correlating taxa with each other or with host variables is easy to do and easy to over-interpret. Compositional methods remove the worst artefacts; careful reading removes most of the rest.

Taxa and host variables

Associations between taxa and continuous metadata, such as a metabolite, a clinical score or age, are usually tested with Spearman correlation on a transformed table, or with per-feature linear models that adjust for covariates. Partial correlations remove the shared influence of a third variable. Because every taxon is tested against every variable, multiple-testing correction is essential.

Taxon–taxon networks

Plain correlations between relative abundances are biased by compositionality (chapter 7). Two methods address this. SparCC estimates correlations between the underlying absolute abundances from log-ratio variances, assuming the network is sparse (Friedman & Alm, 2012). SPIEC-EASI goes further and estimates conditional dependence, through neighbourhood selection or the graphical lasso on CLR data, so that two taxa linked only through a third are not connected directly (Kurtz et al., 2015). SPIEC-EASI networks are sparser and more interpretable, at the cost of more computation and a need for many samples. Several hundred samples are a reasonable target, and networks built from 20 samples are mostly noise.

The problem is old: Karl Pearson warned in 1897 of "spurious correlation" between ratios that share a denominator. Proportions all share the sample total, so two unrelated taxa become negatively correlated when a third one blooms. SparCC and SPIEC-EASI estimate associations between log-ratios and add sparsity, assuming most taxa do not interact directly (Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R, ch. 10). Graph concepts such as degree, hubs, modules and betweenness are explained for biological networks in Baxevanis, Bader & Wishart, Bioinformatics, ch. 13, and the ecology of microbial interactions in Zhong Wang, Introduction to Computational Metagenomics, ch. 10.

Reading a network

Common summaries are node degree, betweenness centrality, modules (clusters of densely connected taxa, for example from the Louvain algorithm (Blondel et al., 2008)) and the balance of positive and negative edges. Highly connected or central taxa are sometimes called keystone taxa, but a statistical hub is not the same as an ecological keystone (Berry & Widder, 2014). An edge means co-variation across samples: it can come from a shared niche, a shared response to diet or a direct interaction, and the network cannot tell these apart (Röttjers & Faust, 2018).

Comparing networks between groups and samples

Comparing summary numbers of two networks, such as density or modularity, says little about where they differ. Differential network analysis asks, for every pair of features, whether their association changes between the groups. With correlations this is a test of the difference between two Fisher-transformed coefficients (Fukushima, 2013; McKenzie et al., 2016). The pairs whose association changes form a differential network. Clustering that network gives modules of features that rewire together, for example a group of taxa that co-vary tightly in controls but fall apart after antibiotics. For each module, the change in connectivity can be tested by permuting group labels. That turns a picture into a statistic and highlights the modules that carry the difference. NetCoMi implements related comparisons for microbiome data (Peschel et al., 2021).

Networks can also be built between samples: each sample is linked to its most similar samples, and community detection splits the graph into clusters of samples with similar whole profiles. Cross-tabulating these clusters against a study variable shows whether groups form their own clusters or mix. This is a graph-based complement to ordination and PERMANOVA (chapter 9).

Differential associations need more samples than a single network: each group must support its own correlation estimates. With fewer than about 20 to 30 samples per group, few changes pass a false discovery threshold. Results are then best read as exploratory.

Check your understanding

Two taxa have a strong positive correlation. Do they interact?

Not necessarily: they may share a habitat preference, respond to the same host factor, or be correlated by the compositional constraint. A network edge is a hypothesis.

Why do networks from 20 samples look denser than from 200?

With few samples, chance correlations are large; sparsity penalties and stability selection need more samples to separate signal from noise.

Key references

  1. Friedman J, Alm EJ. Inferring correlation networks from genomic survey data. PLoS Computational Biology 8, e1002687 (2012).
  2. Kurtz ZD, Müller CL, Miraldi ER, et al. Sparse and compositionally robust inference of microbial ecological networks. PLoS Computational Biology 11, e1004226 (2015).
  3. Röttjers L, Faust K. From hairballs to hypotheses: biological insights from microbial networks. FEMS Microbiology Reviews 42, 761–780 (2018).
  4. Berry D, Widder S. Deciphering microbial interactions and detecting keystone species with co-occurrence networks. Frontiers in Microbiology 5, 219 (2014).
  5. Blondel VD, Guillaume J-L, Lambiotte R, Lefebvre E. Fast unfolding of communities in large networks. Journal of Statistical Mechanics P10008 (2008).
  6. Fukushima A. DiffCorr: an R package to analyze and visualize differential correlations in biological networks. Gene 518, 209–214 (2013).
  7. McKenzie AT, Katsyv I, Song W-M, Wang M, Zhang B. DGCA: a comprehensive R package for differential gene correlation analysis. BMC Systems Biology 10, 106 (2016).
  8. Peschel S, Müller CL, von Mutius E, Boulesteix A-L, Depner M. NetCoMi: network construction and comparison for microbiome data in R. Briefings in Bioinformatics 22, bbaa290 (2021). Read: Ch. 13 Networks and Pathways: graph concepts, hubs and modules.
  9. Wang Z. Introduction to Computational Metagenomics. World Scientific, 2022. Read: Ch. 10 Interactions Between Microbes and Their Environment.
  10. Xia Y, Sun J, Chen D-G. Statistical Analysis of Microbiome Data with R. ICSA Book Series in Statistics, Springer, 2018. Read: Ch. 10: why correlation of compositions is spurious.
  11. Baxevanis AD, Bader GD, Wishart DS (eds.). Bioinformatics, 4th edition. Wiley, 2020.
  12. Pearson K. Mathematical contributions to the theory of evolution. On a form of spurious correlation which may arise when indices are used in the measurement of organs. Proceedings of the Royal Society of London 60, 489–498 (1897).
Part II · Microbiome analysis

Machine learning and biomarkers

Predicting a phenotype from the microbiome tests whether the community carries information about it. Done carelessly, the same exercise produces impressive accuracies that vanish on new data.

Models that suit microbiome data

Microbiome datasets have many more features than samples. Regularised linear models handle this well: LASSO selects a small set of features, ridge shrinks all of them, and the elastic net mixes the two and copes better with correlated taxa (Zou & Hastie, 2005). Random forests capture non-linear effects and interactions and need little tuning (Breiman, 2001). In a systematic comparison, L2-regularised logistic regression performed about as well as more complex models on microbiome classification while being far easier to interpret (Topçuoğlu et al., 2020).

Validation without leakage

The model must be evaluated on samples it never saw during training, including during feature selection and tuning. Three rules prevent most leakage:

  1. Do every data-dependent step inside the cross-validation loop. Filtering by prevalence is safe if it ignores labels; selecting features by their association with the outcome must happen within each training fold.
  2. Keep related samples together. Repeated samples from one subject, or animals from one cage, must fall in the same fold (grouped cross-validation). Otherwise the model learns individuals, not the phenotype.
  3. Tune in an inner loop. Hyperparameters chosen on the test folds give optimistic estimates; nested cross-validation avoids this.

Report the area under the ROC curve (or balanced accuracy) with its spread across folds, and compare with a model trained on shuffled labels. An external validation cohort is the strongest evidence.

James et al., An Introduction to Statistical Learning explains cross-validation and the bootstrap (ch. 5), regularised regression such as the lasso (ch. 6) and random forests (ch. 8) with the clearest diagrams in the literature; it is freely available from the authors.

Interpreting features

Model coefficients and importances show what the model used, not what causes the phenotype. Features that are selected in most folds are more trustworthy than those selected once. Linear discriminant analysis effect size (LEfSe) is a popular biomarker ranking (Segata et al., 2011), but it is a univariate screen, not a predictive model, and should not be reported as one.

Check your understanding

You select the 50 most differential taxa on all samples, then cross-validate a classifier on them. What is wrong?

Feature selection saw the test samples: the reported accuracy is optimistic. Selection must happen inside each training fold.

Why is AUC preferred over accuracy for unbalanced groups?

With 90 % controls, predicting 'control' always gives 90 % accuracy; AUC measures ranking across all thresholds.

Part II · Microbiome analysis

Functional metagenomics

Taxonomy tells you who is there; function tells you what the community can do. Functional profiles add a layer of interpretation, and a layer of assumptions that should be stated.

Where functional profiles come from

  • HUMAnN maps shotgun reads to the pangenomes of species detected by MetaPhlAn, then translates the remaining reads against UniRef proteins. It reports gene families and MetaCyc pathways, each split by the species that contribute them (stratified output) (Beghini et al., 2021).
  • PICRUSt2 predicts gene content from 16S ASVs by placing them in a reference tree and inferring the genomes of their relatives (Douglas et al., 2020). It is a prediction, not a measurement, and is least reliable for environments poorly represented by sequenced genomes.
  • annotates predicted genes, usually from assemblies or metagenome-assembled genomes, with orthologous groups, KEGG orthologs, enzymes, CAZymes and GO terms (Cantalapiedra et al., 2021).

Cleaning and normalising

HUMAnN tables contain UNMAPPED, UNINTEGRATED and UNGROUPED rows that are bookkeeping, not functions; they are usually removed or reported separately. Gene families can be regrouped to KEGG orthologs or enzyme commission numbers for interpretation. Functional tables are compositional just like taxonomic ones, so the same normalisation logic applies (chapter 7). Copies per million (CPM) or relative abundance followed by a log transformation are common choices.

Testing functions

Per-function models with covariates and random effects, in the style of MaAsLin (Mallick et al., 2021), are standard. MaAsLin 3 separates abundance (how much, when present) from prevalence (whether present) and adds a correction for compositionality (Nickols et al., 2024). With thousands of functions, false discovery rate control is essential, and pathway-level results are easier to interpret than gene-family results.

Who carries a function

Stratified output links each function to its contributing taxa, so a change in a pathway can be traced to the species behind it. A function carried by many taxa is functionally redundant: losing one carrier changes little. Tian and colleagues define functional redundancy from the gap between taxonomic and functional diversity (Tian et al., 2020). Contributional diversity, the diversity of carriers for one function, distinguishes functions that depend on a single keystone carrier from those spread across the community. Bipartite species–function networks make these patterns visible; their nestedness (NODF) shows whether specialist carriers hold subsets of the functions of generalists (Almeida-Neto et al., 2008).

A functional profile from DNA describes potential; the community may not express it. The step from genes to activity is covered by metatranscriptomics and by function-based screens, in which DNA from the community is cloned and tested for an activity directly, as described by Charles, Liles & Sessitsch in Functional Metagenomics: Tools and Applications. For the computational side, Zhong Wang, Introduction to Computational Metagenomics, ch. 7, covers gene prediction and annotation in metagenomes and ch. 8 the recovery of genomes (binning), which is what makes "who does what" possible beyond reference databases.

Check your understanding

A pathway is 'present' in a sample. Does the community perform it?

Not necessarily: DNA shows potential. Activity needs RNA (metatranscriptomics), proteins or metabolites.

Why are stratified (per-taxon) functional profiles useful?

They show whether a change in a pathway comes from one taxon gaining abundance or from a shift across many carriers.

Key references

  1. Beghini F, McIver LJ, Blanco-Míguez A, et al. Integrating taxonomic, functional, and strain-level profiling of diverse microbial communities with bioBakery 3. eLife 10, e65088 (2021).
  2. Douglas GM, Maffei VJ, Zaneveld JR, et al. PICRUSt2 for prediction of metagenome functions. Nature Biotechnology 38, 685–688 (2020).
  3. Cantalapiedra CP, Hernández-Plaza A, Letunic I, Bork P, Huerta-Cepas J.  v2: functional annotation, orthology assignments, and domain prediction at the metagenomic scale. Molecular Biology and Evolution 38, 5825–5829 (2021).
  4. Mallick H, Rahnavard A, McIver LJ, et al. Multivariable association discovery in population-scale meta-omics studies. PLoS Computational Biology 17, e1009442 (2021).
  5. Nickols WA, Kuntz T, Shen J, et al. MaAsLin 3: refining and extending generalized multivariable linear models for meta-omic association discovery. bioRxiv (2024).
  6. Tian L, Wang X-W, Wu A-K, et al. Deciphering functional redundancy in the human microbiome. Nature Communications 11, 6217 (2020).
  7. Almeida-Neto M, Guimarães P, Guimarães PR Jr, Loyola RD, Ulrich W. A consistent metric for nestedness analysis in ecological systems: reconciling concept and measurement. Oikos 117, 1227–1239 (2008).
  8. Wang Z. Introduction to Computational Metagenomics. World Scientific, 2022. Read: Ch. 7 Functional Metagenomics: gene and pathway annotation, and ch. 8 Deconvolute Community Metagenome into genomes.
  9. Charles TC, Liles MR, Sessitsch A (eds.). Functional Metagenomics: Tools and Applications. Springer, 2017.
  10. Mitra S (ed.). Metagenomic Data Analysis. Methods in Molecular Biology 2649. Humana Press, 2023. Read: Ch. 12 Recovery and Analysis of Long-Read Metagenome-Assembled Genomes and ch. 15 Tracking Antibiotic Resistance.
  11. Streit WR, Daniel R (eds.). Metagenomics: Methods and Protocols, 2nd edition. Humana Press, 2017.
Viral metagenomics

Viruses and viromes: what makes them different

Most of the methods in this handbook were built for bacteria. Viruses break several of their assumptions: there is no gene every virus shares, most sequences have no relative in any database, and a sample can be full of viral reads from sources that are not in it. This chapter explains why, so that the next two chapters make sense.

Why look at viruses

Viruses are the most numerous biological entities on Earth, and the viruses of bacteria, the bacteriophages or phages, shape bacterial communities by killing cells, moving genes between them and changing their metabolism (Suttle, 2007). In the human gut most of the virome is phage, and most of it has never been cultured (Shkoporov & Hill, 2019). That is why viromes are studied from sequence: there is nothing else to study them from. For a company, phages matter in three ways: as a risk (a phage can carry a toxin or a resistance gene into a production strain), as a tool (phage therapy, biocontrol of pathogens in food) and as a marker of a community's state.

What a virus genome can be

A bacterial genome is double-stranded DNA with a few well-known features. A virus genome can be DNA or RNA, single- or double-stranded, linear or circular, in one piece or several. Baltimore's classification sorts viruses by how their genome becomes messenger RNA, and the practical consequences for a sequencing project are direct:

  • RNA viruses are missed by a DNA workflow. They need RNA extraction and reverse transcription, and they are a different study from a DNA virome. This module covers DNA viruses and the abundance of whatever genomes you provide.
  • Single-stranded DNA viruses (the Microviridae are an example, and phiX174 is the best known) are invisible to library methods that only act on double-stranded DNA, and they are over-represented when the DNA is amplified by multiple displacement amplification, which favours small circular single-stranded templates (Kim & Bae, 2011). A virome made with amplification is not quantitative.
  • Circular and terminally redundant genomes assemble into one contig with a repeat at both ends, which is the first sign that a genome is complete.

Lytic and temperate phages

A virulent (lytic) phage infects a cell, makes new virions and bursts the cell. A temperate phage can do the same, or it can integrate into the host chromosome as a prophage and be copied with the host for generations, until a signal makes it excise and turn lytic. Lysogeny has two consequences for anyone using or monitoring phages. A prophage can carry genes that change the host, such as the Shiga toxin gene carried by phage 933W, which makes its Escherichia coli host a Shiga-toxin-producing strain. And a temperate phage can move genes between hosts, which is the reason regulators of phage therapy look for strictly lytic phages (Philipson et al., 2018). The genes that give a temperate phage away, an integrase or excisionase, are among the first things a safety screen looks for (chapter V3).

How viruses are named

Virus taxonomy is set by the International Committee on Taxonomy of Viruses (ICTV) and it changed fundamentally in 2022. For decades the tailed phages were sorted by the shape of the tail into the families Myoviridae, Siphoviridae and Podoviridae, in the order Caudovirales. That sorting was found not to reflect descent, and the ICTV abolished those names and the order, placing the tailed phages in the class Caudoviricetes with new families based on genome relationships (Turner et al., 2023). A lineage now runs from realm (for tailed phages, Duplodnaviria) through kingdom, phylum, class, order and family to genus and species, and a great many viruses can be placed only part of the way down. Old literature uses names that no longer exist: when you read "Siphoviridae", read "a tailed phage".

Why a virome is hard to analyse

  • There is no universal marker gene. Bacteria and archaea share the 16S rRNA gene, so one amplicon can profile a community. Viruses share nothing: even within the tailed phages, no gene is present in all of them. A virome is read from shotgun sequence, and viruses are recognised by a collection of weak signals (hallmark genes, composition, gene density).
  • Most of what is found has no relative. The share of viral genes with no match in any database, the viral dark matter, is large, so annotation by homology says little about most of a virome.
  • Genomes are mosaics. Phage genomes are built from modules exchanged between lineages, so a single phylogenetic tree does not describe them and a gene's function does not predict the rest of the genome.
  • A species is a decision. There is no biological species concept for viruses that can be applied to sequence alone. In practice genomes are grouped into vOTUs, viral operational taxonomic units: clusters at about 95 % average nucleotide identity over 85 % of the shorter genome, the convention set by the MIUViG standard (Roux et al., 2019).
  • The sample is not the sequence. Viral sequence in a sample can come from the reagents used to prepare it. Contamination matters more for viromes than for bacterial communities, because the starting material is so little (Salter et al., 2014).
  • The data are compositional. Sequencing gives shares of reads, not particle numbers, so if one virus doubles, every other virus appears to fall (Gloor et al., 2017).

Two kinds of virome data

A virus-like-particle (VLP) virome is prepared to enrich virions: the sample is filtered, treated with nucleases to destroy free DNA, and the DNA that remains inside protein coats is extracted (Thurber et al., 2009; Conceição-Neto et al., 2015). A bulk metagenome is not enriched, so it contains viral sequence only as a small share of the reads, together with the prophages integrated in the bacterial genomes. VLP viromes show free virions, bulk metagenomes show the prophages that the bacteria carry, and the two do not give the same picture of the same sample.  estimates how well a virome was enriched, by counting the reads that come from bacterial ribosomal RNA and single-copy genes, which a clean VLP preparation should lack (Zolfo et al., 2019).

The human gut virome in brief

The gut virome is dominated by tailed phages that infect the gut bacteria, and it is individual: each person carries a largely different set that changes slowly over time (Shkoporov & Hill, 2019). One virus stands out. crAssphage was found in 2014 by assembling the sequences in human faecal metagenomes that matched nothing known, and it is among the most abundant viruses of the human gut (Dutilh et al., 2014). Large catalogues now collect the genomes recovered from gut metagenomes, the Gut Virome Database and the Gut Phage Database among them, and they are the reference sets against which a new gut virome is compared (Gregory et al., 2020; Camarillo-Guerrero et al., 2021).

Brock, Biology of Microorganisms, ch. 8 on virus replication and ch. 10 on viral genomics, diversity and ecology, is the best general introduction to the biology (Madigan et al.). For the human virome in a metagenomics setting, read Izard and Rivera, ch. 9 (Metagenomics for Microbiology). The bioinformatics of viruses, from comparative genomics to host prediction, is the subject of Frishman and Marz (eds.), ch. 1, 2 and 10 (Virus Bioinformatics).

Check your understanding

Why can a 16S amplicon profile a bacterial community but not a virome?

16S rRNA is present in every bacterium and archaeon. No gene is shared by all viruses, so there is no amplicon to target, and viruses must be found in shotgun data.

What is the consequence of a phage being temperate for someone developing a phage product?

It can integrate into a host genome and move genes between hosts, including toxin or resistance genes. Products usually use strictly lytic phages.

A paper reports "Siphoviridae" in a gut virome. What does that mean today?

A tailed phage. The family was abolished in 2022 because tail shape does not reflect descent; the tailed phages are now in the class Caudoviricetes.

In a virome, 60 % of the reads belong to vOTUs with no family. Is a bar chart of the families a picture of the virome?

It shows the 40 % that can be named. The unclassified share is the largest part and must be shown with it.

Key references

  1. Camarillo-Guerrero LF, Almeida A, Rangel-Pineros G, Finn RD, Lawley TD. Massive expansion of human gut bacteriophage diversity. Cell 184, 1098–1109 (2021).
  2. Conceição-Neto N, Zeller M, Lefrère H, et al. Modular approach to customise sample preparation procedures for viral metagenomics: a reproducible protocol for virome analysis. Scientific Reports 5, 16532 (2015).
  3. Dutilh BE, Cassman N, McNair K, et al. A highly abundant bacteriophage discovered in the unknown sequences of human faecal metagenomes. Nature Communications 5, 4498 (2014).
  4. Gloor GB, Macklaim JM, Pawlowsky-Glahn V, Egozcue JJ. Microbiome datasets are compositional: and this is not optional. Frontiers in Microbiology 8, 2224 (2017).
  5. Gregory AC, Zablocki O, Zayed AA, et al. The Gut Virome Database reveals age-dependent patterns of virome diversity in the human gut. Cell Host & Microbe 28, 724–740 (2020).
  6. Kim K-H, Bae J-W. Amplification methods bias metagenomic libraries of uncultured single-stranded and double-stranded DNA viruses. Applied and Environmental Microbiology 77, 7663–7668 (2011).
  7. Philipson CW, Voegtly LJ, Lueder MR, et al. Characterizing phage genomes for therapeutic applications. Viruses 10, 188 (2018).
  8. Roux S, Adriaenssens EM, Dutilh BE, et al. Minimum Information about an Uncultivated Virus Genome (MIUViG). Nature Biotechnology 37, 29–37 (2019).
  9. Salter SJ, Cox MJ, Turek EM, et al. Reagent and laboratory contamination can critically impact sequence-based microbiome analyses. BMC Biology 12, 87 (2014).
  10. Shkoporov AN, Hill C. Bacteriophages of the human gut: the “known unknown” of the microbiome. Cell Host & Microbe 25, 195–209 (2019).
  11. Suttle CA. Marine viruses: major players in the global ecosystem. Nature Reviews Microbiology 5, 801–812 (2007).
  12. Thurber RV, Haynes M, Breitbart M, Wegley L, Rohwer F. Laboratory procedures to generate viral metagenomes. Nature Protocols 4, 470–483 (2009).
  13. Turner D, Shkoporov AN, Lood C, et al. Abolishment of morphology-based taxa and change to binomial species names: 2022 taxonomy update of the ICTV bacterial viruses subcommittee. Archives of Virology 168, 74 (2023).
  14. Zolfo M, Pinto F, Asnicar F, et al. Detecting contamination in viromes using . Nature Biotechnology 37, 1408–1412 (2019).
  15. Madigan MT, Bender KS, Buckley DH, Sattley WM, Stahl DA. Brock Biology of Microorganisms, 15th edition. Pearson, 2018. Read: Ch. 8, viruses and their replication; ch. 10, viral genomics, diversity and ecology.
  16. Izard J, Rivera MC (eds.). Metagenomics for Microbiology. Academic Press, 2015. Read: Ch. 9, toward the understanding of the human virome.
  17. Frishman D, Marz M (eds.). Virus Bioinformatics. CRC Press, 2022. Read: Ch. 1, comparative genomics of viruses; ch. 2, metagenomic exploration of phage diversity; ch. 10, host prediction with machine learning.
Viral metagenomics

From reads to vOTUs: finding, judging and counting viruses

A virome table is the end of a pipeline with five decisions in it: which sequences are viral, how good each genome is, which genomes are one virus, what each one is and infects, and how much of each is in each sample. Each decision leaves a trace in the table, and a reader who knows the pipeline can see what the numbers mean.

Assemble, then find the viruses

A virome starts as reads from many viruses and, usually, from many bacteria as well. The reads are assembled into contigs with a metagenome assembler such as  or , or with , which is built for the repeats and uneven coverage of viral genomes (Antipov et al., 2020). Then each contig has to be classified, because an assembly of a virome still contains bacterial chromosome, plasmids and other mobile elements.

Tools identify viruses by combining several weak signals.  classifies a sequence as chromosome, plasmid or virus with a neural network on its sequence, supported by a set of marker genes, assigns a taxonomy that follows the ICTV, and finds proviruses inside larger sequences (Camargo et al., 2024).  uses several classifiers and expert-curated rules (Guo et al., 2021),  uses annotation of protein families (Kieft et al., 2020), and  uses a neural network on the sequence alone, without homology (Ren et al., 2020). They disagree on the borderline, so a study should state which was used and at what score. Every tool trades missed viruses against false calls, and a high score cut-off removes the doubtful sequences and the real but unusual ones together.

Judging a genome: completeness and contamination

A contig that is called viral may be a complete genome, a fragment of one, or a chimera of a virus and its host. estimates completeness by comparing each contig with a database of complete viral genomes, using amino-acid identity to place it and the length of its closest relatives to judge how much is missing. It estimates contamination by counting host genes at the ends of the contig, and it trims host sequence from proviruses (Nayfach et al., 2021). Its output follows the MIUViG standard, which sorts genomes into tiers (Roux et al., 2019):

TierEstimated completenessUse it for
Complete100 %, with evidence such as terminal repeatsgenome-level conclusions
High-quality90 % or moreabundance and most gene-content questions
Medium-quality50 to 90 %abundance with care; gene absence cannot be asserted
Low-qualitybelow 50 %presence of a fragment only
Not-determinedno close relative to judge againstoften novel viruses; treat as unknown, not as bad

Which genomes are one virus: vOTUs

The same virus is assembled again and again, from different samples and with different ends, and counting each copy would inflate the virome. Genomes are therefore clustered into vOTUs. The convention is at least 95 % average nucleotide identity over at least 85 % of the shorter genome (Roux et al., 2019), which for tailed phages is close to species level.  includes scripts that calculate the identity between genomes and cluster them on these thresholds. One representative, usually the longest or most complete, stands for each vOTU, and all later steps use only the representatives. A vOTU table is therefore a table of decisions: change the thresholds and the number of vOTUs changes.

What each virus is, what it infects, how it lives

Lineage. 's taxonomy places a sequence by its marker genes in the ICTV system, and gene-sharing networks, as in , cluster viruses by the proteins they share (Bin Jang et al., 2019). Both stop at family or genus for most sequences, as chapter V1 showed.

Host. A phage's host is not in its sequence. It is inferred from what it leaves behind in bacterial genomes: CRISPR spacers that match the phage, integrated prophages, shared tRNAs and genes, and similarity of k-mer composition to the host.  combines these methods and gives each prediction a confidence score (Roux et al., 2023). A predicted host is a hypothesis, reported at genus level with a confidence score, and many viruses get no prediction at all, especially those of poorly known hosts. A phage may also infect more than one host.

Lifestyle.  predicts whether a phage is virulent or temperate from the conserved protein domains in its genome (Hockenberry & Wilke, 2021). A simpler screen looks for an integrase or excisionase. The absence of an integrase does not make a phage virulent: an incomplete genome may lack the gene.

How much of each is in each sample

Abundance is measured by mapping the reads of each sample back to the vOTU genomes, with a mapper such as Bowtie 2 (Langmead & Salzberg, 2012) or  (Li, 2018). Two numbers per vOTU and sample matter: the reads that map, and the breadth, the share of the genome that those reads cover. The reads alone are misleading, for two reasons.

  • Reads follow length. A virus produces reads in proportion to its genome length and its copy number, so a long genome has more reads than a short one at the same number of copies.
  • Reads can come from a relative. A related virus, or a gene that two viruses share, produces reads that map to part of a genome. Reads over a small fraction of a genome are not evidence that the virus is there.

A widely used rule therefore counts a virus as present only if reads cover at least 75 % of its genome, and works with genomes of at least 10 kb for abundance (Roux et al., 2017). It is a convention, not a law: it removes false presences and some true low-abundance viruses, and the threshold should be stated with every result. Reads should also be filtered for mapping quality, so that a read which maps equally well to two related viruses is not credited to either.

Mitra (ed.), ch. 17, describes the assembly and annotation of viral metagenomes from short reads step by step (Metagenomic Data Analysis), and Frishman and Marz (eds.), ch. 2, reviews the techniques for exploring phage diversity in metagenomes (Virus Bioinformatics). The wet-laboratory side, from isolating phages to preparing them for sequencing, is in Pantaleo and Chiumenti (eds.), ch. 1 (Viral Metagenomics).

Check your understanding

A vOTU has 400 reads in a sample, all mapping to one 8 kb stretch of a 60 kb genome. Is the virus present?

Not by the usual rule: reads cover about 13 % of the genome, far below 75 %. The reads probably come from a relative or a shared gene.

Two phages have the same number of genome copies. One genome is 5 kb and the other 50 kb. What do their read shares look like?

About 9 % and 91 %. Reads are proportional to length times copy number, so abundances must be divided by length before comparison.

 calls a genome "Not-determined". Is it a poor genome?

No. It means the database held no close relative to judge completeness against. Many novel viruses fall in this tier.

Why does the choice of clustering threshold matter for a study?

A vOTU is defined by the threshold (95 % identity over 85 % of the shorter genome), so the number of vOTUs and the diversity computed from them change if the threshold changes.

Key references

  1. Antipov D, Raiko M, Lapidus A, Pevzner PA. Metaviral : assembly of viruses from metagenomic data. Bioinformatics 36, 4126–4129 (2020).
  2. Bin Jang H, Bolduc B, Zablocki O, et al. Taxonomic assignment of uncultivated prokaryotic virus genomes is enabled by gene-sharing networks. Nature Biotechnology 37, 632–639 (2019).
  3. Camargo AP, Roux S, Schulz F, et al. Identification of mobile genetic elements with . Nature Biotechnology 42, 1303–1312 (2024).
  4. Guo J, Bolduc B, Zayed AA, et al. : a multi-classifier, expert-guided approach to detect diverse DNA and RNA viruses. Microbiome 9, 37 (2021).
  5. Hockenberry AJ, Wilke CO. : predicting bacteriophage lifestyle from conserved protein domains. PeerJ 9, e11396 (2021).
  6. Kieft K, Zhou Z, Anantharaman K. : automated recovery, annotation and curation of microbial viruses, and evaluation of viral community function from genomic sequences. Microbiome 8, 90 (2020).
  7. Langmead B, Salzberg SL. Fast gapped-read alignment with Bowtie 2. Nature Methods 9, 357–359 (2012).
  8. Li H. : pairwise alignment for nucleotide sequences. Bioinformatics 34, 3094–3100 (2018).
  9. Nayfach S, Camargo AP, Schulz F, et al.  assesses the quality and completeness of metagenome-assembled viral genomes. Nature Biotechnology 39, 578–585 (2021).
  10. Ren J, Song K, Deng C, et al. Identifying viruses from metagenomic data using deep learning. Quantitative Biology 8, 64–77 (2020).
  11. Roux S, Emerson JB, Eloe-Fadrosh EA, Sullivan MB. Benchmarking viromics: an in silico evaluation of metagenome-enabled estimates of viral community composition and diversity. PeerJ 5, e3817 (2017).
  12. Roux S, Adriaenssens EM, Dutilh BE, et al. Minimum Information about an Uncultivated Virus Genome (MIUViG). Nature Biotechnology 37, 29–37 (2019).
  13. Roux S, Camargo AP, Coutinho FH, et al. : an integrated machine learning framework to maximize host prediction for metagenome-derived viruses of archaea and bacteria. PLoS Biology 21, e3002083 (2023).
  14. Mitra S (ed.). Metagenomic Data Analysis. Methods in Molecular Biology 2649. Humana Press, 2023. Read: Ch. 17, assembly and annotation of viral metagenomes from short-read sequencing data.
  15. Frishman D, Marz M (eds.). Virus Bioinformatics. CRC Press, 2022. Read: Ch. 2, current techniques and approaches for metagenomic exploration of phage diversity.
  16. Pantaleo V, Chiumenti M (eds.). Viral Metagenomics: Methods and Protocols. Methods in Molecular Biology. Humana Press. Read: Ch. 1, host-associated bacteriophage isolation and preparation for viral metagenomics.
Viral metagenomics

Analysing and reporting a virome: controls, statistics and safety

A virome result is only as good as the checks behind it. This chapter follows the order of the MicroRes viral module: decide which genomes and which presences to trust, look for contamination, compare viromes with statistics that respect composition, link viruses to hosts, screen phages for use in a product, and report all of it so that a reviewer can check it.

Start with the controls

The first question about a virome is whether the viruses in it came from the sample. Extraction kits and reagents carry traces of nucleic acid, and with the small amounts of DNA in a virome those traces can make up much of the library (Salter et al., 2014). The defence is a blank, a tube taken through the whole procedure without a sample, sequenced beside the samples. A virus that is more prevalent in the blanks than in the samples is a reagent contaminant. The prevalence method of the decontam package tests this with Fisher's exact test per virus (Davis et al., 2018), and the module applies it in V2.

Some contaminants are known in advance. phiX174 is added to Illumina runs as a sequencing control, usually at 1 % and at higher levels for libraries of low diversity, so it appears in a virome as a real virus genome that was never in the sample. It is a sequencing control, not a finding, and a study should name it and remove it. The module reports it in V2 when a genome named phiX174 is present. A second control is the enrichment itself: if a virome contains many reads from bacterial ribosomal RNA genes, the virus-like-particle preparation did not work, and  quantifies this (Zolfo et al., 2019).

Decide what counts as present

Chapter V2 explained why reads on a genome are not presence. The practical rule is a breadth threshold, set before the analysis and reported with it. Everything downstream is affected by it: the number of vOTUs seen in a sample (richness), the prevalence of each virus, and the core virome. A study that changes the threshold changes its conclusions, so the module records it with the result.

Compare viromes

Once the table is made, the statistical methods of Part IX apply, with the cautions of a compositional table (Gloor et al., 2017).

  • Alpha diversity of vOTUs depends on the depth of sequencing and on the presence rule, because a virus is seen only if it is read deeply enough. Compare samples at similar depth, and read richness as "vOTUs detected at this threshold".
  • Beta diversity between viromes is large and individual in the human gut, so ordinations often separate people before they separate groups. Use the study design (chapter 2) to put subject in the model when the same person is sampled more than once.
  • Differential abundance of single vOTUs has the problems of any sparse table, made worse because most vOTUs are present in few samples. The module runs the same methods as for taxa, and they need read counts, not percentages. A result on a vOTU present in three samples is a hypothesis.
  • Pooling by lineage, host or lifestyle is often more powerful than testing single vOTUs, because the groups are larger. The module lets any step pool vOTUs by family, host genus or lifestyle.

Link viruses to hosts

If bacterial abundances are measured on the same samples, a virus can be compared with its predicted host: does it rise and fall with it? The comparison needs care. Both tables are compositional, so they are put on a centred log-ratio scale before correlation, and the p values are corrected for the number of pairs tested. A positive link supports a host prediction. It does not prove infection, because a virus and a bacterium can both respond to the same condition, and a lytic phage and its host can show the opposite pattern when the phage is killing the host. A link is a reason to look, not a conclusion.

Screen phages for use in a product

A phage intended for therapy, for biocontrol in food or as a feed additive must be shown not to carry genes that would harm. Philipson and colleagues list what a genome characterisation should look for: that the phage is strictly lytic, that it has no integrase or other lysogeny genes, and that it carries no toxin, virulence or antibiotic-resistance gene (Philipson et al., 2018). The module's screen follows this and returns for each vOTU one of three verdicts: Concern (a strong hit to a toxin, virulence or resistance gene), Review (a weak hit, a temperate lifestyle, an incomplete genome or sequence from the host), and No flag found. Each verdict carries its reasons.

Three limits belong in every report. The screen finds only genes that the databases know. A genome that is 85 % complete can hide a gene in the missing 15 %, which is why the screen sends incomplete genomes to review. And a clean screen is not a clearance: the evidence that a regulator requires for a product is wider than a genome screen.

Report it so that it can be checked

A virome report has to answer the questions a reviewer asks before they read a result. What did the controls show? Which genomes were included, and which thresholds were set? How many samples failed? What was fixed before the analysis, and what was decided after it? The MIUViG standard defines the information that should accompany a viral genome (Roux et al., 2019), and the MIxS checklists define the metadata of the samples (Yilmaz et al., 2011).

The module's report opens with a quality gate: a short list of checks (was presence tested, are the blanks clean, did the samples pass, do the reads fall on good genomes, are there enough samples, are the genomes annotated, was the safety screen run), each passed, flagged for review or failed under a rule printed beside it. The overall verdict follows from the checks and is not chosen by the analyst. The rules are Bioinformaniac's own acceptance criteria and not a regulatory standard, and the report says so; a reviewer can apply a stricter rule to the same printed numbers. After the gate come the study, the data fingerprint (a hash of the analysed table, so that two copies of a report can be shown to describe the same data), each analysis with its methods and settings, the limitations and a glossary.

For the statistics, see chapters S1 to S9, especially S4 (counts and compositional data) and S5 (diversity and PERMANOVA). For the human virome as a clinical and ecological subject, read Izard and Rivera, ch. 9 (Metagenomics for Microbiology), and for the computational side of host prediction, Frishman and Marz (eds.), ch. 10 (Virus Bioinformatics).

Check your understanding

A virus is found in both blanks and in 2 of 20 samples. What is it most likely to be?

A reagent contaminant: it is more prevalent in the blanks than in the samples, which is the signature the prevalence method tests for.

Why are two compositional tables log-ratio transformed before a virus is correlated with its host?

Relative abundances sum to a constant, so they are not independent and raw correlations are distorted. The centred log-ratio removes this, to a first approximation.

A phage genome passes the safety screen with "No flag found" but is only 70 % complete. What can you say?

Only that no flag was found in the part that was sequenced. A toxin or resistance gene could lie in the missing 30 %, so the screen sends such genomes to review.

What makes the verdict of a quality gate trustworthy?

It follows mechanically from stated rules applied to printed numbers, so a reviewer can check it, apply a stricter rule or see that a check failed. It does not depend on the analyst's judgement.

Part III · Bacterial genomics

Read quality control for isolate genomes

An isolate genome is only as good as the reads behind it. A few minutes spent on read statistics saves hours of puzzling over a broken assembly.

What to look at first

For every sample, check the number of reads, total bases, mean and N50 read length, and mean quality. Divide total bases by the expected genome size to get coverage. Short-read assemblers need roughly 50 to 100-fold coverage; long-read assemblers do well at 30 to 50-fold of good reads, and more is not always better because very deep data can be subsampled to the longest, most accurate reads.

  • Illumina: quality should stay high across most of the read. Adapter content shows short inserts. Duplication can indicate low input DNA.
  • Nanopore: the read-length distribution tells you whether the extraction preserved long DNA. A peak at a few hundred bases usually means sheared DNA or a failed barcode. Mean quality depends on chemistry and basecaller model.
  • PacBio: HiFi reads cluster tightly in length and are very accurate; CLR reads are longer and noisier and need different assembler settings.

The same checks as in chapter 3 apply, with one addition: an isolate should contain one genome. A k-mer profile with one coverage peak and a read classifier that assigns nearly all reads to one species confirm it. Ismail, Bioinformatics: A Practical Guide to NGS Data Analysis, ch. 1, walks through  and trimming, and Buffalo, Bioinformatics Data Skills, ch. 10, explains the FASTQ format and quality encodings at the level needed to write your own checks.

Pooling, filtering and trimming

Nanopore runs often write many small FASTQ files per barcode, which are concatenated into one file per sample. Unclassified reads and barcodes with very few reads should be inspected: a near-empty barcode is a failed sample, not a small genome. Long reads are filtered by minimum length (for example 1 kb) and mean quality, keeping the longest reads up to a target coverage. Short reads are adapter- and quality-trimmed with  or  (Chen et al., 2018; Bolger et al., 2014).

Contamination at the read level

A quick taxonomic screen of a subset of reads (for example with ) reveals mixed cultures or contamination before assembly. Two species in one culture produce an assembly roughly twice the expected size and high contamination scores later (chapter 17).

Check your understanding

You have 500× coverage of a 5 Mb bacterium. Should you use it all?

Usually not: beyond about 100× short reads add little and can slow assemblers or add error k-mers. Subsample to a target coverage.

A k-mer profile shows two coverage peaks of similar height. What might it be?

Two strains or species in the culture. Check with a read classifier before assembling.

Part III · Bacterial genomics

Genome assembly

Assembly reconstructs a genome from overlapping reads. Repeats are what make it difficult, and read length is what solves them.

Graphs

Short-read assemblers build a de Bruijn graph: reads are broken into k-mers, and k-mers that overlap by k−1 bases are linked. A unique region becomes a simple path; a repeat becomes a node where paths cross.  uses several k-mer sizes and paired-end information to resolve as much as possible (Bankevich et al., 2012). Long-read assemblers mostly use overlap or repeat graphs.  builds a repeat graph from approximate overlaps and resolves repeats using reads that span them (Kolmogorov et al., 2019). When a read is longer than every repeat, the graph collapses to one circle per replicon.

A worked de Bruijn graph, with the repeat that breaks it, is in chapter 31 (fungal assembly); the logic is the same for bacteria, whose main repeats are rRNA operons (about 5 kb, often 5–10 copies) and insertion sequences (1–2.5 kb). Reads longer than the longest repeat resolve the whole chromosome, which is why a single Nanopore or HiFi run now routinely gives complete bacterial genomes. Ismail, Bioinformatics: A Practical Guide to NGS Data Analysis, ch. 3, compares greedy, overlap–layout–consensus and de Bruijn assemblers; Low & Tammi, Practical Bioinformatics for Beginners, ch. 6, compares short-read, long-read and hybrid assemblies of one genome.

Choosing an assembler

DataCommon choiceTypical result
Illumina onlyTens to hundreds of contigs; repeats and rRNA operons collapsed
Nanopore only, then polishingComplete circular chromosome and plasmids at good coverage
PacBio HiFi (HiFi mode) or Complete genomes, very accurate without polishing
Short + long reads (hybrid)Complete genome with short-read base accuracy

 builds a short-read assembly graph and uses long reads to bridge its repeats, which gives complete genomes with high base accuracy when both data types are good (Wick et al., 2017). Benchmarks of long-read assemblers show that each has characteristic failure modes, such as missing small plasmids or duplicating the start and end of a circular contig, and that combining several assemblies into a consensus (Trycycler) gives the most reliable finished genomes (Wick & Holt, 2019; Wick et al., 2021).

Small plasmids

Small plasmids (a few kilobases) are often lost by long-read assemblers because size selection removes their short molecules, or because they are filtered out as low-length reads. If plasmids matter, check the short-read assembly or rerun with relaxed read-length filters.

Check your understanding

Why does a 7 kb rRNA operon present in 7 copies break a short-read assembly?

Reads shorter than the repeat cannot tell which copy they come from, so the graph collapses the copies and contigs end at their borders.

Why might a long-read assembler miss a 3 kb plasmid?

Small plasmids are under-represented in size-selected long-read libraries and may be filtered as short reads; short-read or hybrid assembly recovers them.

Part III · Bacterial genomics

Polishing and circularisation

A long-read assembly gets the structure right. Polishing fixes the remaining base-level errors, and circularisation gives the genome a clean, conventional start.

Why polish

Residual errors in long-read assemblies are mostly small insertions and deletions in homopolymers. Each indel in a coding region shifts the reading frame and splits a gene into fragments, which inflates gene counts and creates false pseudogenes. Polishing aligns reads back to the draft and corrects positions where the consensus disagrees with the draft.

  • Long-read polishing with  uses a neural network trained for a specific pore and basecaller model, so the model setting must match how the reads were basecalled. With R10.4.1 data and current basecallers, long-read-only genomes can reach very high accuracy (Sereika et al., 2022), and  does not improve them further (Soto-Serrano et al., 2024): use it for older flow cells (R9.4.1, R10.3), not for R10.4 or R10.4.1.
  • Short-read polishing aligns Illumina reads (for example with  (Li & Durbin, 2009)) and fixes remaining errors. It must be careful in repeats, where short reads map ambiguously and can introduce new errors.

Circularisation and rotation

A circular replicon assembled as a linear contig may have overlapping ends. Circularisation trims the overlap, and rotation sets the start position, by convention at the dnaA gene for chromosomes and at repA for plasmids where possible. A consistent start makes genomes easier to compare and annotate.

Check your understanding

Why do frameshifts in a draft assembly inflate the number of predicted genes?

An indel inside a gene shifts the reading frame, creating a premature stop; the gene is predicted as two or more fragments.

Why rotate a chromosome to start at dnaA?

It gives a standard coordinate system, so genomes can be compared and annotations do not split a gene across the sequence ends.

Part III · Bacterial genomics

Assembly quality:  and 

Two different questions decide whether an assembly is fit for use: is it contiguous, and is it complete and clean?

Contiguity with 

 reports the number of contigs, total length, largest contig, N50, L50 and GC content (Gurevich et al., 2013). N50 is the length such that contigs of that size or longer contain half the assembly. A finished bacterial genome has one contig per replicon and an N50 close to the chromosome size; short-read assemblies typically have N50 values of tens to hundreds of kilobases. Total length and GC content should match what is known for the species. With a reference genome,  also counts misassemblies and mismatches.

Completeness and contamination with 

 uses sets of marker genes that are present in a single copy in nearly all genomes of a lineage (Parks et al., 2015). Missing markers lower completeness; duplicated markers raise contamination. Lineage-specific marker sets, chosen by placing the genome in a reference tree, are more precise than universal sets.  replaces fixed marker sets with a machine-learning model and works better for novel or reduced genomes (Chklovski et al., 2023).

Category (MIMAG)CompletenessContamination
High quality> 90 %< 5 %, plus rRNA genes and ≥ 18 tRNAs
Medium quality≥ 50 %< 10 %
Low quality< 50 %< 10 %

These thresholds were defined for genomes from metagenomes (Bowers et al., 2017); an isolate genome should comfortably exceed the high-quality bar. High contamination in an isolate usually means a mixed culture.

Check your understanding

Contigs of 100, 80, 60, 40 and 20 kb. What are the N50 and L50?

Total 300 kb; half is 150 kb, reached at the second contig. N50 = 80 kb, L50 = 2.

 reports 99 % complete and 8 % contamination. Is the genome usable?

Completeness is excellent, but 8 % contamination exceeds the usual 5 % limit for high quality: check for a second organism before publishing.

Part III · Bacterial genomics

Genome taxonomy: ANI and 

With a whole genome in hand, species identity is decided by genome-wide similarity, not by a single marker gene.

Average nucleotide identity

Average nucleotide identity (ANI) compares two genomes over their shared regions. Across many thousands of genome pairs, ANI values show a clear gap around 95 percent, which matches traditional species boundaries well (Jain et al., 2018). Two genomes above about 95 percent ANI are generally the same species. Fast estimators such as  use k-mer sketches to screen large collections before an exact calculation (Ondov et al., 2016).

Christensen (ed.), Introduction to Bioinformatics in Microbiology, ch. 7, explains the move from 16S identity and DNA–DNA hybridisation to genome-based species definitions; ch. 6 covers the phylogenetic methods behind .

The Genome Taxonomy Database

 is a genome-based taxonomy built from a phylogeny of conserved single-copy proteins and normalised so that ranks correspond to comparable evolutionary depth (Parks et al., 2022). It reorganised many groups whose traditional names did not reflect their phylogeny, so  names sometimes differ from those in older literature and from .  places a genome in the  reference tree and assigns species by ANI to the closest reference (Chaumeil et al., 2022). A genome with no species-level match is reported to genus, which may indicate a new species.

Check your understanding

Two genomes share 97 % ANI. Same species?

Very likely yes: the boundary is 95–96 %. Check that the aligned fraction is high enough.

Why does  sometimes rename well-known species?

It defines taxa from a genome phylogeny with consistent rank depths, splitting groups that were lumped together by phenotype.

Part III · Bacterial genomics

Genome annotation

Annotation finds the genes and other features in a genome and gives them names. The quality of every downstream biological claim depends on it.

Structural and functional annotation

Structural annotation locates features: protein-coding sequences (CDS), predicted by  in most bacterial pipelines, plus rRNAs, tRNAs, tmRNA, non-coding RNAs, CRISPR arrays and origins of replication. Functional annotation assigns a product name and identifiers to each CDS by comparison with protein databases.  made this fast and standard for a decade (Seemann, 2014).  uses a large precomputed database of  protein clusters, identifies known proteins by exact hashing, and returns database cross-references for a high share of genes (Schwengers et al., 2021).

Bacterial gene finders such as  score open reading frames with the codon and start-site statistics of the genome itself, a simple form of the hidden Markov models that eukaryotic gene finders need (chapter 33). Baxevanis, Bader & Wishart, Bioinformatics, ch. 5, and Zvelebil & Baum, Understanding Bioinformatics, ch. 9–10, explain both, including how accuracy is measured. Christensen (ed.), Introduction to Bioinformatics in Microbiology, ch. 2–3, describe the databases used for functional annotation and how far a function can be transferred from a distant homolog.

Reading the result

A typical bacterial genome has about one gene per kilobase. Counts far above that often mean frameshift errors fragmenting genes (chapter 16); counts far below suggest a fragmented assembly. Missing rRNA operons in a short-read assembly are usually an assembly artefact, because identical operon copies collapse. A large fraction of "hypothetical protein" annotations is normal, especially for less studied genera.

Check your understanding

Why is prokaryotic gene finding easier than eukaryotic?

Bacterial genes are mostly continuous open reading frames without introns and the genome is densely coding, so long ORFs with good codon statistics are usually genes.

A third of the genes are 'hypothetical protein'. Is the annotation poor?

Not necessarily: it is typical for bacteria. It means no characterised homolog was found, not that the gene is fake.

Part III · Bacterial genomics

Typing, plasmids and resistance

Once a genome is annotated, the practical questions are usually about lineage, mobile elements and genes of concern.

Multilocus sequence typing

 assigns an allele number to each of about seven housekeeping genes; the combination defines a sequence type (ST). STs are stable, comparable across laboratories and widely used in epidemiology. Schemes and allele definitions are curated in  (Jolley, Bray & Maiden, 2018). An "untyped" result usually means no scheme exists for the species or a new allele is present.

 assigns an allele number to each of seven housekeeping gene sequences and a sequence type (ST) to each combination of alleles: two isolates share an ST only if all seven alleles are identical. Core-genome  does the same with hundreds to thousands of genes and resolves outbreaks. Christensen (ed.), Introduction to Bioinformatics in Microbiology, ch. 11, traces typing from  to whole-genome methods.

Plasmids and viruses

Plasmids and prophages carry many of the genes that make strains different, including resistance and virulence genes. In a complete assembly, small circular contigs are plasmid candidates.  classifies sequences as chromosome, plasmid or virus using marker genes and a neural network, and annotates them (Camargo et al., 2024). It works on whole genomes and on metagenome assemblies.

Antimicrobial resistance genes

Resistance genes are found by comparison with curated databases such as 's  database or  (Feldgarden et al., 2021). A gene's presence predicts potential resistance but not the phenotype: expression, point mutations and the genetic context matter. Report the database version and identity thresholds, and confirm important findings with phenotypic testing.

Check your understanding

Two isolates share an  sequence type. Are they from one outbreak?

Not necessarily:  uses 7 genes and is coarse. Use cgMLST or SNP distances to judge recent transmission.

A resistance gene is detected. Will the isolate be resistant?

Usually, but not always: the gene may be truncated, not expressed, or need a partner. Phenotypic testing remains the reference.

Part III · Bacterial genomics

Variant calling and comparative genomics

Comparing a strain with its parent or with close relatives reveals the mutations behind an evolved trait, an outbreak link or a lab adaptation.

Small variants

Reads from the strain of interest are aligned to a reference genome, ideally a complete genome of the parent strain, with  for short reads or  for long reads (Li & Durbin, 2009; Li, 2018). A variant caller then reports single-nucleotide polymorphisms (SNPs) and short indels with their read support.  is a widely used haplotype-based caller (Garrison & Marth, 2012). For haploid bacteria, true variants should be supported by nearly all reads at a position; variants supported by a fraction of reads point to mixed populations, mapping errors in repeats or contamination.

Variant calling is two problems: mapping each read to the right place, then deciding whether a difference is real. Ismail, Bioinformatics: A Practical Guide to NGS Data Analysis, ch. 2 and 4, cover both, including base and mapping qualities, duplicate marking and the statistics of genotype calling; Buffalo, Bioinformatics Data Skills, ch. 11, explains the SAM/BAM format you will meet at every step.

Structural variants

Insertions, deletions, inversions and duplications larger than about 50 bases are detected from split and discordant alignments. Long reads make this much more reliable because a single read can span the whole event (Sedlazeck et al., 2018). Comparing two complete assemblies directly is an alternative that avoids reference bias.

Gene content

Across several genomes, gene presence and absence (the pangenome) shows which genes are shared and which are strain-specific. A heatmap of gene families across genomes, clustered by similarity, quickly shows groups of strains with similar accessory content.

Check your understanding

Why can a SNP call in a repeated gene be wrong?

Reads from different copies map to the same place; differences between copies look like variants. Mask repeats or require unique mapping.

What are the core and accessory genomes?

The core is genes present in (nearly) all strains; the accessory genome is genes present in some, often on mobile elements.

Part IV · Practice

Reproducibility and workflows

An analysis is reproducible when someone else, or you in a year, can rerun it and get the same result. That depends less on good intentions than on how the work is set up.

Pin software and databases

Results change with tool versions and, even more, with reference database versions. Record both for every step. Package managers such as  with Bioconda channels make exact versions installable (Grüning et al., 2018); containers (, ) freeze the whole environment. A one-line note such as "SILVA 138.1, DADA2 1.30, MetaPhlAn 4.1 with database vJun23" is worth more than a paragraph of prose.

Use a workflow manager

Workflow managers describe each step, its inputs and outputs, and its software environment, then run the steps in the right order, in parallel where possible, and resume after a failure without redoing finished work.  is widely used in genomics, including by the nf-core community, and runs the same workflow on a laptop, a cluster or the cloud (Di Tommaso et al., 2017).  follows a similar idea with Python-like rules.

Buffalo, Bioinformatics Data Skills is the best single book on this side of the work: project layout and documentation (ch. 2), version control with Git (ch. 5), Unix data tools (ch. 7) and writing robust pipelines (ch. 12). Its rule of thumb applies everywhere: anything you did by hand you will need to redo, so script it.

Keep data, code and results apart

Raw data should be read-only and stored once. Code lives in version control. Results are generated and can always be deleted and remade. Random seeds should be fixed for anything stochastic: rarefaction, permutation tests, cross-validation splits and random forests.

Share data properly

Deposit raw reads in an archive of the International Nucleotide Sequence Database Collaboration (ENA, SRA or DDBJ), with sample metadata that follows the MIxS checklists. Genomes go to the same archives with their annotation. The FAIR principles (findable, accessible, interoperable and reusable) summarise what reviewers and funders now expect (Wilkinson et al., 2016).

Check your understanding

Your result changes after reinstalling a tool. What should have prevented it?

Recording exact versions (a  environment file or container) for the tool and its database.

Why keep raw data read-only?

Every result can then be regenerated from the same starting point; accidental edits to raw data cannot be undone.

Part IV · Practice

Reporting a microbiome or genome study

A good methods section lets a reader judge every choice that could have changed the result. A good results section separates what the data show from what they suggest.

Methods checklist for microbiome studies

  • Sample collection, storage and time to freezing; DNA extraction kit and any bead-beating.
  • Marker region and primers, or shotgun library type; sequencing platform and read length.
  • Negative and positive controls and how they were used.
  • Read processing and feature construction (tool and version), reference database and version.
  • Filtering thresholds and how many features and reads each removed.
  • Normalisation and transformation for each analysis; how depth was handled for alpha diversity.
  • Distances, ordination and tests (with permutations and strata); differential abundance methods, covariates, random effects and FDR control.
  • For prediction: model, cross-validation scheme (grouped, nested), metrics and validation cohort.

Methods checklist for genome studies

  • Culture and DNA extraction; sequencing platform, chemistry and basecaller model.
  • Read filtering; assembler, version and key settings; polishing tools and rounds.
  • Assembly statistics (contigs, N50, size, GC) and completeness and contamination.
  • Taxonomic method and database release; annotation tool and database version.
  • Accession numbers for reads and assemblies.

Writing results

Give effect sizes with uncertainty: PERMANOVA R², fold changes with confidence intervals, AUC with its spread. State how many methods agreed on a differentially abundant taxon. Distinguish association from causation, and genetic potential from activity. Negative results reported with their power are useful to the field.

Report what was estimated, not only what was significant: effect sizes with intervals, the number of tests and the false discovery rate. When thousands of features are tested, a q-value of 0.05 means that about 5 % of the reported list is expected to be false; Efron & Hastie explain this and the related local false discovery rate in plain terms (Efron & Hastie, Computer Age Statistical Inference, ch. 15).

Check your understanding

Why report the database version used for taxonomy?

Names and assignments change between releases; without the version the result cannot be reproduced or compared.

Is 'taxon X was significantly increased (q < 0.05)' a complete result?

No: add the effect size (for example log fold change with a confidence interval), the method and the number of tests.

Part V · Transcriptomics

What RNA-seq measures, and how to design the study

RNA-seq turns the messenger RNA in a sample into millions of short sequences and counts them. Every later step, from normalisation to pathways, rests on two things decided before the first read exists: what RNA you captured, and how the samples were arranged.

From genes to reads

A gene is transcribed into RNA; only a small part of a cell's RNA is protein-coding messenger RNA (mRNA, about 2–4 %), while most is ribosomal RNA. In eukaryotes the primary transcript contains exons and introns; introns are spliced out (most begin with GT and end with AG), and one gene can produce several isoforms by alternative splicing. Bacterial genes have no introns and are often organised in operons, several genes transcribed together from one promoter. These differences shape everything that follows: eukaryotic reads must be aligned across splice junctions, and bacterial libraries cannot be built by catching poly(A) tails (Ismail, Bioinformatics: A Practical Guide to Next Generation Sequencing Data Analysis, ch. 5.1).

The library is made by enriching mRNA, reverse-transcribing it into complementary DNA (cDNA), fragmenting it, adding adapters and sequencing. Each read is a random fragment of a random transcript. A gene's count is therefore proportional to how many copies of its transcript were present multiplied by its length: a long transcript breaks into more fragments than a short one at the same expression.

Capturing the right RNA

  • Poly(A) selection uses oligo-dT beads to keep mature eukaryotic mRNA. It is efficient and clean, but it misses non-polyadenylated RNA and performs poorly with partly degraded RNA.
  • rRNA depletion removes ribosomal RNA instead. It is required for bacteria, whose mRNA has no poly(A) tail, and useful for degraded samples and for non-coding RNA (Christensen, Introduction to Bioinformatics in Microbiology, ch. 10).
  • Stranded libraries (for example the dUTP method) record which DNA strand a transcript came from. This matters wherever genes overlap on opposite strands, which is common in compact bacterial and fungal genomes.
  • RNA quality is summarised by the RNA integrity number (RIN). In good total RNA the large ribosomal band is about twice as intense as the small one; a smear below the bands signals degradation.

Replicates, depth and power

A biological replicate is an independent organism, culture or patient; a technical replicate is the same RNA prepared or sequenced twice. Differential expression asks whether a difference between groups is larger than the difference between independent individuals, so only biological replicates answer it. Three per group is a bare minimum; six or more is safer for modest effects, and beyond about 20 million reads per sample, more replicates add far more power than more depth (Christensen, ch. 10; Schurch et al. 2016).

Baxevanis and colleagues recommend a small pilot experiment: it gives a realistic estimate of variability, which makes a proper power calculation possible before the full study is funded (Baxevanis, Bader & Wishart, Bioinformatics, ch. 10).

Batches and confounding

A batch effect is any systematic difference introduced by handling: extraction day, kit lot, technician, sequencing run or lane. Batch effects are harmless when every batch contains every group, because the model can estimate and remove them. They are fatal when a batch coincides with a group (all controls on Monday, all treated on Tuesday): then no method can separate treatment from Monday (Baxevanis et al., Bioinformatics, ch. 10).

Transcriptome
All RNA molecules present in a sample at the time of collection.
Isoform
One of several transcripts produced from a gene by alternative splicing or start sites.
Library size
The number of reads assigned to genes in a sample; also called sequencing depth.
Biological replicate
An independent unit (organism, culture, patient) measured once.
Batch effect
A systematic technical difference between groups of samples processed together.
Stranded library
A library that keeps the information about which DNA strand was transcribed.

Check your understanding

Two genes have the same number of transcript copies, one 1 kb and one 5 kb long. Which has more reads?

The 5-kb gene, about five times as many, because it breaks into five times as many fragments. Comparing genes within a sample therefore needs length correction (TPM).

You can afford 120 million reads. Is it better to sequence 3 + 3 samples at 20 M or 6 + 6 at 10 M?

Usually 6 + 6. Power to detect differences comes mainly from the number of independent biological replicates; 10 million reads per sample already quantify most expressed genes.

Why can rRNA depletion, but not poly(A) selection, be used for E. coli?

Bacterial mRNA lacks a poly(A) tail, so oligo-dT beads cannot catch it; instead, the abundant ribosomal RNA is removed.

Part V · Transcriptomics

From reads to counts

The goal of the first half of an RNA-seq analysis is one table: genes in rows, samples in columns, a number in each cell. There are three ways to get there, and choosing between them is a question of what you need to know.

Quality control and trimming

Raw reads are first inspected with  (one report per file) and summarised across samples with . The usual problems are adapter sequence at read ends (the fragment was shorter than the read), falling base quality towards the 3′ end, over-represented sequences (often ribosomal RNA) and very high duplication. Tools such as ,  or  remove adapters and low-quality tails;  is several times faster on large datasets (Low & Tammi, Practical Bioinformatics for Beginners, ch. 8). Duplication is normal in RNA-seq: a highly expressed gene produces many identical fragments, so duplicates are not removed as they are in genome sequencing.

Three routes to a count table

1. Splice-aware alignment to a genome

 and  align reads to the genome and can split a read across an intron.  first finds the longest exact matches of each read ("seeds"), then stitches seeds that lie near each other into a spliced alignment (Ismail, ch. 5.3.2). Alignment gives positions, so it supports splicing analysis, variant calling and the discovery of unannotated genes; it needs a genome and an annotation (GTF) and, for , a lot of memory.

2. Pseudo-alignment or selective alignment to a transcriptome

 and  skip base-by-base alignment. They determine which transcripts are compatible with each read and then use an expectation-maximisation algorithm to share reads that fit several isoforms. They are fast, model multi-mapping reads and correct for fragment-level biases (GC content, sequence context). Transcript estimates are summed to genes for gene-level testing (). For gene-level differential expression they are as accurate as alignment-based counting.

3. De novo transcriptome assembly

Without a reference,  or  assemble reads into transcripts. This is the route for non-model organisms; it is slower and produces redundant transcripts that need clustering (Low & Tammi, ch. 8).

Counting reads per gene

After alignment,  or  assign each read to a gene using the annotation. A read overlapping two genes is ambiguous and, in the default "union" mode, not counted; a read aligning to several places is multi-mapping and usually skipped (Ismail, ch. 5.3.4). With a stranded library, telling the counter the right strand setting matters: the wrong setting can silently lose most reads. Pseudo-aligners make this choice for you ('s -l A detects the library type).

Is the library usable?

  • Mapping rate. Typically above 70–80 % to a good reference; far lower suggests contamination, rRNA, a wrong reference or a wrong strand setting.
  • Uniquely versus multi-mapped reads and the number of splice junctions ('s Log.final.out).
  • Gene-body coverage. Reads should cover transcripts evenly; a pile-up at the 3′ end means degraded RNA (Qualimap, ).
  • Composition. Share of rRNA and mitochondrial reads: high values point to failed depletion or stressed cells.
Adapter
Synthetic sequence added to fragments; appears in a read when the fragment is shorter than the read length.
Splice-aware aligner
An aligner that can split a read across an intron (, ).
Pseudo-alignment
Assigning a read to compatible transcripts without base-level alignment (, ).
Multi-mapping read
A read that fits several genomic locations or transcripts equally well.
GTF/GFF
Annotation files that list gene, transcript and exon coordinates.

Check your understanding

When would you choose  over ?

When you need positions: splice-junction or isoform discovery, variant calling in transcripts, or genes missing from the annotation. For gene-level differential expression with a good transcriptome,  is faster and equally accurate.

Why are PCR duplicates usually not removed in RNA-seq?

Because a highly expressed gene genuinely produces many identical fragments; removing them would compress exactly the differences you want to measure.

Most reads of one sample are "ambiguous" in . What would you check first?

The strand setting: with a reversed strand parameter, reads land on the wrong strand and overlap antisense or neighbouring genes.

Part V · Transcriptomics

Normalisation and exploring samples

Raw counts mix three things: how much a gene is expressed, how deeply the sample was sequenced, and what else was in the sample. Normalisation removes the last two so that the first can be compared. Then, before any test, you look at the samples.

Units for comparing within a sample

With k reads on a gene of length l in a library of N reads (Ismail, ch. 5.3.5):

CPM = k / N × 10⁶ · RPKM = k / (l/10³ × N/10⁶) · TPM = (k/l) / Σ(kⱼ/lⱼ) × 10⁶

FPKM is RPKM counted in fragments (read pairs). TPM divides by length first and then rescales, so TPM always sums to one million in every sample; RPKM does not, which makes RPKM values from different samples subtly incomparable.

Comparing between samples: composition bias

Scaling by library size assumes the total amount of RNA is shared out the same way in every sample. When a few genes become very highly expressed in one condition, they take reads from every other gene, and simple scaling makes unchanged genes look down-regulated. Two robust methods assume instead that most genes do not change:

  • TMM () compares each sample with a reference sample, computes log-ratios M for all genes, trims the most extreme 30 % of M and 5 % of the average abundances, and takes a precision-weighted mean of what remains (Ismail, ch. 5.3.5.4; Robinson & Oshlack 2010).
  • Median of ratios () divides each count by the gene's geometric mean across samples and takes the median of these ratios per sample (Ismail, ch. 5.3.5.5).

Filtering genes before testing

Genes with almost no reads carry no information and add to the multiple-testing burden. 's keeps genes with a minimum count-per-million in at least as many samples as the smallest group (Ismail, ch. 5.3.7.4).

A scale for plots

Count variance grows with the mean, so on the raw scale a few highly expressed genes dominate every distance, and on a naive log scale the noisy low counts dominate. The variance-stabilising transformation (VST) or regularised log of , and log-CPM with a prior count in , give roughly constant variance across the range. Use them for PCA, clustering and heatmaps, never for the test itself.

Looking at the samples

PCA on the 500 most variable genes is the first plot to make. Colour it by condition, then by every technical variable: if samples separate by extraction date or sequencing run, a batch effect is present (Baxevanis et al., Bioinformatics, ch. 10). Hierarchical clustering joins the two closest samples, recomputes distances, and repeats until one tree remains; single, complete and average linkage differ only in how the distance between groups is defined (Baxevanis et al., ch. 10; Zvelebil & Baum, Understanding Bioinformatics, ch. 16). A heatmap of row-scaled expression with its dendrograms shows which genes drive the groups.

Sanity checks catch errors no statistic can: expression of Y-chromosome genes (or XIST) must match the recorded sex. Baxevanis and colleagues describe a dataset in which samples clustered by sex rather than by tissue, revealing mislabelled samples.

Checking sample identity: the sex check

Sample swaps happen in every large study. A cheap test uses genes whose expression depends on sex: XIST is expressed from the inactive X in females, and genes on the Y chromosome (RPS4Y1, DDX3Y, KDM5D, UTY, EIF1AY) only in males. Plotting one against the other separates the samples into two clouds; a sample in the wrong cloud for its recorded sex was probably swapped or mislabelled (Baxevanis, Bader & Wishart, Bioinformatics, ch. 10). RNA-realm draws this plot in step 02 for human and mouse data and flags disagreements with a sex column in the metadata. The demo study contains one deliberately swapped label.

Check your understanding

Why does TPM, but not RPKM, sum to the same number in every sample?

TPM normalises by length first and then divides by the sum of the length-normalised values, so the total is fixed at 10⁶; RPKM divides by the read total before length correction, so its sum depends on the length distribution of the sample's reads.

In the median-of-ratios example, what would happen if half of all genes changed?

The median would itself move, and the size factors would be biased. Both TMM and median of ratios assume that most genes do not change; global shifts (for example, overall transcription doubling) need spike-in controls.

Your PCA separates samples by RNA extraction date along PC1. What do you do?

Check whether dates are balanced across conditions. If they are, include date as a covariate in the model. If one condition was extracted on one date only, the effect cannot be separated; report it as a limitation.

Part V · Transcriptomics

Differential expression, step by step

Differential expression asks, for every gene at once, whether the difference between groups is larger than the difference between replicates. The statistics look complicated, but they rest on four ideas: counts are over-dispersed, genes can borrow strength from each other, the design must be written down, and thousands of tests need a different kind of error control.

From binomial to negative binomial

Each read either comes from gene g or not, so the count of a gene in one library is binomial; because any single gene takes a tiny share of millions of reads, this becomes a Poisson distribution, whose variance equals its mean (Ismail, ch. 5.3.6). The Poisson describes technical sampling only. Independent biological replicates vary more, so the variance is modelled as

Var(Y) = μ + φ·μ²

where φ is the dispersion. Its square root is the biological coefficient of variation (BCV): φ = 0.1 means replicates differ by about 32 % (Ismail, ch. 5.3.7; Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R, ch. 11).

Borrowing strength between genes

With three replicates per group, a gene's own variance estimate is very noisy. The solution is empirical Bayes shrinkage: fit a trend of dispersion against mean expression across all genes, then pull each gene's estimate part-way towards the trend; genes with less information are pulled further.  computes a common, a trended and a gene-wise ("tagwise") dispersion;  and  use the same principle (Ismail, ch. 5.3.7.6).

Efron and Hastie explain why this works with a classic example: predicting baseball players' season batting averages from their first 90 at-bats. Shrinking every player's early average towards the overall mean (the James–Stein estimator) reduces the total error substantially, because each player borrows indirect evidence from the others. They also show the cost: a few genuinely extreme players are pulled too far (Efron & Hastie, Computer Age Statistical Inference, ch. 7). The same is true for genes, which is why shrunken fold changes ( in ) are best for ranking, while a gene with a large, well-supported effect keeps most of it.

Writing down the design

Each gene is fitted with a generalised linear model on the log scale, with the library size as an offset (Ismail, ch. 5.3.6; Faraway, Extending the Linear Model with R, ch. 5):

log μgi = xiᵀ βg + log Ni

Three families of methods

  • : negative binomial GLM, empirical Bayes dispersions, Wald test (or likelihood-ratio test for several levels), automatic handling of outlier counts (Cook's distance) and independent filtering.
  • : negative binomial GLM; the quasi-likelihood F-test () adds a second, gene-specific variability and is recommended for most designs (Ismail, ch. 5.3.7.8).
  • -: transforms counts to log-CPM, estimates the mean–variance trend and gives each observation a precision weight, then uses 's moderated t-statistics, originally developed for microarrays (Gentleman et al., Bioinformatics and Computational Biology Solutions Using R and Bioconductor, ch. 23).

All three control false discoveries well in designed experiments with few replicates. In large human cohorts (dozens of samples per group) their distributional assumptions can fail; the Wilcoxon rank-sum test then controls the false discovery rate better (Li et al. 2022).

Twenty thousand tests at once

At p < 0.05, testing 20,000 unchanged genes yields about 1,000 false positives. Controlling the family-wise error rate (Bonferroni) prevents even one, but is too strict to find anything. The false discovery rate (FDR) controls instead the expected proportion of false discoveries among the genes you report (Efron & Hastie, ch. 15).

The Benjamini–Hochberg recipe: order the N p-values, find the largest i with p(i) ≤ i·q/N, and reject hypotheses 1 to i.

Reading the results

  • Volcano plot: log fold change against −log₁₀ p. Top corners are large, well-supported changes.
  • MA plot: log fold change against mean expression. Before shrinkage the low-count genes fan out; after shrinkage they collapse towards zero (Ismail, ch. 5.3.8).
  • p-value histogram: a flat floor with a peak near zero is healthy. A hill in the middle or a peak near one points to a missing covariate or a wrong model.
  • Look at single genes. Plot normalised counts per sample for the top genes; a "hit" driven by one sample is a warning.

Beyond the adjusted p-value: the local false discovery rate

The Benjamini–Hochberg q-value controls the false discovery rate of the whole list above a threshold. Efron's local false discovery rate answers a sharper question for each gene: what is the probability that this particular gene is null? It needs two densities: f(z), the distribution of all genes' z-values, and f0(z), the distribution of null genes. Then lfdr(z) = π0 f0(z) / f(z), where π0 is the share of null genes (Efron & Hastie, Computer Age Statistical Inference, ch. 15).

Efron's key observation is that the null need not be the theoretical N(0, 1). Correlation between genes, hidden batches or unmodelled covariates widen or shift the bulk of the z-values; estimating the null from that bulk ("empirical null", by fitting a parabola to the log density in the middle, called central matching) gives more honest calls.

RNA-realm estimates the empirical null in step 05, draws the histogram with f(z) and π0f0(z), and adds an lfdr column to the results table.

Dispersion (φ)
The extra-Poisson variance of a gene between biological replicates; √φ is the biological coefficient of variation.
Empirical Bayes
Estimating a prior from all genes and using it to stabilise each gene's estimate.
Design matrix
The table of 0/1 and numeric columns that encodes groups and covariates in the model.
Contrast
The comparison you test, for example treated versus control.
FDR
The expected proportion of false discoveries among reported genes.
LFC shrinkage
Pulling noisy log fold changes of low-information genes towards zero.

Check your understanding

Why is the Poisson distribution not enough for biological replicates?

It assumes variance equals the mean, which only describes sampling of reads. Real individuals differ biologically, adding variance proportional to the mean squared; the negative binomial's dispersion term captures this.

In the BH example, would gene 5 be reported at q = 0.10?

Yes. The thresholds double (0.05 for i = 5), and 0.030 ≤ 0.050; in fact gene 6 (0.040 ≤ 0.060) would be reported too, giving six genes.

A gene has log₂ fold change 4.2 before shrinkage and 0.3 after. What does that tell you?

The large estimate rested on very little information (low counts or high variance); the data cannot support a big change for this gene. Rank it by the shrunken value.

Key references

  1. Ismail HD. Bioinformatics: A Practical Guide to Next Generation Sequencing Data Analysis. CRC Press, 2023. Read: Ch. 5.3.6–5.3.8: the binomial–Poisson–negative binomial argument, design matrices,  dispersions, quasi-likelihood fitting and result plots.
  2. Efron B, Hastie T. Computer Age Statistical Inference: Algorithms, Evidence, and Data Science. Cambridge University Press, 2016. Read: Ch. 7 James–Stein Estimation and Ridge Regression (indirect evidence and its cost) and ch. 15 Large-Scale Hypothesis Testing and FDRs.
  3. Gentleman R, Carey V, Huber W, Irizarry R, Dudoit S (eds.). Bioinformatics and Computational Biology Solutions Using R and Bioconductor. Springer, 2005. Read: Ch. 15 Multiple Testing Procedures and ch. 23 : Linear Models for Microarray Data.
  4. Faraway JJ. Extending the Linear Model with R: Generalized Linear, Mixed Effects and Nonparametric Regression Models, 2nd edition. CRC Press, 2016. Read: Ch. 5 Count Regression: Poisson and negative binomial GLMs.
  5. Love MI, Huber W, Anders S. Moderated estimation of fold change and dispersion for RNA-seq data with . Genome Biology 15, 550 (2014).
  6. Law CW, Chen Y, Shi W, Smyth GK. : precision weights unlock linear model analysis tools for RNA-seq read counts. Genome Biology 15, R29 (2014).
  7. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B 57, 289–300 (1995).
  8. Li Y, Ge X, Peng F, Li W, Li JJ. Exaggerated false positives by popular differential expression methods when analyzing human population samples. Genome Biology 23, 79 (2022).
  9. Statistical Analysis of Microbiome Data with R (Xia, Sun & Chen), ch. 7 Exploratory Analysis of Microbiome Data and Beyond. Read: Ch. 11 Modeling Over-Dispersed Microbiome Data: the same negative binomial ideas applied to microbial counts.
Part V · Transcriptomics

From genes to biology: pathways and co-expression

A list of 400 differentially expressed genes is not yet a result. Gene-set methods ask which processes those genes belong to; co-expression methods ask which genes rise and fall together. Both turn individual genes into biology.

Over-representation analysis (ORA)

ORA asks whether a gene set (for example "interferon response") contains more of your significant genes than chance would give. The probability comes from the hypergeometric distribution, as when drawing coloured balls from an urn without replacement.

The background must be the genes that could have been called significant (those tested), not the whole genome: lowly expressed tissue-specific genes would otherwise make every expressed pathway look enriched (Timmons et al. 2015).

Gene set enrichment analysis ()

ORA throws away everything below the significance cut-off.  uses the whole list ranked by the test statistic. Walking down the list, a running score rises at each gene in the set and falls otherwise; the maximum deviation is the enrichment score. A set whose genes cluster at the top (or bottom) of the ranking is enriched even if no single gene is significant, which is how coordinated small shifts in, say, mitochondrial genes are found. Significance comes from permuting gene sets of the same size.

Gene-set libraries

Start with the 50  Hallmark sets: curated, largely non-redundant summaries of well-defined processes. Then use GO Biological Process or Reactome for detail. Gene Ontology terms overlap heavily, so dozens of related "significant" terms often describe one signal. KEGG pathways are free for academic use only (Baxevanis et al., Bioinformatics, ch. 13 Biological Networks and Pathways).

Clustering genes by their profiles

Genes that respond together are often regulated together. Compeau and Pevzner tell this story through yeast: when Saccharomyces cerevisiae runs out of glucose it inverts its metabolism and starts consuming the ethanol it produced, the diauxic shift, and hundreds of genes change expression in coordinated waves. Clustering their expression profiles over time recovers groups of genes with shared functions (Compeau & Pevzner, Bioinformatics Algorithms: An Active Learning Approach, ch. 8 How Did Yeast Become a Wine Maker?).

  • k-means chooses k centres and alternates between assigning each gene to its nearest centre and moving each centre to the mean of its genes (Lloyd's algorithm). It needs k in advance and finds roughly spherical clusters.
  • Hierarchical clustering repeatedly merges the closest genes or clusters into a tree; cutting the tree gives clusters at any resolution.
  • Distance matters more than the algorithm: Pearson correlation (1 − r) groups genes with the same shape of response regardless of level, which is usually what you want.

Co-expression networks ()

 builds a network where every pair of genes is connected with a weight from their correlation, raised to a soft-threshold power β that keeps strong correlations and fades weak ones. Topological overlap then measures how many neighbours two genes share, and hierarchical clustering of 1 − overlap gives modules. Each module is summarised by its eigengene (its first principal component), which is correlated with sample traits; genes most correlated with the eigengene are the hubs.

Modules need many samples: with fewer than 15, correlations between genes are too noisy, and modules are hypotheses at best.

Gene set
A named list of genes sharing a function, location or pathway.
ORA
Over-representation analysis: a hypergeometric test of significant genes in a set.
Gene set enrichment analysis on the full ranked gene list.
Leading edge
The genes of a set that drive its  enrichment score.
Module eigengene
The first principal component of a co-expression module's expression.

Check your understanding

Why can  find a pathway in which no gene is significant on its own?

Because it tests whether the set's genes are concentrated at one end of the ranked list; many small shifts in the same direction add up even when each is too small to pass a per-gene threshold.

In the ORA example, what happens to the expected overlap if you use all 20,000 genes as background?

The expected count drops to 100 × 500 / 20,000 = 2.5, doubling the apparent enrichment. Unexpressed genes could never have been significant, so this inflates every result.

Why does k-means usually use correlation distance for expression profiles?

Two genes that both double after treatment have the same response even if one is 100 times more abundant; correlation compares shapes, Euclidean distance compares levels.

Part V · Transcriptomics

Guide: an RNA-seq study in RNA-realm

A walk through a complete bulk RNA-seq analysis in RNA-realm, from FASTQ files or a count table to a methods section, with the decision that matters at each step and the chapter that explains it.

Step 0 · Reads → counts (optional)

Upload paired (_R1/_R2) or single-end FASTQ files; they go up in pieces, so tens of gigabytes are fine. Choose the  human or mouse transcriptome, or upload your organism's transcript FASTA. RNA-realm trims with , quantifies with  (selective alignment, GC and sequence bias correction) and sums transcripts to genes. Open the  report: mapping rate, adapter content and duplication tell you whether each library is usable (chapter 25). Then load the results with a sample sheet.

Step 1 · Import counts

Or start from counts: one table with all samples, or one file per sample from , , , ,  or . The first column of the sample sheet must contain the sample names exactly as in the count file; every other column is a variable. Human and mouse genes get symbols, gene types and chromosomes automatically.

Step 2 · Quality control

Check library sizes, the gene-type composition, the mitochondrial share and the sample correlation heatmap. Flagged outliers are not removed: look at them in the PCA first (chapter 26).

Step 3 · Filter and normalise

Choose the grouping variable for ; keep TMM as a robust default, or median of ratios for  size factors. Keep VST for plots.

Step 4 · Explore

Colour the PCA by condition and then by every technical variable. The "drivers" heatmap gives the R² of each variable on each component; a technical variable with high R² on PC1 or PC2 belongs in the model. "Remove a batch for display" changes only plots.

Step 5 · Differential expression

Choose the variable, the reference level and the test level; tick covariates. Run  and - together; genes found by both form the consensus. With eight or more independent samples per group, add the Wilcoxon test. Read the p-value histogram before the gene list, and plot single genes (chapter 27).

Step 6 · Pathways

Start with the Hallmark collection, then GO or Reactome. Click a term for its running enrichment score and leading-edge genes (chapter 28).

Step 7 · Co-expression

With fifteen or more samples, find modules, look at which module follows your condition and which follows batch, and read each module's hub genes and best Hallmark term.

Step 8 · Report

The report writes a methods paragraph for exactly the steps you ran, with linked references. The Download menu exports every table.

Check your understanding

The demo's PCA separates samples by batch on PC2. Should you add batch to the DE model?

Yes. Batch is balanced across conditions in the demo, so including it removes its variance and increases power without biasing the comparison.

 reports 300 genes, - 250, and 220 are shared. Which list should you report?

Report the method you chose in advance with its FDR; the 220 consensus genes are the most robust and are a good starting point for interpretation, but the choice should be stated before looking.

Part VI · Fungal genomics

Fungal biology for genome analysts

Fungal genomes make sense only in the light of how fungi live: they digest food outside their cells, grow as networks of hyphae that may hold many nuclei, switch between haploid and diploid stages, and make chemical weapons. Each of these habits leaves a mark that a genome analyst has to recognise.

What fungi are

Fungi include moulds, yeasts and mushrooms. About 100,000 species have been described, and estimates of the real number reach 1.5 million or more; environmental sequencing suggests that over 90 % of fungal species remain undiscovered. Fungi are the closest relatives of animals among eukaryotes (Madigan et al., Brock Biology of Microorganisms, ch. 18).

Fungi are chemoorganotrophs that feed by secreting enzymes: they digest polymers such as cellulose, chitin and proteins outside the cell and absorb the monomers. Decomposition of wood is a hallmark: white-rot basidiomycetes degrade both cellulose and lignin, while brown-rot fungi take the cellulose and leave the lignin (Brock, ch. 18.8). In a genome this lifestyle appears as large families of carbohydrate-active enzymes (CAZymes) and secreted proteins.

The major groups

  • Microsporidia: tiny obligate parasites with drastically reduced genomes; Encephalitozoon has only 2.9 Mb and about 2,000 genes, fewer than E. coli, and lacks genes for whole metabolic pathways.
  • Chytridiomycota: the earliest-diverging fungi, with motile, flagellated zoospores; Batrachochytrium dendrobatidis is driving amphibian declines.
  • Zygomycota / Mucoromycota and Glomeromycota: coenocytic hyphae (bread mould Rhizopus); the arbuscular mycorrhizal fungi that colonise most plant roots.
  • Ascomycota: the largest group, including yeasts (Saccharomyces, Candida) and moulds (Aspergillus, Penicillium, Fusarium); sexual spores form in a sac (ascus).
  • Basidiomycota: mushrooms, rusts, smuts and the pathogen Cryptococcus; sexual spores form on a club-shaped basidium.

(Brock, ch. 18.9–18.13)

Hyphae, nuclei and ploidy: why they matter for sequencing

Most fungi grow as hyphae, tubular cells that extend at the tip and form a mycelium. Hyphae can be divided by cross-walls (septate) or be one continuous cell with hundreds of nuclei (coenocytic). In many basidiomycetes, two compatible haploid mycelia fuse and the resulting dikaryon keeps two genetically different nuclei in every cell for most of its life (Brock, ch. 18.13). For a genome project this means:

  • DNA from a dikaryon or a diploid yeast contains two haplotypes, which appear as heterozygosity and can make an assembler produce two copies of the same region;
  • DNA from a coenocytic or multinucleate isolate may mix slightly different nuclei;
  • single-spore or single-cell isolation is the best way to obtain a clean haploid genome.

Mating types: a built-in repeat

In baker's yeast, the active MAT locus determines mating type a or α. Two silent copies of the mating-type genes sit elsewhere in the genome, and a haploid cell can switch type by copying a silent cassette into MAT (Brock, ch. 18.12). Three near-identical copies of the same sequence are exactly the kind of repeat that breaks short-read assemblies. Mating-type loci are also of interest in their own right: their structure (one locus or two, two alleles or many) defines whether a species is heterothallic or homothallic.

What fungal genomes look like

FeatureTypical rangeConsequence for analysis
Genome size2.9 Mb (Microsporidia) to > 1 Gb (some rusts); most 20–60 MbLong reads and 30–50× coverage are affordable
Genes~2,000 to ~20,000Gene prediction needs a species-appropriate model
IntronsFew in yeasts, several per gene in filamentous fungiAb initio predictors must find splice sites
RepeatsUnder 5 % in yeasts to over 50 % in some pathogensMask before predicting genes
Chromosome endsTelomeric repeat TTAGGG in most fungi; irregular TG-rich repeats in budding yeastsComplete chromosomes have telomeres at both ends
Ribosomal DNATandem arrays of 18S–ITS1–5.8S–ITS2–28SCollapsed in assemblies; source of the ITS barcode

Genome history matters too. Compeau and Pevzner recount how the ancestor of Saccharomyces underwent a whole-genome duplication; most duplicates were later lost, leaving about 13 % of genes in pairs, such as the alcohol dehydrogenases ADH1 and ADH2 that let yeast make ethanol and then consume it (Compeau & Pevzner, Bioinformatics Algorithms, ch. 8). Such ancient duplicates explain why even a perfect yeast assembly shows a few "duplicated" genes in  (chapter 32).

Chemistry: secondary metabolites

Fungi produce antibiotics, toxins and pigments: penicillin from Penicillium, statins, aflatoxins from Aspergillus. The genes for one compound are usually clustered in the genome as a biosynthetic gene cluster, which is why tools such as  search for clusters rather than single genes (chapter 34).

Hypha / mycelium
A tubular fungal filament; the network of hyphae is the mycelium.
Coenocytic
Hyphae without cross-walls, with many nuclei in a shared cytoplasm.
Dikaryon
A cell or mycelium with two genetically different haploid nuclei.
MAT locus
The genomic locus that determines mating type.
ITS
Internal transcribed spacers between ribosomal RNA genes; the official fungal barcode.
CAZyme
Carbohydrate-active enzyme; families of enzymes that build or break down sugars.

Check your understanding

You sequence DNA from a mushroom's fruiting body. Why might the assembly be twice the expected size?

Fruiting bodies are dikaryotic: each cell carries two different haploid nuclei. Where the two haplotypes differ enough, the assembler builds both copies; purging duplicated haplotypes or assembling haplotypes separately solves this.

Why might a white-rot fungus have more lignin-related enzymes than a brown-rot fungus?

White-rot fungi degrade lignin as well as cellulose, which requires oxidative enzymes (CAZy "auxiliary activities" such as class II peroxidases); brown-rot fungi leave lignin largely intact.

Why can the Encephalitozoon genome be smaller than a bacterium's?

As an obligate intracellular parasite it relies on its host for basic metabolism, and lost the genes for whole pathways such as the citric acid cycle.

Part VI · Fungal genomics

Profiling and assembling a fungal genome

Before assembling, count k-mers: they tell you the genome size, how heterozygous it is and how much of it repeats. Then choose an assembler that matches your reads, and judge the result with numbers you understand.

Reading a k-mer spectrum

Cut every read into overlapping words of length k (k = 21 is typical) and count how often each distinct k-mer occurs. Plot the number of distinct k-mers against their count, and the genome speaks:

  • a spike at count 1–3 is sequencing errors, which create k-mers that occur once;
  • the main peak sits at the k-mer coverage of the genome;
  • a second peak at half that coverage means a heterozygous diploid (or a dikaryon): k-mers that differ between the two haplotypes are covered half as often;
  • a long tail at high counts is repeats.
main peakerrorsk-mer coverage →Haploid ½ coveragefull coveragek-mer coverage →Heterozygous diploid
Schematic k-mer spectra. The heterozygous peak at half coverage contains k-mers unique to one haplotype.

Three ways to assemble

Ismail groups assembly algorithms into three families (Ismail, Bioinformatics: A Practical Guide to Next Generation Sequencing Data Analysis, ch. 3):

  • Greedy: repeatedly merge the two reads with the best overlap. Simple, but a repeat can lead it down the wrong path for good.
  • Overlap–layout–consensus (OLC): reads are nodes, overlaps are edges, and the assembly is a path visiting every read once (a Hamiltonian path). Used for long reads (;  uses a related repeat graph).
  • de Bruijn graph: reads are cut into k-mers; each (k−1)-mer is a node and each k-mer an edge, and the assembly is a path using every edge once (an Eulerian path), which is computationally easy. Used for short reads (, ).

Long reads, short reads and polishing

Long reads (Oxford Nanopore, PacBio HiFi) span most fungal repeats, including transposons and rDNA units, and routinely give chromosome-scale contigs;  and  are standard choices. Short reads alone () give hundreds of contigs but accurate bases. A hybrid strategy assembles long reads and then polishes the consensus with short reads (, ), correcting the small insertions and deletions that remain in long-read assemblies, especially in homopolymers (Low & Tammi, Practical Bioinformatics for Beginners, ch. 6). Polishing should converge: the number of changes should fall sharply from one round to the next.

Contiguity: N50 and L50

Sort contigs from longest to shortest and add their lengths until half of the total is reached. The length of the contig that crosses the half-way mark is the N50; the number of contigs needed is the L50 (Ismail, ch. 3).

k-mer
A substring of length k from a read.
Coverage (depth)
The average number of reads covering each base of the genome.
Contig / scaffold
A contiguous assembled sequence; scaffolds join contigs with gaps of estimated size.
de Bruijn graph
A graph of overlapping k-mers whose Eulerian path spells the genome.
Polishing
Correcting residual base errors in an assembly using aligned reads.
N50 / L50
The contig length, and count, at which half the assembly is reached.

Check your understanding

Your k-mer spectrum has peaks at 25 and 50. What does that suggest?

A heterozygous diploid (or dikaryon) with homozygous k-mers at 50 and haplotype-specific k-mers at 25. Expect heterozygosity to complicate the assembly.

In the de Bruijn example, why does increasing k help?

With a larger k the repeated words (TC, CA) become longer, unique k-mers, so the branch disappears. Repeats shorter than k no longer create ambiguity.

 makes 128 changes in round one and 13 in round two. Should you run round three?

It is optional: the changes are converging. Another round may fix a few more, but large numbers in later rounds would suggest the reads disagree with the assembly for other reasons (repeats, heterozygosity).

Part VI · Fungal genomics

Is the assembly complete, clean and correctly named?

Three questions decide whether an assembly can be trusted: does it contain the whole genome, does it contain only this genome, and is the organism what you think it is?

Completeness with 

 searches the assembly for genes that are present in a single copy in almost every member of a lineage (for example 758 genes for all fungi, 2,137 for Saccharomycetes). The narrower the lineage, the more genes and the more sensitive the test (Low & Tammi, Practical Bioinformatics for Beginners, ch. 6). Each gene is reported as:

  • Complete, single copy (S): found once, full length. This should be the large majority.
  • Complete, duplicated (D): found more than once. A few percent can be real (yeast's ancient whole-genome duplication, chapter 30); many points to an unpurged diploid assembly or contamination with a relative.
  • Fragmented (F): only part found, often because of assembly errors or genes split across contigs.
  • Missing (M): not found at all.

Run  twice: on the genome (in genome mode, using the  aligner) and on the predicted proteins. A gap between the two (complete in the genome, missing among proteins) points to gene-prediction problems, not a missing genome.

Telomeres: counting complete chromosomes

Most fungi end their chromosomes with the telomeric repeat TTAGGG (budding yeasts use irregular TG₁₋₃ repeats). A contig with telomeric repeats at both ends is a complete chromosome. Counting them is the simplest proof of a telomere-to-telomere assembly; a contig with a telomere at one end is a chromosome arm.

Contamination: the GC/depth (blob) plot

Plot every contig by its GC content against its read depth, with dot size proportional to length:

  • the nuclear genome forms one cloud at the expected depth and GC;
  • the mitochondrial genome sits apart, usually at much higher depth (many copies per cell) and lower GC;
  • bacterial contaminants appear as a separate cloud at their own GC and depth;
  • small contigs at half depth in a diploid can be haplotype duplicates.

 classifies each contig as nuclear, mitochondrial, plastid or prokaryotic with a neural network trained on k-mer composition, which makes these clouds easy to name.

Identity: the ITS barcode

The internal transcribed spacers (ITS1 and ITS2) between the ribosomal RNA genes are the official DNA barcode of fungi: variable enough to separate species, flanked by conserved genes that are easy to find.  extracts the region; a  search against  or 's curated fungal ITS reference set confirms the species. Identification by ITS is as good as the reference database, and some genera (for example Fusarium, Aspergillus section-level) need secondary markers such as TEF1 or β-tubulin.

Benchmarking Universal Single-Copy Orthologs: a completeness test based on conserved genes.
Telomere-to-telomere
A chromosome assembled from one end to the other without gaps.
Blob plot
A GC-versus-coverage plot of contigs used to spot contamination.
Mitogenome
The mitochondrial genome; circular, high-copy, usually AT-rich in fungi.

Check your understanding

 reports C:96 %, D:42 %. What is the most likely explanation, and the fix?

The assembly contains both haplotypes of a heterozygous genome (or two related genomes). Check the k-mer profile; purge duplicated haplotigs () or assemble haplotypes separately.

One contig of 85 kb has 30-fold higher depth than the rest and 18 % GC. What is it?

Almost certainly the mitochondrial genome: high copy number per cell, low GC.

Why is ITS preferred over 18S rRNA for naming fungal species?

18S is too conserved to separate close species; the spacers evolve faster, giving species-level resolution, while the flanking rRNA genes make the region easy to amplify and find.

Key references

  1. Low LWY, Tammi MT (eds.). Practical Bioinformatics for Beginners: From Raw Sequence Analysis to Machine Learning Applications. World Scientific, 2023. Read: Ch. 6: evaluating assemblies with  and , with interpretation of the reports.
  2. Madigan MT, Bender KS, Buckley DH, Sattley WM, Stahl DA. Brock Biology of Microorganisms, 15th edition. Pearson, 2019. Read: Ch. 18 and ch. 19 Taking the Measure of Microbial Systems: why molecular identification uncovered the hidden diversity of fungi.
  3. Manni M, Berkeley MR, Seppey M, Simão FA, Zdobnov EM.  update: novel and streamlined workflows along with broader and deeper phylogenetic coverage for scoring of eukaryotic, prokaryotic, and viral genomes. Molecular Biology and Evolution 38, 4647–4654 (2021).
  4. Karlicki M, Antonowicz S, Karnkowska A. : deep learning-based classification system for eukaryotic sequences. Bioinformatics 38, 344–350 (2022).
  5. Bengtsson-Palme J, Ryberg M, Hartmann M, et al. Improved software detection and extraction of ITS1 and ITS2 from ribosomal ITS sequences of fungi and other eukaryotes for analysis of environmental sequencing data. Methods in Ecology and Evolution 4, 914–919 (2013).
  6. Gurevich A, Saveliev V, Vyahhi N, Tesler G. : quality assessment tool for genome assemblies. Bioinformatics 29, 1072–1075 (2013).
  7. Introduction to Bioinformatics in Microbiology (Christensen), ch. 2 DNA Sequence Assembly and Annotation of Genes. Read: Ch. 7 Sequence-Based Classification and Identification: principles of identifying organisms by marker genes, written for prokaryotes but equally valid for ITS.
Part VI · Fungal genomics

Repeats and gene prediction

Finding genes in a eukaryotic genome means recognising a grammar: a start codon, exons, splice sites, introns, a stop codon. Repeats must be hidden first, or the predictor will happily call every transposon a gene.

Repeats and masking

Fungal genomes carry transposable elements (retrotransposons with long terminal repeats, DNA transposons), simple sequence repeats, rDNA arrays and segmental duplications. Repeat content ranges from a few percent in yeasts to more than half the genome in some plant pathogens, where effector genes often sit in repeat-rich, fast-evolving compartments.  builds a species-specific library of repeat families de novo;  then marks every copy. Soft-masking writes repeats in lower case (ACGTacgtacgtACGT) instead of replacing them with N, so gene predictors can avoid starting genes in repeats yet still extend a real gene across a short repeat.

The grammar of a gene

A eukaryotic protein-coding gene is a sequence of states: intergenic region, start codon (usually in a Kozak-like context), initial exon, donor splice site (GT), intron, acceptor splice site (AG), internal exons, terminal exon, stop codon (Baxevanis, Bader & Wishart, Bioinformatics, ch. 5). A hidden Markov model (HMM) represents each element as a state with its own sequence statistics (codon usage in exons, base composition in introns, position-specific signals at splice sites) and transition probabilities between states. Given a genome sequence, the Viterbi algorithm finds the most probable path through the states, which is the predicted gene structure (Zvelebil & Baum, Understanding Bioinformatics, ch. 10).

The parameters come from a training set of genes whose structure is known, which is why the species model matters:  ships models for Saccharomyces, Aspergillus, Fusarium, Neurospora and others, and a model trained on a distant species misplaces introns.  introduced this design for human genes; , -ES (self-training) and  follow it (Baxevanis et al., ch. 5).

Ab initio versus evidence-based

  • Ab initio (intrinsic) predictors use only the genome sequence and the model. They are fast and need no extra data, but miss untypical genes and misjudge untranslated regions.
  • Evidence-based (extrinsic) predictors add RNA-seq alignments (which show exactly where introns are), protein alignments from related species, or both.  and  combine evidence with  and  to produce markedly better gene models.

RNA-seq of the same strain is the single most valuable addition to a fungal annotation project; the RNA-realm chapters (24–29) describe how to produce it.

How good are the predictions?

Accuracy is measured against a curated reference at the nucleotide, exon or gene level, counting true positives (TP), false positives (FP) and false negatives (FN) (Baxevanis et al., ch. 5).

In practice, without a curated reference, check  in protein mode, the distribution of protein lengths (a spike of very short proteins suggests fragmented models) and the share of genes with  domains.

Non-coding genes

tRNA genes are found with , which combines a fast first-pass search with covariance models of the tRNA secondary structure; ribosomal RNA genes with  or RNAmmer.

Codon usage: the statistics behind gene finding

The 61 sense codons encode 20 amino acids, so most amino acids have synonymous codons. Genomes do not use them equally: base composition (mutation bias) pushes towards GC- or AT-ending codons, and in highly expressed genes selection favours codons matching abundant tRNAs. Gene finders exploit this bias to tell coding from non-coding sequence (Zvelebil & Baum, Understanding Bioinformatics, ch. 10).

Wright's effective number of codons (ENC) summarises the bias of a gene in one number, from 20 (one codon per amino acid) to 61 (all synonymous codons equally used). Plotted against GC at synonymous third positions (GC3s), genes whose codon choice is explained by composition alone fall on the curve ENC = 2 + s + 29 / (s² + (1 − s)²); genes well below it are shaped by selection. Relative synonymous codon usage (RSCU) shows which codon is preferred for each amino acid.

FungFlow draws the ENC–GC3s plot and the RSCU table in the Genes results.

Soft-masking
Writing repeat sequence in lower case so that tools can treat it differently without losing it.
HMM
A probabilistic model of hidden states (exon, intron…) that emit observed bases.
Splice donor / acceptor
The 5′ (GT) and 3′ (AG) ends of an intron.
Training set
Genes of known structure used to estimate a predictor's parameters.
Sensitivity / precision
The share of true features found / the share of predictions that are true.

Check your understanding

Why does an HMM for gene prediction need a separate state for introns rather than treating them as "not exon"?

Introns have their own composition, length distribution and boundary signals; modelling them explicitly lets the model enforce the grammar (an intron must start with a donor and end with an acceptor, and must be followed by an exon).

A prediction has sensitivity 0.95 and precision 0.60. What went wrong?

It calls too much: almost all real coding sequence is found, but 40 % of what it calls coding is not. Typical causes are unmasked repeats or a model that over-predicts short exons.

 is 99 % complete on the genome but 85 % on the proteins. Where is the problem?

In gene prediction: the genes are in the assembly but the models are incomplete or split. Better species model or RNA-seq evidence will help.

Key references

  1. Baxevanis AD, Bader GD, Wishart DS (eds.). Bioinformatics, 4th edition. Wiley, 2020. Read: Ch. 5 Genome Annotation: intrinsic and extrinsic gene finding, Markov and hidden Markov models, eukaryotic gene structure,  and accuracy measures.
  2. Zvelebil M, Baum JO. Understanding Bioinformatics. Garland Science, 2008. Read: Ch. 9 Revealing Genome Features and ch. 10 Gene Detection and Genome Annotation (theory, including HMM decoding).
  3. Stanke M, Keller O, Gunduz I, et al. : ab initio prediction of alternative transcripts. Nucleic Acids Research 34, W435–W439 (2006).
  4. Burge C, Karlin S. Prediction of complete gene structures in human genomic DNA. Journal of Molecular Biology 268, 78–94 (1997).
  5. Flynn JM, Hubley R, Goubert C, et al.  for automated genomic discovery of transposable element families. PNAS 117, 9451–9457 (2020).
  6. Gabriel L, Brůna T, Hoff KJ, et al. : fully automated genome annotation using RNA-seq and protein evidence with -ETP, , and . Genome Research 34, 769–777 (2024).
  7. Chan PP, Lin BY, Mak AJ, Lowe TM. : improved detection and functional classification of transfer RNA genes. Nucleic Acids Research 49, 9077–9096 (2021).
  8. Introduction to Bioinformatics in Microbiology (Christensen), ch. 2 DNA Sequence Assembly and Annotation of Genes. Read: Ch. 2 DNA Sequence Assembly and Annotation of Genes.
  9. Wright F. The 'effective number of codons' used in a gene. Gene 87, 23–29 (1990).
  10. Sharp PM, Li W-H. The codon adaptation index: a measure of directional synonymous codon usage bias, and its potential applications. Nucleic Acids Research 15, 1281–1295 (1987).
Part VI · Fungal genomics

What the genes do: functional annotation of fungi

A list of 10,000 predicted proteins becomes biology when you know which ones digest plant walls, which make toxins, which are secreted to manipulate a host, and how this compares with relatives.

Transferring function by orthology

Most proteins are annotated by similarity to proteins whose function is known. Orthologs (genes separated by speciation) usually keep their function; paralogs (separated by duplication) often diverge.  assigns each protein to a precomputed orthologous group and transfers its annotations: gene name, GO terms, EC numbers, KEGG pathways and COG functional category. Protein domains (, ) add evidence at the level of building blocks (Baxevanis et al., Bioinformatics, ch. 7 Predictive Methods Using Protein Sequences).

CAZymes: reading a lifestyle from enzymes

Because fungi digest food outside their cells (chapter 30), the enzymes they secrete are a direct readout of what they eat. The CAZy database groups them into glycoside hydrolases (GH), glycosyltransferases (GT), polysaccharide lyases (PL), carbohydrate esterases (CE), auxiliary activities (AA, mostly oxidative enzymes, including those that attack lignin) and carbohydrate-binding modules (CBM).  annotates them with HMMs and sequence similarity.

Secondary metabolism: biosynthetic gene clusters

Genes for a fungal natural product usually lie together: a core enzyme (polyketide synthase, non-ribosomal peptide synthetase, terpene synthase), tailoring enzymes, a transporter and often a regulator.  finds these clusters, predicts their class and compares them with clusters of known products. Penicillin, made by Penicillium, is the classic example (Brock, ch. 18.9). A typical filamentous ascomycete has 30–70 clusters, most of whose products are unknown; this "cryptic" chemistry is why fungal genomes are mined for new drugs.

Secretome and effectors

Secreted proteins carry an N-terminal signal peptide (predicted by  or similar tools) and lack transmembrane domains. Plant- and animal-pathogenic fungi secrete effectors, often small, cysteine-rich proteins that manipulate the host;  predicts them from sequence features. Predictions are a shortlist for experiments, not proof of function.

Putting it in context

Annotation becomes meaningful by comparison: gene families expanded or lost relative to relatives (orthogroups from ), clusters unique to one strain, CAZyme profiles that set a pathogen apart from its saprotrophic sister species (Bergman, Comparative Genomics).

Ortholog / paralog
Homologous genes separated by speciation / by duplication.
COG category
A one-letter functional class (for example G, carbohydrate metabolism) from the COG/ system.
BGC
Biosynthetic gene cluster: co-located genes that make one natural product.
Secretome
The set of proteins an organism secretes.
Effector
A secreted protein that alters host cells to benefit the pathogen.

Check your understanding

Why is function transfer more reliable from an ortholog than from a paralog?

After a duplication, one copy is free to change function (chapter 30's ADH1/ADH2 example); orthologs in different species more often keep the ancestral function.

A newly sequenced isolate has many AA2 (class II peroxidase) genes and CBM1 domains. What lifestyle do you suspect?

A wood decayer, probably a white-rot fungus able to degrade lignin.

 reports 45 clusters. How many compounds do you expect to identify from the genome alone?

Usually only a handful: most fungal clusters are cryptic, with no close match to a characterised cluster. Their products need expression and chemistry to identify.

Key references

  1. Baxevanis AD, Bader GD, Wishart DS (eds.). Bioinformatics, 4th edition. Wiley, 2020. Read: Ch. 3 Assessing Pairwise Sequence Similarity (), ch. 7 Predictive Methods Using Protein Sequences and ch. 9 Molecular Evolution (orthology).
  2. Madigan MT, Bender KS, Buckley DH, Sattley WM, Stahl DA. Brock Biology of Microorganisms, 15th edition. Pearson, 2019. Read: Ch. 18: fungal nutrition, wood rot and natural products such as penicillin.
  3. Bergman NH (ed.). Comparative Genomics. Humana Press, 2007. Read: : methods for comparing genomes and finding conserved and lineage-specific elements.
  4. Charles TC, Liles MR, Sessitsch A (eds.). Functional Metagenomics: Tools and Applications. Springer, 2017. Read: : screening for enzymes and natural products, the same targets seen from the community side.
  5. Cantalapiedra CP, Hernández-Plaza A, Letunic I, Bork P, Huerta-Cepas J.  v2: functional annotation, orthology assignments, and domain prediction at the metagenomic scale. Molecular Biology and Evolution 38, 5825–5829 (2021).
  6. Zheng J, Ge Q, Yan Y, et al. : automated carbohydrate-active enzyme and substrate annotation. Nucleic Acids Research 51, W115–W121 (2023).
  7. Blin K, Shaw S, Augustijn HE, et al. : new and improved predictions for detection, regulation, chemical structures and visualisation. Nucleic Acids Research 51, W46–W50 (2023).
  8. Floudas D, Binder M, Riley R, et al. The Paleozoic origin of enzymatic lignin decomposition reconstructed from 31 fungal genomes. Science 336, 1715–1719 (2012).
Part VI · Fungal genomics

Guide: a fungal genome in FungFlow

How to go from sequencing reads of one fungal isolate to an assembled, checked and annotated genome in FungFlow, and how to read each part of the report, with the chapter that explains it.

1. Reads, or an assembly

In Reads, upload Illumina pairs, Nanopore or PacBio HiFi FASTQ files. If you already have an assembly, upload the FASTA in Assembly instead: FungFlow then starts at quality control. For a new isolate the best value today is Nanopore R10 (30–50×) plus Illumina (40×) for polishing, or HiFi alone (chapter 31).

2. Set up the run

In New run, mark each file (Illumina R1, R2, long reads, assembly). Give the expected genome size if you know it, the ploidy (haploid for most filamentous ascomycetes), the narrowest  lineage that contains your fungus, the  gene model of the closest species, and the telomere repeat (TTAGGG for most fungi). De novo repeat modelling is slower but gives much better masking. Runs queue on the server and continue when you close the page; follow them in Runs.

3. Read the report

  • Read quality: yield, long-read N50, Q30 of short reads (chapter 31).
  • Genome profile: the k-mer estimate of size and heterozygosity, independent of assembly; compare it with the assembly size.
  • Assembly: size, contigs, N50 and the cumulative length curve;  rounds should drop to few changes.
  • Completeness:  of the genome and of the proteins (chapter 32).
  • Contigs & identity: the GC/depth plot and  classes, contigs with telomeres at both ends, and the ITS barcode with a link to  .
  • Repeats, genes, function: masked fraction, gene and protein statistics, tRNAs, COG categories, CAZyme classes and biosynthetic gene clusters (chapters 33–34).

4. Before you publish

Remove contaminant contigs, check that  duplication is low, and consider RNA-seq evidence for gene models ( or ) if genes are your focus. Submit the genome to  or ENA with the run's methods text and the tool versions from the log.

Check your understanding

 estimates 32 Mb but the assembly is 61 Mb. What do you check?

Heterozygosity in the k-mer profile and  duplication: both haplotypes are probably assembled. , or assemble the haplotypes separately.

The example yeast has 19 contigs for 16 chromosomes. What could the other three be?

The mitochondrial genome and small contigs from repeats such as the rDNA array or subtelomeric regions; the GC/depth plot and  show which is which.

Part VII · Going further

The library: books and the state of the art

These docs are a map. This chapter points to the territory: the textbooks that teach each module properly, the order in which to read them, and the papers that define the current state of the art. Every entry is in the reference list at the end, with a link.

A learning path

StageGoalRead
1 · FoundationsWhat sequencing data are and how they are handledChristensen, Introduction to Bioinformatics in Microbiology (ch. 1–3); Low & Tammi, Practical Bioinformatics for Beginners (ch. 3–4); Buffalo, Bioinformatics Data Skills (ch. 2–3, 10–11)
2 · Core methodsAlignment, assembly, annotation, expressionIsmail, Bioinformatics: A Practical Guide to NGS Data Analysis (ch. 1–5); Baxevanis, Bader & Wishart, Bioinformatics (ch. 3, 5, 10)
3 · StatisticsModels, testing and learning from many variablesJames et al., An Introduction to Statistical Learning (ch. 2–6, 12–13); Faraway, Extending the Linear Model with R (ch. 5)
4 · Your moduleThe specialist texts belowXia, Sun & Chen and Xia & Sun for microbiomes; Christensen and Bergman for bacterial genomes; Ismail ch. 5 and Gentleman et al. for RNA-seq; Brock ch. 18 with Ismail ch. 3 for fungi
5 · Research levelWhy the methods work and where they failEfron & Hastie, Computer Age Statistical Inference (ch. 6–7, 15); Compeau & Pevzner, Bioinformatics Algorithms; Zvelebil & Baum, Understanding Bioinformatics; the papers below

Foundations shared by all modules

Christensen (ed.), Introduction to Bioinformatics in Microbiology (2018)
A course-sized textbook written for microbiologists: databases, alignment, phylogeny, identification, amplicon and shotgun metagenomics, transcriptomics and typing, each in one chapter with exercises. The best first book for MicroRes and BactFlow users.
Low & Tammi (eds.), Practical Bioinformatics for Beginners (2023)
Hands-on chapters from raw-read quality control to assembly, transcriptomics, metagenomics and machine learning, with commands to run.
Ismail, Bioinformatics: A Practical Guide to NGS Data Analysis (2023)
Each chapter explains the algorithm and then runs the tools: quality control, mapping, de novo assembly, variant discovery, RNA-seq and metagenomics.
Baxevanis, Bader & Wishart (eds.), Bioinformatics, 4th ed. (2020)
The standard reference for concepts: sequence similarity, genome annotation, expression analysis, networks and pathways.
Zvelebil & Baum, Understanding Bioinformatics (2008)
The theory behind the tools, with derivations: hidden Markov models, gene detection, clustering and statistics.
Compeau & Pevzner, Bioinformatics Algorithms (2015)
Algorithms taught through biological questions; volume II's chapter 8 uses yeast's switch to wine-making to teach clustering of expression data.
Buffalo, Bioinformatics Data Skills (2015)
The craft: Unix, file formats, Git, R and robust pipelines. Read it before your first large project.

Statistics

James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed., 2021)
Regression, classification, resampling, regularisation, trees, unsupervised learning and multiple testing, with labs; free from the authors' website.
Efron & Hastie, Computer Age Statistical Inference (2016)
How statistics changed with computing: the bootstrap, empirical Bayes, the missing-species problem, false discovery rates and the empirical null. Used in MicroRes (unseen taxa) and RNA-realm (local fdr).
Faraway, Extending the Linear Model with R (2nd ed., 2016)
Generalised linear and mixed models; chapter 5 on count regression explains over-dispersion and the negative binomial.
Gentleman et al. (eds.), Bioinformatics and Computational Biology Solutions Using R and Bioconductor (2005)
The origins of  and of multiple-testing practice in genomics, from the people who wrote the software.
Lee (ed.), Statistical Bioinformatics (2010)
A collection of statistical methods for high-throughput biological data.

MicroRes: microbiome analysis

Books. Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R (2018), is the statistician's text: power and sample size (ch. 5), diversity (ch. 6–9), compositional analysis (ch. 10) and models for over-dispersed and zero-inflated counts (ch. 11–12). Its successor, Xia & Sun, Bioinformatic and Statistical Analysis of Microbiome Data (2023), adds the path from raw reads (QIIME 2, DADA2, ASVs) to longitudinal mixed models. Zhong Wang's Introduction to Computational Metagenomics (2022) explains the algorithms of shotgun metagenomics; Mitra's Metagenomic Data Analysis (2023) gives protocols; Streit & Daniel's Metagenomics: Methods and Protocols and Izard & Rivera's Metagenomics for Microbiology cover the laboratory side; Charles, Liles & Sessitsch's Functional Metagenomics covers function-based screening; Paradis's Analysis of Phylogenetics and Evolution with R covers trees.

The state of the art. Microbiome data are compositional and must be analysed as such (Gloor et al. 2017). Exact sequence variants replaced OTUs (Callahan et al. 2016, 2017). For shotgun data, MetaPhlAn 4 extended marker-based profiling to unnamed species (Blanco-Míguez et al. 2023), and HUMAnN 3 profiles function (Beghini et al. 2021). Diversity should be compared at equal sample coverage (Chao & Jost 2012); whether to rarefy is still debated, with recent evidence in favour of repeated rarefaction (Schloss 2024). For differential abundance, a benchmark of 14 methods on 38 datasets found that they disagree widely, and recommended consensus across methods (Nearing et al. 2022); bias-corrected methods such as ANCOM-BC2 (Lin & Peddada 2024) and linear models such as MaAsLin 2 (Mallick et al. 2021) are current standards. Networks use compositionally aware estimators (Kurtz et al. 2015). The STORMS checklist sets the reporting standard (Mirzayi et al. 2021).

BactFlow: bacterial genomes

Books. Christensen's chapters on assembly and annotation (ch. 2), identification (ch. 7) and typing (ch. 11); Ismail on quality control, mapping, assembly and variants (ch. 1–4); Low & Tammi on comparing assemblies (ch. 6); Bergman's Comparative Genomics for whole-genome comparison.

The state of the art. With Nanopore R10.4 chemistry, long reads alone now give near-finished bacterial genomes (Sereika et al. 2022); consensus assembly from several assemblers improves reliability further (Wick et al. 2021), and short-read polishing that respects repeats remains useful (Wick & Holt 2022, ). Deep-learning variant callers outperform classical ones on bacterial Nanopore data (Hall et al. 2024). Annotation moved to fast, database-rich tools such as  (Schwengers et al. 2021); completeness and contamination to machine learning (; Chklovski et al. 2023); taxonomy to the Genome Taxonomy Database and  (Parks et al. 2022; Chaumeil et al. 2022) with fast ANI (Jain et al. 2018; Shaw & Yu 2023). Pangenomes are built with error-aware tools such as Panaroo (Tonkin-Hill et al. 2020), and resistance genes found with curated databases such as  (Feldgarden et al. 2021).

RNA-realm: transcriptomics

Books. Ismail's RNA-seq chapter (ch. 5) covers quantification, normalisation and differential expression with worked commands; Christensen (ch. 10) and Low & Tammi (ch. 8) give shorter introductions; Baxevanis et al. (ch. 10) cover design, batch effects and exploration; Gentleman et al. explain  and multiple testing; Efron & Hastie (ch. 15) the false discovery rate and empirical null; Compeau & Pevzner (vol. II, ch. 8) and James et al. (ch. 12) clustering.

The state of the art. A community survey still frames good practice (Conesa et al. 2016). Lightweight, bias-aware quantification (; Patro et al. 2017) imported at gene level (Soneson et al. 2015) is standard.  (Love et al. 2014) with heavy-tailed fold-change shrinkage (Zhu et al. 2019) and - (Law et al. 2014) remain the reference methods; with large human cohorts, however, they can produce excess false positives and rank tests are safer (Li et al. 2022). At least six biological replicates per group are recommended (Schurch et al. 2016). Batch effects in counts can be adjusted with  (Zhang et al. 2020). Gene-set analysis uses fast pre-ranked  (Korotkevich et al. 2021) on curated sets such as the Hallmarks (Liberzon et al. 2015), and co-expression networks follow  (Langfelder & Horvath 2008).

FungFlow: fungal genomes

Books. Brock Biology of Microorganisms (ch. 18) for fungal biology; Compeau & Pevzner for yeast genome duplication; Ismail (ch. 3) and Low & Tammi (ch. 6) for assembly; Baxevanis et al. (ch. 5) and Zvelebil & Baum (ch. 9–10) for gene prediction.

The state of the art. Fungal genomics is organised around large sequencing programmes such as the 1000 Fungal Genomes project and the  portal (Grigoriev et al. 2014); Stajich (2017) reviews what genomes revealed about the kingdom. Genome profiling uses k-mers (; Ranallo-Benavidez et al. 2020); assembly uses long reads with  (Kolmogorov et al. 2019) or, for HiFi data,  (Cheng et al. 2021). Completeness is measured with  (Manni et al. 2021). Repeats are modelled with  (Flynn et al. 2020); gene prediction now combines RNA-seq and protein evidence (; Gabriel et al. 2024). Function is annotated with  for CAZymes (Zheng et al. 2023),  7 for secondary metabolism (Blin et al. 2023),  6 for secretion (Teufel et al. 2022) and  3 for effectors (Sperschneider & Dodds 2022). The evolution of wood decay was reconstructed from CAZyme repertoires across 31 genomes (Floudas et al. 2012).

Keeping up

  • Benchmarks beat claims: community challenges such as CAMI compare metagenomic tools on the same data (Meyer et al. 2022), and independent benchmarks such as Nearing et al. (2022) are worth more than a method's own paper.
  • Read method papers' supplementary benchmarks and check the data they were tested on resemble yours.
  • Pin versions: a method that is state of the art today will be updated next year; the methods text each Bioinformaniac tool writes records the versions.

Check your understanding

You are new to the field and want to analyse 16S data properly. Which three books, in which order?

Christensen (ch. 1–3 and 8) for foundations, then Xia & Sun (2023) for the path from reads to tables and tests, then Xia, Sun & Chen (2018) for the statistics in depth.

Why read an independent benchmark rather than the paper introducing a method?

Method papers are written by the developers, on data and settings where the method works well. Independent benchmarks test many methods on the same, varied datasets.

Key references

  1. Christensen H (ed.). Introduction to Bioinformatics in Microbiology. Learning Materials in Biosciences, Springer, 2018.
  2. Low LWY, Tammi MT (eds.). Practical Bioinformatics for Beginners: From Raw Sequence Analysis to Machine Learning Applications. World Scientific, 2023.
  3. Ismail HD. Bioinformatics: A Practical Guide to Next Generation Sequencing Data Analysis. CRC Press, 2023.
  4. Baxevanis AD, Bader GD, Wishart DS (eds.). Bioinformatics, 4th edition. Wiley, 2020.
  5. Zvelebil M, Baum JO. Understanding Bioinformatics. Garland Science, 2008.
  6. Compeau P, Pevzner P. Bioinformatics Algorithms: An Active Learning Approach, 2nd edition, vol. II. Active Learning Publishers, 2015.
  7. Buffalo V. Bioinformatics Data Skills: Reproducible and Robust Research with Open Source Tools. O'Reilly, 2015.
  8. James G, Witten D, Hastie T, Tibshirani R. An Introduction to Statistical Learning with Applications in R, 2nd edition. Springer, 2021.
  9. Efron B, Hastie T. Computer Age Statistical Inference: Algorithms, Evidence, and Data Science. Cambridge University Press, 2016.
  10. Faraway JJ. Extending the Linear Model with R, 2nd edition. CRC Press, 2016.
  11. Gentleman R, Carey V, Huber W, Irizarry R, Dudoit S (eds.). Bioinformatics and Computational Biology Solutions Using R and Bioconductor. Springer, 2005.
  12. Lee JK (ed.). Statistical Bioinformatics: A Guide for Life and Biomedical Science Researchers. Wiley-Blackwell, 2010.
  13. Xia Y, Sun J, Chen D-G. Statistical Analysis of Microbiome Data with R. ICSA Book Series in Statistics, Springer, 2018.
  14. Xia Y, Sun J. Bioinformatic and Statistical Analysis of Microbiome Data: From Raw Sequences to Advanced Modeling with QIIME 2 and R. Springer, 2023.
  15. Wang Z. Introduction to Computational Metagenomics. World Scientific, 2022.
  16. Mitra S (ed.). Metagenomic Data Analysis. Methods in Molecular Biology 2649. Humana Press, 2023.
  17. Streit WR, Daniel R (eds.). Metagenomics: Methods and Protocols, 2nd edition. Humana Press, 2017.
  18. Izard J, Rivera MC (eds.). Metagenomics for Microbiology. Academic Press, 2015.
  19. Charles TC, Liles MR, Sessitsch A (eds.). Functional Metagenomics: Tools and Applications. Springer, 2017.
  20. Paradis E. Analysis of Phylogenetics and Evolution with R, 2nd edition. Springer, 2012.
  21. Bergman NH (ed.). Comparative Genomics. Humana Press, 2007.
  22. Madigan MT, Bender KS, Buckley DH, Sattley WM, Stahl DA. Brock Biology of Microorganisms, 15th edition. Pearson, 2019.
  23. Gloor GB, Macklaim JM, Pawlowsky-Glahn V, Egozcue JJ. Microbiome datasets are compositional: and this is not optional. Frontiers in Microbiology 8, 2224 (2017).
  24. Callahan BJ, McMurdie PJ, Rosen MJ, et al. DADA2: high-resolution sample inference from Illumina amplicon data. Nature Methods 13, 581–583 (2016).
  25. Callahan BJ, McMurdie PJ, Holmes SP. Exact sequence variants should replace operational taxonomic units in marker-gene data analysis. The ISME Journal 11, 2639–2643 (2017).
  26. Blanco-Míguez A, Beghini F, Cumbo F, et al. Extending and improving metagenomic taxonomic profiling with uncharacterized species using MetaPhlAn 4. Nature Biotechnology 41, 1633–1644 (2023).
  27. Beghini F, McIver LJ, Blanco-Míguez A, et al. Integrating taxonomic, functional, and strain-level profiling of diverse microbial communities with bioBakery 3. eLife 10, e65088 (2021).
  28. Chao A, Jost L. Coverage-based rarefaction and extrapolation: standardizing samples by completeness rather than size. Ecology 93, 2533–2547 (2012).
  29. Schloss PD. Rarefaction is currently the best approach to control for uneven sequencing effort in amplicon sequence analyses. mSphere 9, e00354-23 (2024).
  30. Nearing JT, Douglas GM, Hayes MG, et al. Microbiome differential abundance methods produce different results across 38 datasets. Nature Communications 13, 342 (2022).
  31. Lin H, Peddada SD. Multigroup analysis of compositions of microbiomes with covariate adjustments and repeated measures. Nature Methods 21, 83–91 (2024).
  32. Mallick H, Rahnavard A, McIver LJ, et al. Multivariable association discovery in population-scale meta-omics studies. PLoS Computational Biology 17, e1009442 (2021).
  33. Kurtz ZD, Müller CL, Miraldi ER, et al. Sparse and compositionally robust inference of microbial ecological networks. PLoS Computational Biology 11, e1004226 (2015).
  34. Mirzayi C, Renson A, Genomic Standards Consortium, et al. Reporting guidelines for human microbiome research: the STORMS checklist. Nature Medicine 27, 1885–1892 (2021).
  35. Sereika M, Kirkegaard RH, Karst SM, et al. Oxford Nanopore R10.4 long-read sequencing enables the generation of near-finished bacterial genomes from pure cultures and metagenomes without short-read or reference polishing. Nature Methods 19, 823–826 (2022).
  36. Wick RR, Judd LM, Cerdeira LT, et al. Trycycler: consensus long-read assemblies for bacterial genomes. Genome Biology 22, 266 (2021).
  37. Wick RR, Holt KE. : short-read polishing of long-read bacterial genome assemblies. PLoS Computational Biology 18, e1009802 (2022).
  38. Hall MB, Wick RR, Judd LM, et al. Benchmarking reveals superiority of deep learning variant callers on bacterial nanopore sequence data. eLife 13, RP98300 (2024).
  39. Schwengers O, Jelonek L, Dieckmann MA, et al. : rapid and standardized annotation of bacterial genomes via alignment-free sequence identification. Microbial Genomics 7, 000685 (2021).
  40. Chklovski A, Parks DH, Woodcroft BJ, Tyson GW. : a rapid, scalable and accurate tool for assessing microbial genome quality using machine learning. Nature Methods 20, 1203–1212 (2023).
  41. Parks DH, Chuvochina M, Rinke C, et al. : an ongoing census of bacterial and archaeal diversity through a phylogenetically consistent, rank normalized and complete genome-based taxonomy. Nucleic Acids Research 50, D785–D794 (2022).
  42. Chaumeil P-A, Mussig AJ, Hugenholtz P, Parks DH.  v2: memory friendly classification with the genome taxonomy database. Bioinformatics 38, 5315–5316 (2022).
  43. Jain C, Rodriguez-R LM, Phillippy AM, Konstantinidis KT, Aluru S. High throughput ANI analysis of 90K prokaryotic genomes reveals clear species boundaries. Nature Communications 9, 5114 (2018).
  44. Shaw J, Yu YW. Fast and robust metagenomic sequence comparison through sparse chaining with . Nature Methods 20, 1661–1665 (2023).
  45. Tonkin-Hill G, MacAlasdair N, Ruis C, et al. Producing polished prokaryotic pangenomes with the Panaroo pipeline. Genome Biology 21, 180 (2020).
  46. Feldgarden M, Brover V, Gonzalez-Escalona N, et al.  and the Reference Gene Catalog facilitate examination of the genomic links among antimicrobial resistance, stress response, and virulence. Scientific Reports 11, 12728 (2021).
  47. Conesa A, Madrigal P, Tarazona S, et al. A survey of best practices for RNA-seq data analysis. Genome Biology 17, 13 (2016).
  48. Patro R, Duggal G, Love MI, Irizarry RA, Kingsford C.  provides fast and bias-aware quantification of transcript expression. Nature Methods 14, 417–419 (2017).
  49. Soneson C, Love MI, Robinson MD. Differential analyses for RNA-seq: transcript-level estimates improve gene-level inferences. F1000Research 4, 1521 (2015).
  50. Love MI, Huber W, Anders S. Moderated estimation of fold change and dispersion for RNA-seq data with . Genome Biology 15, 550 (2014).
  51. Zhu A, Ibrahim JG, Love MI. Heavy-tailed prior distributions for sequence count data: removing the noise and preserving large differences. Bioinformatics 35, 2084–2092 (2019).
  52. Law CW, Chen Y, Shi W, Smyth GK. : precision weights unlock linear model analysis tools for RNA-seq read counts. Genome Biology 15, R29 (2014).
  53. Li Y, Ge X, Peng F, Li W, Li JJ. Exaggerated false positives by popular differential expression methods when analyzing human population samples. Genome Biology 23, 79 (2022).
  54. Schurch NJ, Schofield P, Gierliński M, et al. How many biological replicates are needed in an RNA-seq experiment and which differential expression tool should you use? RNA 22, 839–851 (2016).
  55. Zhang Y, Parmigiani G, Johnson WE. : batch effect adjustment for RNA-seq count data. NAR Genomics and Bioinformatics 2, lqaa078 (2020).
  56. Korotkevich G, Sukhov V, Budin N, et al. Fast gene set enrichment analysis. bioRxiv 060012 (2021).
  57. Liberzon A, Birger C, Thorvaldsdóttir H, et al. The Molecular Signatures Database Hallmark gene set collection. Cell Systems 1, 417–425 (2015).
  58. Langfelder P, Horvath S. : an R package for weighted correlation network analysis. BMC Bioinformatics 9, 559 (2008).
  59. Grigoriev IV, Nikitin R, Haridas S, et al.  portal: gearing up for 1000 fungal genomes. Nucleic Acids Research 42, D699–D704 (2014).
  60. Stajich JE. Fungal genomes and insights into the evolution of the kingdom. Microbiology Spectrum 5, FUNK-0055-2016 (2017).
  61. Ranallo-Benavidez TR, Jaron KS, Schatz MC.  and Smudgeplot for reference-free profiling of polyploid genomes. Nature Communications 11, 1432 (2020).
  62. Kolmogorov M, Yuan J, Lin Y, Pevzner PA. Assembly of long, error-prone reads using repeat graphs. Nature Biotechnology 37, 540–546 (2019).
  63. Cheng H, Concepcion GT, Feng X, Zhang H, Li H. Haplotype-resolved de novo assembly using phased assembly graphs with . Nature Methods 18, 170–175 (2021).
  64. Manni M, Berkeley MR, Seppey M, Simão FA, Zdobnov EM.  update: novel and streamlined workflows along with broader and deeper phylogenetic coverage for scoring of eukaryotic, prokaryotic, and viral genomes. Molecular Biology and Evolution 38, 4647–4654 (2021).
  65. Flynn JM, Hubley R, Goubert C, et al.  for automated genomic discovery of transposable element families. PNAS 117, 9451–9457 (2020).
  66. Gabriel L, Brůna T, Hoff KJ, et al. : fully automated genome annotation using RNA-seq and protein evidence with -ETP, , and . Genome Research 34, 769–777 (2024).
  67. Zheng J, Ge Q, Yan Y, et al. : automated carbohydrate-active enzyme and substrate annotation. Nucleic Acids Research 51, W115–W121 (2023).
  68. Blin K, Shaw S, Augustijn HE, et al. : new and improved predictions for detection, regulation, chemical structures and visualisation. Nucleic Acids Research 51, W46–W50 (2023).
  69. Teufel F, Almagro Armenteros JJ, Johansen AR, et al.  predicts all five types of signal peptides using protein language models. Nature Biotechnology 40, 1023–1025 (2022).
  70. Sperschneider J, Dodds PN. : prediction of apoplastic and cytoplasmic effectors in fungi and oomycetes. Molecular Plant-Microbe Interactions 35, 146–156 (2022).
  71. Floudas D, Binder M, Riley R, et al. The Paleozoic origin of enzymatic lignin decomposition reconstructed from 31 fungal genomes. Science 336, 1715–1719 (2012).
  72. Meyer F, Fritz A, Deng Z-L, et al. Critical Assessment of Metagenome Interpretation: the second round of challenges. Nature Methods 19, 429–440 (2022).
Part VIII · User manual

Getting started on Bioinformaniac

This manual is the place to look when you are stuck. Part I–VII of the handbook explain the science; this part explains the website: where your data goes, how a run is started and followed, where the results are, how to read them and how to take them home. Each tool has its own chapter (M2–M8) with the same structure, so once you know one you can find your way in all of them.

How the website is laid out

The top bar is the same on every page. Tools opens the seven workbenches; each opens inside the site, so the top bar, your account and the status bar stay with you. Docs is this handbook, My data holds everything you uploaded and every result, and your picture opens Account. On a phone the same entries sit in the tab bar at the bottom.

Your dataYour questionToolManual
Feature or count tables (QIIME 2, DADA2, MetaPhlAn, /Bracken, HUMAnN, PICRUSt2)Who is there, how communities differ, which taxa or functions changeMicroResM2
Reads of one bacterial isolateAssemble, check and annotate a bacterial genomeBactFlowM3
RNA-seq FASTQ files or a gene count tableWhich genes and pathways change between conditionsRNA-realmM4
Reads of one fungal isolate (Nanopore, HiFi, Illumina)A complete, annotated fungal genome; species; mitochondrion; food-safety genesFungFlowM5
Human genome, exome or panel reads, BAM/CRAM or VCFVariants, their interpretation, families, tumour–normalHumaniacM6
Insect reads, or many individuals on one referenceA chromosome-level insect genome, or population structureInsectraM7
Reads or an assembly of a bacterial strain used in food or feedIs the strain safe? A dossier along EFSA's requirementsBactomaniacM8

Accounts, plans and guests

Without an account you can look at every tool, open the example and demo results and read the handbook. Uploading data, starting runs, downloading reports and asking our AI need a free account: press Sign in or Create a free account in the top bar. Pricing lists the plans; a plan sets how many CPUs and how much memory your runs may use at once and how much storage you have. Your current plan and usage are under Account → Plan & usage.

In Account you can also change your name, affiliation and picture (they appear on your reports), your password, and whether you receive an e-mail when a run ends. Your data → Download my data gives you everything the site stores about you; Delete account removes the account and all its files and cannot be undone.

Uploading data

Every tool has a Data page (in the left menu of the tool) with a Choose files button. Files go up in 8 MB pieces, so files of many gigabytes are fine; a progress bar shows each file and a Cancel button stops it. You can leave the page open while it uploads; closing the tab stops the upload.

  • Reads: FASTQ, gzipped or not. Choose all files of one sequencing run at once (MinKNOW writes many small files; the tools join them). Illumina pairs must be named _R1/_R2 (or _1/_2) so the pairs are found.
  • Assemblies and references: FASTA (.fa, .fasta, .fna), with GFF3 or  files where a tool asks for them.
  • Tables (MicroRes, RNA-realm): TSV, CSV, TXT or XLSX.

Administrators see Use files already on the server: paste full paths, one per line (a folder or a pattern such as /data/run7/*.fastq.gz works too). The files are linked, not copied, and removing them from Data never deletes the original.

Runs, the status bar and e-mails

Heavy work runs on the server's queue (), not in your browser. When you press Start, every step of the run becomes its own job; you can close the page, and the run goes on. The status bar under the top bar shows what is running for you on any page, with its progress; click a run there to open its live page. When a run ends you get an e-mail (switch it off in Account).

Your plan's CPUs and memory are shared by your running steps: when they are all in use, further steps wait in the queue until one finishes. Steps that run out of memory are retried with more, up to your plan's limit.

What every pipeline tool shares

FungFlow, Insectra, Humaniac and Bactomaniac are built the same way, and RNA-realm and MicroRes follow the same ideas:

  • Left menu: Home, Data, the numbered modules (or steps), Runs and Results.
  • A module page: the inputs and settings on the left; on the right Steps of this setup shows which steps will run before you start; then Start. While the module runs, the same page shows its live station: one node per step with its state and time, an ⓘ bubble with the estimated time and resources, the  jobs and a coloured live log in which errors are shown as cards with their cause.
  • Runs: every run as a card with Live, Results so far / Show results, Stop, Resume (continues a stopped or failed run and reuses every step that finished), Run this step again, Delete and Clean up (removes temporary work files; results are kept).
  • Results: pick a run at the top (tick several to see them together and compare), then a tab per part of the analysis. Every card has a title; figures are numbered Fig. 1, Fig. 2 … and tables Table 1, Table 2 … across the whole report, the same numbers as in the PDF report, so you can point at one in an e-mail or ask our AI about it. The ⓘ How to read this button on a card explains it.
  • Download ▾: Everything (zip) and each result file on its own. Report ▾: open the report in a new tab ready to print, or save it as PDF or a single HTML file. When several runs are open, the Report menu also offers one compiled report of all of them.

Working with figures and tables

Figures are interactive: hover for values, drag to zoom, double-click to reset, click a legend entry to hide a series, and use the camera icon in the figure's corner to save it as an image (SVG, which stays sharp at any size). Tables can be searched with the box above them (every column is searched), paged with ‹ and ›, and downloaded as CSV with the CSV button; numbers are right-aligned. When several runs are compared, the rows where the runs differ are highlighted. Very large result tables also come as files in Download ▾.

My data

My data (top bar) lists, per tool, everything you uploaded and every result the tools wrote, with the storage you use. Download a single file, or a whole folder as a ZIP, or delete what you no longer need. MicroRes keeps your loaded datasets and saved analysis there too.

Where to see the validation

The Validation tab in the top menu (and the link at the foot of every page) shows how FungFlow and Bactomaniac perform on public genomes with a known biology: strains made, strains correct, each acceptance criterion with its measured value, and its status (met, missed or not run yet). Three buttons follow: Open the full report (a printable page with every strain and check), Download the evidence pack (accessions, expected results, measured results and software versions, as files) and Validate on your own strains (opens Contact with the request filled in). The address is /validation. Each tool's own manual has a section "Validation tests for the pipeline".

Getting help: Ask our AI

The ✦ Ask our AI button is on every page. It answers questions about using the site, about what is on your screen and your own runs, and about the bioinformatics behind them, from this handbook first and with references.

  • It sees your screen: the tool, the run and tab you have open, and every numbered figure and table with its data. Ask "How do I read Fig. 3?" or "Is Table 2 good?".
  • Select text anywhere (a value, a gene name, a warning) and press Ask our AI about this; or copy text and paste it into the question box. The quote and where it came from go with your question.
  • One conversation thread per tool: FungFlow questions stay in the FungFlow chat, handbook questions in the Handbook chat. The clock button lists earlier conversations; you can reopen or delete them. + starts a new one.
  • The battery shows how many questions you have left today; it refills at midnight.
  • Move it: drag the button with the mouse, or touch it and drag it with your finger on a phone (it wiggles to show it is ready). It stays where you put it.

It reads only your own runs and cannot start, stop or delete anything. Answers can be wrong: check the numbers in your report.

Part VIII · User manual

MicroRes manual: microbiome analysis

MicroRes analyses microbiome tables in the browser: you load counts or profiles, clean and normalise them, then explore diversity, test differences and build a report. Nothing is queued: each step computes when you press its button, and every result adds a methods paragraph to the report. The science is in chapters 4–13.

Layout

The left menu has two modules that fold open: Taxonomic analysis in four phases (Data: Import, Quality control, Contaminants, Normalise & transform; Explore: Composition, Alpha diversity, Beta diversity, Phylogeny, Core microbiome; Test: Differential abundance, Associations, Co-occurrence networks; Predict: Biomarkers & prediction) and Functional metagenomics (F1 Import functions to F8 Antimicrobial resistance) and Viral metagenomics (V1 Import viral data to V7 Clinical report). Below them are Report, Guide & walkthrough and About. The CPU and RAM wheels in the menu show what MicroRes uses; click them to change the cores and memory budget.

Every step has a control deck on the left (settings, grouped and foldable; settings that apply to one choice appear only when it is chosen) and the results on the right. Once a result is drawn, changing a setting that feeds it redraws it. The How to read the results button at the top of each step explains every output; the ? on a card explains that card.

Loading data (step 01)

Open Import and choose the format under What are you loading?. Each format shows its own file boxes and a note:

  • Count table + metadata + taxonomy (optionally a phylogenetic tree and representative sequences): features (OTUs/ASVs/taxa) in rows, samples in columns; .tsv, .txt, .csv or .xlsx. Say which column holds the IDs, whether there is a header row, and whether the taxonomy is one lineage string or one column per rank.
  • Taxonomic profiles: MetaPhlAn profiles,  reports (--report, not the per-read output), Bracken outputs, per-sample taxonomy tables or a data matrix. Profiles of any number of samples are merged on the full lineage; a taxon missing from a sample becomes 0. A one-sample file is named after the file up to the first _.

The Demo data ▾ menu loads a simulated diet study with known effects, so you can check that each method finds them; the demo input files can be downloaded to see the expected formats. The orange Clean session button in the top bar (after a confirmation) clears everything loaded and computed, including the reference, so that you can load new data; for signed-in users it also deletes the saved analysis, while uploaded files and downloads stay untouched.

Cleaning and quality control (steps 02–03)

Quality control removes features and samples that add noise: chloroplasts, mitochondria and unassigned features, chosen lineages, shallow samples, and features that are rare (prevalence and mean-abundance filters). Every change updates a live preview: the filter funnel (features and share of reads left after each filter), the prevalence–abundance map (removed features should sit low and to the left), sample checks (depth against Good's coverage; flagged samples are diamonds) and the list of removed features with the reason. Nothing changes until you press Apply quality control; Use all data keeps the data as it is. A single sample can be cleaned too: the filters only refuse when they would remove every sample or every feature. If your table holds relative abundances (every sample sums to 1 or 100) and the filters remove features, a Rescale switch appears: on, every sample is scaled back to its original total (100 % or 1) after the removal; off (the default), the remaining abundances are left as they are and no longer add up to the total.

Contaminants (decontam) finds reagent and kit contaminants from negative controls or DNA concentrations. Use it for low-biomass samples when you have controls or concentrations; skip it otherwise. Undo restores the data.

Normalise and transform (step 04)

Sequencing depth differs between samples for technical reasons, so this step is required before any result: first a normalisation (correcting depth), then a transformation (changing the scale for the analysis). Choose the normalisation from your data and the transformation from the analysis you plan; Compare all transformations shows how each behaves on your data. The science is in chapter 7.

Exploring and testing (steps 05–13)

  • Composition: stacked bars of the most abundant taxa per sample or group.
  • Alpha diversity: richness and evenness compared between groups at equal sampling effort (ch. 8).
  • Beta diversity: choose a distance, ordinate, and test whether sample variables explain the structure (PERMANOVA with a dispersion check; ch. 9).
  • Phylogeny and Core microbiome: an interactive tree with abundance rings, and the taxa consistently present.
  • Differential abundance: by default five methods run on the same comparison and the consensus is reported, because methods disagree (ch. 10).
  • Associations, Co-occurrence networks and Biomarkers & prediction (cross-validated models and LEfSe).

The functional module repeats the pattern for HUMAnN, PICRUSt2 or  profiles: import (F1), clean and normalise (F2), then profiles, beta diversity, differential functions, who carries each function, and species–function networks.

Analyse it for me

As soon as your data are loaded, MicroRes asks how you want to continue. I'll analyse it myself takes you to quality control and every step with your own settings. Analyse it for me asks one question: is the study descriptive (describe the sample or samples) or comparative? For a comparison, choose the variable of interest (for example Diet), tick the groups to compare and the reference group they are compared with. Two more choices follow. Covariates are the other columns that could change the result, such as sex, age, BMI or batch: tick the ones to adjust for and the PERMANOVA, the association tests, differential abundance, functional beta diversity and differential functions include them (likely ones are ticked for you). A random effect is for repeated measures, when the same subject, patient or cage was sampled more than once: choose it and the alpha-diversity and functional models use a random intercept for it, so the repeated samples are not treated as independent. MicroRes then runs every suitable analysis with sound defaults (quality control, normalisation, composition, diversity, core, pangenome, differential abundance of each group against the reference with five methods, associations, network, prediction, and the functional steps when a profile is loaded, ending with the antimicrobial resistance screen) and tells you when the report is ready. Every result stays in its step, so you can open any of them, change a setting and run it again. The Analyse it for me button in the top bar brings the choice back. While it runs, Stop the analysis cancels it: the analyses that had finished are kept (and the report covers them), the rest are not run.

Display. Every figure follows the theme (Light, Plasma, Dark: dark background and light text in the dark themes, also in the report preview; downloads and the PDF stay light) and the size of its card. Wherever you choose how many taxa or functions to draw, the slider has a number box beside it and goes up to the number in your data (the label says all when everything is selected); a figure too tall or too wide for its card then scrolls inside the card. Colour legends keep a short, fixed length. The circular pangenome zooms with the mouse wheel and pans by dragging. Every report starts with the logo of the module that made it.

Antimicrobial resistance (step F8)

Is the sample free of antimicrobial resistance? Step F8 reads the names and identifiers of the functions in the profile you imported, recognises the resistance determinants with a curated rule table (beta-lactamases and carbapenemases, vancomycin Van ligases, PBP2a, tetracycline, macrolide, aminoglycoside, quinolone, sulfonamide, trimethoprim, phenicol, colistin MCR, fosfomycin and others) and gives every sample a verdict: no determinant found, only intrinsic or generic functions, determinants of common classes, determinants of clinical concern, or determinants against last-resort drugs. The result goes down to the drug class, the mechanism, the KEGG resistance pathway map where one exists, and the gene, with the taxa that carry it when the profile has per-taxon strata. The report has its own chapter with the same layers.

Read the verdict with its limits. AMR-safe here means no acquired determinant was found above the detection floor (1 per million annotated functions by default) in this profile. It needs a profile with gene-level names: KEGG orthologs,  annotations or a named table give the full view; enzyme classes and reactions show only enzymatic resistance, and the verdict then says it cannot confirm the absence; pathway tables and unnamed UniRef gene families cannot show resistance genes at all, and the step says so instead of calling the sample safe. A gene that is present is not a resistant organism, point mutations (gyrA, parC, rpoB) are invisible, and generic functions such as multidrug efflux pumps and chromosomal AmpC are listed as context only. Confirm any finding on assembled contigs with  or RGI/.

Pangenome (step 14)

The pangenome idea from bacterial genomics applied to your communities: everything found across the samples (the pangenome) is split into a core shared by (almost) all samples and an accessory part that varies (shell and the rare cloud). Accumulation curves show whether each new sample still brings new taxa or functions (an open pangenome) or not (closed). With a variable of interest you get one pangenome per group, their shared core and the taxa or functions found only in one group. With a stratified functional profile every main species gets its own pangenome of functions across the samples where it occurs, which shows how variable its strains are. With one sample, the functions are compared across the species of that sample.

Two tables count the taxa (or functions) in each partition: one per group of your variable, one per sample (how many of the sample's taxa are hard core, soft core, shell or cloud, and the accessory share, which is high for unusual communities). The circular pangenome shows presence and absence at a glance: the most abundant taxa sit around the circle, ordered from hard core to cloud, and each ring is one sample, with the rings of a group side by side (the coloured strip in the gap names the group). Choose how many taxa to draw (10–500) and whether to draw features, species, genera or families before you press Build pangenome. The Core microbiome chapter of the report has the same tables for genera.

Reference ranges (module R1)

If you have a reference population (a healthy cohort, a product specification, a published study), load it in the Taxonomy reference and Functional reference fields, each in one of two ways: a JSON file ({"key-map": {"species": [{"name": …, "min", "q_10", "q_25", "median", "mean", "q_75", "q_90", "max", "sd"}]}}, optionally with alpha-diversity statistics of the cohort) or a cohort table with taxa (or functions) in the rows and the samples of the cohort, for instance 100, in the columns; the statistics are then calculated for you. Templates and a 50-sample example table can be downloaded from the step. Without a reference, Build your own reference makes one from the samples you have loaded (all of them or one group); it is switched off while you load your own files, and the result can be downloaded and reused. Every sample is placed in the reference feature by feature: below the range, low, typical (Q1–Q3), high or above the range, with its percentile. A position strip per taxon shows the reference cohort as a mini density spectrum from its minimum to its 90th percentile with an arrow where your sample lands, and the figures show reference boxes with your samples or group medians on top, a percentile map and the share of features in each zone. Choose how many features to show, or all of them; every setting updates the results at once.

Viral metagenomics module (steps V1 to V7, below the Report in the side menu)

The module under Viral metagenomics in the left menu analyses a virome: the abundance of viral genomes (vOTUs) in each sample. It loads its data into the same slot as a taxonomic dataset, so steps 04 to 13 (normalise, alpha, beta, core, differential abundance, associations, networks, prediction) run on the vOTUs unchanged, and the module adds the steps a virome needs. Read the chapters on viruses and viromes (V1 to V3 of the handbook) first: they explain why each step is there.

  1. V1 Import viral data. Give the vOTU by sample table of reads, or a  table; the breadth of coverage per vOTU and sample, which the presence test needs; the annotation files of the genomes ( virus summary,  quality summary, a host table from  or any vOTU and host table,  output,  hits, or any vOTU table; each is recognised from its header and several can be given, the first file listed winning where two give the same column); the sample metadata; and, optionally, a genus by sample table of bacterial counts for the virus-host links. The page reports how many vOTUs each annotation covers and how much of the virome can be named at each rank. Load the viral demo gives a ready example: real phage genomes and real tool output with simulated reads, labelled as such wherever it appears.
  2. V2 Quality and detection. Four checks in one step. Genome quality: a minimum length (10 kb by default, the advice for abundance work), a minimum  tier, a minimum  score and a limit on host contamination, with a funnel that accounts for every vOTU that is not analysed. Presence: a vOTU counts as present in a sample only if reads cover at least the chosen share of its genome (0.75 by default); cells below it are set to zero, and a scatter of breadth against reads shows what the rule removes. Negative controls: choose the column and value that mark the blanks; vOTUs more prevalent in the blanks than in the samples are flagged as reagent contaminants and removed unless you keep them, and a known control sequence such as phiX174 is reported. Samples: minimum viral reads, viral share of the library and number of vOTUs, with pass or fail for every sample and the option to leave failing ones out. The blanks are left out of the analysis by default. When it is applied, steps 04 to 13 open.
  3. V3 Composition. The virome by any rank of the lineage, or by genome quality, lifestyle, host phylum or host genus, with the unclassified share kept as its own category and shown beside the share of vOTUs and reads that can be named at each rank.
  4. V4 Hosts and links. Composition by predicted host genus, the number of vOTUs per host and, if a bacterial table was imported, whether each virus rises and falls with its predicted host: centred log-ratio abundances, rank or Pearson correlation, p values corrected for the number of pairs. A link supports a host prediction; it does not prove infection.
  5. V5 Lifestyle and safety. A screen of each vOTU for toxin, virulence and resistance genes (from the imported  table), a temperate lifestyle (a lifestyle prediction, or an integrase or similar gene) and genome completeness. The verdict is Concern for a strong gene hit, Review for a weak hit, a temperate lifestyle, an incomplete genome or host-derived sequence, and No flag found otherwise; the reason for every verdict is written out with the method and identity of each hit. It is a screen and not a clearance.
  6. V6 Study report. The verdict and the quality gate come first: up to eight checks (presence tested, blanks clean, samples passing, reads on good genomes, enough samples, genomes annotated, safety screen run), each passed, flagged for review or failed under a rule printed beside it. Then one chapter per analysis (data quality, what is in the virome, diversity, what differs, hosts, lifestyle and safety), each with its figures, a caption that says what the figure shows in this study, and a note on how to read it. The last chapter holds the methods, the limitations, a glossary, the fingerprint of the analysed table and an optional sign-off block. Read it in the page; accounts with download access can save it as PDF or HTML.
  7. V7 Clinical report. For patient specimens. The input is the table of reads per virus and specimen, with the breadth of coverage, and negative controls named blank, NTC or water. The report gives one result per patient (virus detected, low level to verify, not detected, or invalid), the evidence behind it (reads, reads per million, how much of the genome is covered, the ratio to the negative control), the run quality that makes a negative believable (library size, internal control, negative controls), notes on each virus found, and the reporting rules and limits in full. Viruses that most people carry for life (herpesviruses, polyomaviruses, anelloviruses) are labelled as such. The rules are Bioinformaniac's own and are printed in the report; they are not a validated diagnostic procedure, and a laboratory must validate them on its own assay before using them for patient care.

Reports and downloads

Figures export as PNG, SVG, PDF or HTML and tables as CSV from the card's own buttons; networks also for Cytoscape. Download ▾ in the top bar holds your data at each stage (after QC, contaminant removal, normalisation, functional cleaning).

Report in the left menu shows the report in the page before you download it. It reads your results the way an analyst would, in clickable chapters: an Overview with a verdict per group, a signal board (diversity, butyrate producers, pathobionts, short-chain fatty acid and inflammatory functions and more, each marked favourable or unfavourable against the reference group) and the main findings ranked by strength; then Data & quality, Who is there (with ecological guilds), Diversity, Core microbiome, What differs, Links to measurements, Networks, Prediction, Functions & health (vitamins, SCFA, fibre and mucus use, protein fermentation, sulfide, inflammatory signals, bile acids, TMA, GABA, resistance), Who does what (each function traced to the microbes that carry it and linked to the taxa that changed), Against the reference and a Conclusion with precautions and recommendations. Chapters appear only for analyses that were run. Every chapter starts with why it matters; Who is there introduces the main players and portrays each group and sample, and Pangenome shows what is shared and what is individual. How we classified lists every health module and guild with its matching rule, the functions or genera it matched in your data, and the references, so you can check each assignment. Every sentence is filled from your numbers with fixed thresholds (effect sizes, agreement between methods, p-values), so the same data always give the same text. Download it as HTML (interactive figures), PDF (every chapter, A4) or Markdown; the methods text alone is under the preview.

Troubleshooting

Part VIII · User manual

FungFlow manual: fungal genomes

FungFlow turns the reads of a fungal isolate into a complete, checked and annotated genome in four modules: Read QC, Assembly, Taxonomy & mitochondrion, and Annotation. Run them one by one or chained; each heavy step is its own job on the server. The science is in chapters 30–35.

Layout

The left menu: Home, Data (My data), the four modules numbered 01–04, Runs (all runs), Results and Visualize the genome. Load the example on Home opens a finished run of a yeast with simulated reads, so you can see every result before running your own.

Loading data

Open Data and press Choose files. Upload:

  • Long reads as FASTQ: Oxford Nanopore (R10.4.1, R10.3 or R9.4.1; you choose the flow cell) or PacBio HiFi/CLR. Choose all files of a run together; FungFlow joins them. You say which technology they are in Read QC.
  • Illumina pairs named _R1/_R2 (or _1/_2), for polishing, hybrid assembly,  QV or Illumina-only assembly.
  • Optionally an assembly FASTA (to start at Taxonomy or Annotation) and a reference genome FASTA of the closest species (for , chromosome names and reference scaffolding).

Module 01 · Read QC

Choose the read files, say which kind of long reads they are, and set the minimum read quality and length (defaults Q10 and 1 kb suit Nanopore R10; raise them when you have more than 80× coverage). Start runs  (Illumina),  and  (long reads) and Jellyfish +  (genome profile).

  • Read N50: half of all bases sit in reads at least this long; above 15 kb gives chromosome-level fungal contigs.
  • Coverage: 40–80× of long reads is ideal; below 25× expect a fragmented assembly.
  • : one peak = haploid or homozygous; two peaks (the first at half the depth) = dikaryon or diploid. Heterozygosity above ~0.5 % → turn on in Assembly.

On the QC results, Assemble these reads opens Assembly with this run's filtered reads and recommended settings filled in; check them and press Start.

Module 02 · Assembly

Choose the reads (a QC run's filtered reads, recommended, or files from Data) and tick one or more assemblers: , , (long reads), (Illumina only) or  hybrid. Running two lets you compare; FungFlow keeps all and passes the best on. With both long and Illumina reads, the long-read assemblies are polished with  and  automatically; on the older Nanopore flow cells (R9.4.1, R10.3)  runs first.

  • (): for dikaryotic mushrooms and diploids, removes the second copy of regions where the nuclei differ.
  • Reference genome (optional):  compares against it, chromosomes are named after it, and Scaffold onto the reference () orders contigs along it.
  • Scaffold with the long reads (): joins contigs that reads span, without a reference.

You get chromosomes (telomere to telomere when the repeat is at both ends), scaffolds, contigs and an AGP file. The Genome quality card compares  ,  QV and k-mer completeness, contiguity and contamination with the Earth BioGenome Project minimum (contig N50 ≥ 1 Mb, QV ≥ 40,  ≥ 90 %). The score ranks the assemblies; Keep another assembly forces a different one for the next modules.

Module 03 · Taxonomy & mitochondrion

Choose an assembly (the best of an Assembly run, or a FASTA) and keep the reads (a run's own reads are used automatically) so the mitochondrion can also be rebuilt from them. Set the mitochondrial genetic code: 4 for almost all fungi, 3 only for the Saccharomycetaceae, 16 for chytrids such as Spizellomyces.

  • Organisms in the assembly: every 200 kb of every contig is matched to the closest fungal genome in . A mixed sample shows up as contigs that belong to different organisms. In the Genome browser (Stack view) Keep an organism… lists them with their size; tick the one to keep, the contigs of the others are taken out, and you download the cleaned FASTA or save it as the assembly for the next steps (no re-assembly needed; annotation has to be run again on the cleaned genome). Contigs without a close genome stay by default, and a contig made of two organisms is flagged as mixed.
  • Taxonomy:  contig classes (bacterial or archaeal contigs are contamination), the ITS barcode against  (≥ 98.5 % identity usually the same species) and the whole genome against every fungal genome in  with  (ANI ≥ 95 % same species).
  • Mitochondrion: contigs that match fungal mitogenomes are found; the mitochondrion is also rebuilt from the reads in two rounds (the second bait round closes the circle);  circularises, rotates and annotates it and  annotates it again. The card says Circular: yes only when the circle was closed; otherwise it says why, and how many pieces were found.
  • Nuclear genome: the assembly without the mitochondrion, used by Annotation.

Module 04 · Annotation

Choose an assembly (best: the nuclear genome of a Taxonomy & mitochondrion run), the gene caller (: deep learning, most accurate default; : trains  with protein evidence, slower; with a pre-trained species: quickest), the repeat mode (de novo  library, best for new genomes, or quick) and the function tools (, , , ). Genes of interest (food safety) runs too.

  • Genes: most fungi have 6,000–15,000; mushrooms 12,000–20,000. Many more usually means split genes, unmasked transposons, or both nuclei kept in the assembly.
  •  on proteins should be within a few points of the genome .
  • CAZymes, gene clusters () and peptidases describe the lifestyle and chemistry.

Following and repairing runs

After Start, the module page shows the run's station: each step with its state and time, the ⓘ bubble with the estimated time and resources, the  jobs and the live log. Runs lists every run. A step that fails shows a red error card with the tool's message; fix the cause, then press Resume (it reuses every step that finished) or Run this step again for one step. Start stays off while the same module is still running in another run. Resuming a run made before the Organisms in the assembly step existed adds that step to its chain; the finished steps stay cached and only the new one runs. A finished run's genome can be made the default input that later modules take from it.

Results tabs

Open Results, pick a run (tick several to compare them; differing rows are highlighted), and use the tabs: Read QC, Assembly, Taxonomy & mitochondrion, Annotation and Methods (a methods text ready to paste, with references). Figures and tables carry numbers (Fig. 1, Table 1…) across the whole report; the ⓘ on a card explains how to read it.

Visualize the genome

Visualize the genome opens a genome run together with the runs that annotated it: the Genome map (circular, karyotype or mitogenome view; exports as SVG or PNG; click a region to open it in the browser), the Genome browser (zoom from the whole genome to bases; layers for genes, repeats, GC, depth and annotations; search a gene) and Genes of interest · food safety (mycotoxin and mushroom-toxin pathways, allergens and food enzymes, and whether a pathway's genes sit together as a cluster).

Reports and downloads

Download ▾ on a run: Everything (zip) (assemblies, mitogenome, genes, proteins, reports, methods) and each file on its own. Report ▾: open the report in a new tab ready to print, PDF (results, methods, references and run log, about a minute to make) or one self-contained HTML file. The report carries your name, the date, a run log with every step,  job and tool version. With several runs open, the Report menu offers a compiled report of all of them.

Validation tests for the pipeline

Before you rely on FungFlow for a release decision, you can see how it performs on fungi whose biology is known. We run the pipeline on 15 public genomes and score every answer against a truth that does not come from our own software. The results, the strain list and the evidence pack are on the Validation page of the site (see Getting started, "Where to see the validation").

The strains

  • Toxin producers and pathogens: Aspergillus flavus NRRL 3357 (aflatoxin), A. fumigatus Af293, Fusarium graminearum PH-1 (trichothecenes), F. verticillioides 7600 (fumonisin), F. oxysporum 4287, Candida albicans SC5314, Candidozyma auris B8441 (formerly Candida auris), Cryptococcus neoformans H99.
  • Safe or useful strains: A. oryzae RIB40 (koji), A. niger CBS 513.88, Penicillium rubens IBT 27055, F. oxysporum Fo47 (a biocontrol strain), Saccharomyces cerevisiae S288C.
  • Models: A. nidulans FGSC A4, Neurospora crassa OR74A.

Two of them are deliberate traps. A. oryzae RIB40 carries a complete aflatoxin cluster at the DNA level, but the cluster is silent in this domesticated koji mould, so a good tool finds the cluster and does not call the strain a toxin producer without further evidence. P. rubens has a degraded patulin cluster, which tests that "present" and "functional" are told apart.

Where the truth comes from

The expected clusters come from , the curated database of experimentally characterised biosynthetic clusters, not from , the tool under test. For each cluster we fetch its own  record and locate it in the genome with  (dc-megablast). A cluster counts as present when at least 80 % of it is covered, absent when 25 % or less is covered, and partial in between. Every claim in the panel was confirmed this way before it was used. Several claims we first wrote from memory turned out wrong and were corrected from the measured data.

What is measured

  • Species from the taxonomy step (renamed species such as Candidozyma auris are accepted as the same taxon), ITS and  completeness.
  • Clusters: every curated cluster that must be present is a sensitivity test; every cluster that must be absent is a false-call test. A cluster counts as found when an  region overlaps its location, and a region is a false call when its closest known cluster is one the genome does not carry.
  • Mitochondrion size against the published one.
  • Assembly (yeasts only): reads are simulated from the published genome as Nanopore R10.4.1 (40x, about 1 % error), assembled with  and compared with the published genome: base accuracy (QV) and the share recovered.

Pass marks, fixed beforehand

The acceptance criteria were written down on 17 November 2026, before the first result existed, and are not changed afterwards. A measure that misses its criterion is shown as missed.

MeasureCriterion
Species called correctlyat least 90 %
Curated toxin and drug clusters foundat least 90 %
Clusters called that the genome does not carryat most 1
Assembly base accuracy, simulated R10.4.1 reads (QV)at least 40
Share of the published genome recoveredat least 98 %

What it does not show

  • The taxonomy database is , so most panel genomes are in it and their species call is easy. The telling cases are assemblies published after 2022, such as P. rubens IBT 27055.
  • Reads are simulated, not sequenced. Real runs add contamination and uneven depth.
  • Azole-resistance alleles (such as TR34/L98H in A. fumigatus) and F. oxysporum effector genes are not covered yet.
  • A passed validation shows that the tools find what is known to be there and nothing that is not. Whether a strain may be released is your decision.

Run it yourself

Every strain is listed by its  accession in the evidence pack, so you can run the same genomes through FungFlow or any other tool and compare. To validate on your own strains or against your own acceptance criteria, use Contact; we fix your criteria before we look at the result. For the team, the code is in FungFlow/tests/reference/ (panel.tsv, _truth.py, score.py, run_validation.py); the raw results are in FungFlow/docs/validation.json and validation.md.

Troubleshooting

Part IX · Statistics in bioinformatics

Thinking statistically: variation, estimates and uncertainty

Every number a Bioinformaniac pipeline gives you, a Shannon index, a fold change, an ANI, a QV, is an estimate made from a sample. This part of the handbook teaches the statistics behind those numbers, so that you can read every result of MicroRes and the other tools the way a statistician would: what was estimated, how uncertain it is, and what it can and cannot support.

Why a biologist needs statistics

Biology varies. Two mice on the same diet carry different gut communities; two cultures of the same strain grow at slightly different rates; two libraries made from the same DNA give slightly different counts. Statistics is the discipline of drawing conclusions despite this variation: separating a difference that reflects the biology from one that the sampling alone could produce. Holmes and Huber put it plainly in the opening of Modern Statistics for Modern Biology: modern biology generates data whose variability is part of the signal, and the analysis has to model it rather than wish it away (Holmes & Huber, 2019, ch. 1).

In sequencing studies three layers of variation stack on top of each other. Biological variation is real difference between individuals, plants, cultures or time points: it is what we want to study. Technical variation comes from extraction, library preparation, sequencing depth and batches: it is noise, and sometimes bias. Sampling variation arises because a sequencer reads a random subset of the molecules in a library: a taxon at 0.01 % gives 2 reads in one library of 20,000 and 0 in another. Good design (chapter 2) controls the first two; the statistics in this part quantify all three.

Population, sample, parameter, estimate

The population is everything you want to make a statement about: all healthy adults, all batches of a yoghurt, all isolates of a lineage. A parameter is a number that describes it, such as the true mean relative abundance of Faecalibacterium in healthy adults. You never see the parameter. You see a sample, and compute an estimate, a statistic such as the sample mean. Writing μ (mu) for the parameter and x̄ ("x-bar") for the estimate keeps the two apart. Every downstream question, a test, an interval, a model, is about how far x̄ may lie from μ.

The sample has to represent the population for this to work. Twenty stool samples from one family represent that family, not "healthy adults". This is the single most common reason a statistically perfect analysis gives a wrong biological answer, and no method in this part repairs it (Whitlock & Schluter, The Analysis of Biological Data, ch. 1).

Distributions you will meet in our results

A distribution describes how likely each value is. Four families cover most of what the tools compute.

DistributionWhat it describesWhere you meet it
Normal (Gaussian)sums of many small effects; symmetric, mean μ, standard deviation σmeans, log fold changes, CLR values, regression errors
Binomialsuccesses out of n independent trials with probability p completeness, allele balance in a heterozygote, accuracy of a classifier
Poissoncounts of rare independent events, variance = meanreads covering a base, technical replicates of a library
Negative binomialcounts with extra (biological) variance, σ² = μ + μ²/θtaxa and gene counts across biological replicates (, )

Relative abundances, P values and proportions live on [0, 1]; counts are whole numbers ≥ 0; ANI sits near 100 %. Recognising the kind of number in front of you tells you which methods are allowed: a t-test on raw counts, or a Pearson correlation on proportions, ignores what the numbers are (chapters S4 and S7).

Spread, standard error and the central limit theorem

The standard deviation (SD, s) describes how much single observations vary. The standard error (SE) describes how much an estimate would vary if you repeated the whole study. For a mean of n independent observations:

SE(x̄) = s / √n

Four times more samples halves the uncertainty of a mean; it does not halve the spread of individuals. Confusing SD and SE is one of the commonest errors in figures: error bars must say which they show (Krzywinski & Altman, 2013). The central limit theorem says that means (and sums) of independent values are close to normally distributed even when single values are not, provided n is not tiny. Figure S1.1 shows it on a skewed variable typical of taxon abundances.

051015200.00.20.4valuedensitySingle values vs means of n = 100–0.5: 0.0340.5–1: 0.1791–1.5: 0.2511.5–2: 0.252–2.5: 0.2122.5–3: 0.1793–3.5: 0.1533.5–4: 0.1194–4.5: 0.1024.5–5: 0.0855–5.5: 0.0655.5–6: 0.0626–6.5: 0.0466.5–7: 0.0387–7.5: 0.0317.5–8: 0.0268–8.5: 0.028.5–9: 0.0199–9.5: 0.0159.5–10: 0.01310–10.5: 0.01410.5–11: 0.0111–11.5: 0.00711.5–12: 0.00712–12.5: 0.00712.5–13: 0.00513–13.5: 0.00513.5–14: 0.00514–14.5: 0.00414.5–15: 0.00415–15.5: 0.00415.5–16: 0.00316–16.5: 0.00216.5–17: 0.00217–17.5: 0.00217.5–18: 0.00218–18.5: 0.00118.5–19: 0.00119–19.5: 0.00119.5–20: 0.0011–1.5: 0.0011.5–2: 0.0352–2.5: 0.172.5–3: 0.3443–3.5: 0.4163.5–4: 0.3594–4.5: 0.2624.5–5: 0.1855–5.5: 0.1075.5–6: 0.0536–6.5: 0.0296.5–7: 0.0117–7.5: 0.0117.5–8: 0.0078–8.5: 0.0058.5–9: 0.0039–9.5: 0.0029.5–10: 0.00110–10.5: 0.00112–12.5: 0.001single values: skewedmeans of 10: close to normal024680.00.51.0median of a resampledensityBootstrap: 5,000 resamples of n = 150.8–1: 0.0021–1.2: 0.0221.2–1.4: 0.1141.6–1.8: 0.3321.8–2: 0.6352.2–2.4: 0.8712.4–2.6: 1.0423–3.2: 0.9053.2–3.4: 0.6443.4–3.6: 0.3023.6–3.8: 0.1064.4–4.6: 0.0227.8–8: 0.0032.5 % = 1.3297.5 % = 3.79sample median 2.49
Figure S1.1. Left: 20,000 values of a skewed (log-normal) variable and 4,000 means of samples of 10. Averages are far less skewed and far less spread: the standard error is σ/√n. Right: the percentile bootstrap gives a 95 % interval (1.32–3.79) for a median, where no simple formula exists. Simulated, seed 1.

Confidence intervals and the bootstrap

A 95 % confidence interval (CI) is a range built by a procedure that, in repeated studies, would contain the true parameter 95 % of the time. For a mean, x̄ ± t0.975, n−1·SE. It is not "a 95 % probability that μ is in this particular interval": μ is fixed, the interval is random. The practical reading is simpler: the interval shows which values of the parameter are compatible with your data, and its width shows how much the data can tell you. A narrow interval around a tiny effect is a precise "almost nothing"; a wide interval that spans zero is "we cannot tell", not "no effect".

The bootstrap builds intervals when no formula exists, for a median, a diversity index, a ratio of two estimates or an AUC (Efron, 1979; Efron & Hastie, 2016, ch. 10). Draw n samples with replacement from your n samples, recompute the statistic, repeat thousands of times; the spread of the recomputed values approximates the sampling distribution, and its 2.5th and 97.5th percentiles give a 95 % interval (right panel of Figure S1.1). The bootstrap resamples the independent units: whole subjects, not the reads inside a sample and not the repeated visits of one subject treated as separate people.

Check your understanding

You sequence one sample three times. What is n for comparing it with another person?

One. The replicates measure technical noise of that sample; they do not add independent biological information.

Why are the error bars of a mean narrower than the spread of the data points?

They show the standard error s/√n, the uncertainty of the mean, not the standard deviation of individuals.

Part IX · Statistics in bioinformatics

Hypothesis tests, P values and effect sizes

A test answers one narrow question: could chance alone, as defined by a null hypothesis, have produced a difference at least this large? This chapter explains what a P value is and is not, which test fits which data, and why every P value in a report should travel with an effect size and its uncertainty.

The logic of a test

Every test has the same four parts. A null hypothesis H₀ states "no difference" in a precise way, for example "the mean Shannon index is the same in both diets". A test statistic summarises the difference in your data, for example t = (x̄₁ − x̄₂)/SE. A null distribution says how that statistic would vary if H₀ were true. The P value is the probability, under H₀, of a statistic at least as extreme as the one observed.

A small P value says the data would be surprising if H₀ were true. It does not give the probability that H₀ is true, it does not measure the size or importance of an effect, and P > 0.05 does not show that there is no effect. These are the core points of the American Statistical Association's statement on P values (Wasserstein & Lazar, 2016). P values are also fickle: repeat a modest study and P can easily move from 0.01 to 0.3 (Halsey et al., 2015).

Two kinds of error, and power

H₀ true (no effect)H₀ false (real effect)
Test significantType I error (false positive), rate αcorrect: a discovery (rate = power)
Not significantcorrectType II error (false negative), rate β

Power = 1 − β is the probability of detecting an effect of a given size. It grows with the effect size, the sample size and α, and falls with the noise. Figure S2.1 computes it exactly for a two-group t-test: with α = 0.05, 80 % power needs about 64 samples per group for a medium effect (d = 0.5) but only 26 for a large one (d = 0.8). Microbiome effects are often small, which is why many published microbiome studies are underpowered (Kelly et al., 2015); RNA-seq experiments need at least six biological replicates per group to find most changes reliably (Schurch et al., 2016).

-4-202460.00.20.4measurement (in SD units)densityTwo groups, Cohen's d = 0.8controltreatedcontroltreatedoverlap 69%02040608010000.50.81samples per grouppowerPower of a t-test, α = 0.05d = 0.5d = 0.5: n = 64 per group for 80 % powerd = 0.5: n ≈ 64 per groupd = 0.8d = 0.8: n = 26 per group for 80 % powerd = 0.8: n ≈ 26 per groupd = 1.2d = 1.2: n = 12 per group for 80 % powerd = 1.2: n ≈ 12 per group80 %
Figure S2.1. Left: an effect of d = 0.8 (a "large" effect by Cohen's convention) still leaves the two distributions overlapping heavily, so single samples cannot be classified from it. Right: exact power (non-central t) of a two-sided t-test; the dots mark the group size needed for 80 % power.

Choosing a test

Data and questionParametric testRank-based / resampling
Two independent groups, a continuous measureWelch t-testWilcoxon rank-sum (Mann–Whitney U)
Paired measurements (before/after, same subject)paired t-testWilcoxon signed-rank
Three or more groupsone-way ANOVA (Welch)Kruskal–Wallis, then Dunn's test with adjustment
Two categorical variables (e.g. gene present × group)χ² testFisher's exact test
Any statistic, any design you can permutepermutation test: shuffle the labels, recompute, compare

The Welch t-test does not assume equal variances and should be the default two-group t-test. The Wilcoxon rank-sum test replaces values by ranks; it is robust to outliers and skew, and tests whether values in one group tend to be larger than in the other (it is not, strictly, a test of medians). Kruskal–Wallis generalises it to several groups and only says that some group differs; a post-hoc test (Dunn's) says which pairs, with a multiple-testing adjustment. A permutation test builds the null distribution by shuffling the group labels many times: it needs only that samples are exchangeable under H₀, and it is how PERMANOVA (chapter S5) and many network and machine-learning checks obtain their P values (Good, 2005; Holmes & Huber, 2019, ch. 6).

Effect sizes: how big, not only whether

With enough samples any trivial difference becomes "significant"; with too few, an important one does not. An effect size answers the question that matters to the biologist.

  • Difference of means in the original units, with its CI: "Shannon index 0.4 higher (95 % CI 0.1–0.7)".
  • Log₂ fold change for abundances and expression: +1 is a doubling, −1 a halving; it is the x-axis of every volcano plot.
  • Cohen's d = (x̄₁ − x̄₂)/spooled, a difference in SD units (0.2 small, 0.5 medium, 0.8 large, by Cohen's convention). Figure S2.1 shows that even d = 0.8 leaves the two groups overlapping by about 69 %.
  • Cliff's delta and the AUC are rank-based: the probability that a random value of one group exceeds a random value of the other (re-scaled for Cliff's delta to −1 … +1). They pair naturally with Wilcoxon.
  • R²: the share of variation explained, as PERMANOVA reports for a metadata variable.

The habit to build: report the estimate, its uncertainty and the P value together, and judge importance from the first two (Sullivan & Feinn, 2012; Gelman, Hill & Vehtari, 2020, ch. 4).

Bayesian reasoning in one paragraph

A P value conditions on H₀; a decision often needs the reverse, the probability that a finding is real. Bayes' rule gives it from the prior probability. A pathogen marker with 95 % sensitivity and 95 % specificity, used where only 1 % of samples truly carry the pathogen, gives a positive predictive value of 0.95 × 0.01 / (0.95 × 0.01 + 0.05 × 0.99) ≈ 16 %: most positives are false. The same logic explains why many "significant" findings from small exploratory studies do not replicate (Ioannidis, 2005), and it is the basis of genotype calling (chapter S8). Bayesian models make the prior explicit and give posterior intervals for the quantities of interest (Gelman et al., 2013).

Check your understanding

P = 0.32 for a difference in richness between two diets. Can you conclude the diets give the same richness?

No. Absence of evidence is not evidence of absence; look at the confidence interval of the difference to see which effects remain compatible with the data.

Why is Welch's t-test preferred over Student's?

It does not assume equal variances in the two groups, and loses almost nothing when they are equal.

Key references

  1. Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. The American Statistician 70, 129–133 (2016).
  2. Halsey LG, Curran-Everett D, Vowler SL, Drummond GB. The fickle P value generates irreproducible results. Nature Methods 12, 179–185 (2015).
  3. Krzywinski M, Altman N. Points of significance: Significance, P values and t-tests. Nature Methods 10, 1041–1042 (2013).
  4. Kelly BJ, Gross R, Bittinger K, et al. Power and sample-size estimation for microbiome studies using pairwise distances and PERMANOVA. Bioinformatics 31, 2461–2468 (2015).
  5. Schurch NJ, Schofield P, Gierliński M, et al. How many biological replicates are needed in an RNA-seq experiment and which differential expression tool should you use? RNA 22, 839–851 (2016).
  6. Sullivan GM, Feinn R. Using effect size, or why the P value is not enough. Journal of Graduate Medical Education 4, 279–282 (2012).
  7. Ioannidis JPA. Why most published research findings are false. PLoS Medicine 2, e124 (2005).
  8. Good PI. Permutation, Parametric, and Bootstrap Tests of Hypotheses, 3rd edition. Springer, 2005.
  9. Gelman A, Hill J, Vehtari A. Regression and Other Stories. Cambridge University Press, 2020. Read: ch. 4 (statistical inference), 6–11 (linear regression), 13–15 (logistic and generalised linear models).
  10. Gelman A, Carlin JB, Stern HS, Dunson DB, Vehtari A, Rubin DB. Bayesian Data Analysis, 3rd edition. CRC Press, 2013.
  11. Holmes S, Huber W. Modern Statistics for Modern Biology. Cambridge University Press, 2019 (free online edition). Read: ch. 1–4 (generative models, mixtures), 6 (testing), 8 (count data), 9 (multivariate methods).
  12. Whitlock MC, Schluter D. The Analysis of Biological Data, 3rd edition. Macmillan Learning / W. H. Freeman, 2020. Read: ch. 1–6 for samples, estimation and hypothesis testing in biology.
Part IX · Statistics in bioinformatics

Thousands of tests at once: multiple testing and the false discovery rate

A differential abundance analysis tests every taxon; an RNA-seq study tests every gene. With thousands of tests, a 5 % false-positive rate per test means hundreds of false discoveries. This chapter explains the adjustments every omics result relies on, and how to read a q value and a volcano plot.

The problem

Test 5,000 taxa that do not differ at all, each at α = 0.05, and about 250 will be "significant" by chance. The probability of at least one false positive among m independent tests is 1 − (1 − α)m, which is already 64 % for m = 20. Two kinds of control answer this, and they answer different questions.

The family-wise error rate (FWER) is the probability of making even one false discovery. Bonferroni controls it by testing each hypothesis at α/m; Holm's step-down version is uniformly more powerful and should replace it. FWER control is right when a single false claim is costly: a regulatory safety claim, a clinical variant, a small set of planned comparisons such as Dunn's pairwise tests.

The false discovery rate (FDR) is the expected proportion of false discoveries among the discoveries. Controlling it at 5 % accepts that about one in twenty listed taxa or genes may be a false lead, in exchange for far more power. For screening thousands of features it is the standard (Benjamini & Hochberg, 1995).

Benjamini–Hochberg, step by step

Sort the m P values: p(1) ≤ p(2) ≤ … ≤ p(m). Find the largest rank k with

p(k) ≤ (k / m) · q

and declare the k smallest P values significant. Equivalently, each test gets an adjusted P value (often called a q value in software): padj,(i) = min over j ≥ i of (m · p(j) / j), capped at 1; keep the features with padj ≤ 0.05. Storey's q value refines this by estimating π₀, the proportion of true nulls, from the flat part of the P-value histogram, and gains power when many features truly change (Storey & Tibshirani, 2003).

00.250.50.75104008001200P valuetests5,000 tests, 10 % real effects0–0.05: 6520.05–0.1: 2530.1–0.15: 2480.15–0.2: 2340.2–0.25: 2260.25–0.3: 2310.3–0.35: 2210.35–0.4: 2320.4–0.45: 2660.45–0.5: 2250.5–0.55: 2140.55–0.6: 2510.6–0.65: 2130.65–0.7: 2270.7–0.75: 2050.75–0.8: 2030.8–0.85: 2390.85–0.9: 2190.9–0.95: 2140.95–1: 227height if every null were truereal effects pile up near 003006009000.0000.0040.0080.012rank i of the P valueP valueBenjamini–Hochberg at q = 5 %BH line i·q/m244 discoveries9 of them false (3.7%)line i·q/m
Figure S3.1. Left: the P-value histogram of a typical omics screen: flat (uniform) for true nulls, a spike at zero for real effects. Right: Benjamini–Hochberg keeps every P value up to the last one below the line i·q/m. Here it keeps 244 tests, of which 9 (3.7%) are false: within the promised 5 %. Bonferroni (P ≤ 0.05/5,000) would keep only 45. Simulated, seed 3.

Reading the P-value histogram

Always look at it before trusting the list of hits (Holmes & Huber, 2019, ch. 6). A flat histogram with a spike at zero (Figure S3.1) is healthy. A histogram with no spike means no detectable signal. A hump in the middle, or a pile-up near 1, means the test's assumptions are violated (wrong model, discrete counts, too few samples for the permutations), and adjusted P values are then not trustworthy either.

Volcano plots

-6-4-202460481216log₂ fold change−log₁₀ PA volcano plot (2,500 taxa or genes)q = 0.05 (BH)higher in B (43)higher in A (39)big fold change, few reads: not significant
Figure S3.2. Each dot is one feature: effect size (x) against evidence (y). The dashed line is where the BH-adjusted q value crosses 0.05; it is not a fixed P value, because it depends on all the tests together. Features far left or right but low are large, noisy estimates: low counts give wide standard errors. Simulated, seed 4.

A volcano plot (Figure S3.2) places each feature by effect (log₂ fold change) and evidence (−log₁₀ P). The interesting features sit high and far from zero. Points far out but low are features with large but uncertain estimates, typically rare taxa with few reads: that is why  and RNA-realm report shrunken fold changes, which pull noisy estimates towards zero in proportion to their uncertainty and so rank genes more reliably (Love, Huber & Anders, 2014; Zhu, Ibrahim & Love, 2019).

Fewer, better tests

Every feature you test costs power. Removing features that could never be significant before testing, using a criterion that does not look at the groups, such as overall prevalence or mean abundance, raises power without biasing the FDR. This is independent filtering, done automatically by  and by the prevalence filter of MicroRes (Bourgon, Gentleman & Huber, 2010). Filtering by the observed difference between groups is not independent and invalidates the adjustment.

Check your understanding

200 genes have q < 0.05. How many are expected to be false?

About 10 (5 % of 200) on average; the FDR is a rate over the list, not a property of each gene.

When would you prefer Holm to BH?

When even one false positive is costly and the number of tests is small, e.g. a handful of planned comparisons or a safety claim.

Part IX · Statistics in bioinformatics

Counts, overdispersion and compositional data

Sequencing gives counts, and the counts of one sample add up to a total set by the machine, not by the biology. These two facts, overdispersed counts and compositions, explain why microbiome and RNA-seq data need their own statistics, and why MicroRes asks you to normalise and transform before anything else.

From molecules to counts

If a library were sequenced again and again, the count of one taxon would vary like a Poisson variable: variance equal to the mean. Across biological replicates the variance is much larger, because the taxon's true abundance itself differs between individuals. The negative binomial (NB) captures this with a dispersion parameter: σ² = μ + αμ² (α = 1/θ). For abundant features the αμ² term dominates (Figure S4.1, left).  and  model each feature as NB and share information across features to estimate α reliably from few replicates (Robinson, McCarthy & Smyth, 2010; Love, Huber & Anders, 2014). Holmes and Huber derive the NB as a Poisson whose rate is itself gamma-distributed, a "mixture" that is the natural model of heterogeneous biology (Holmes & Huber, 2019, ch. 4 and 8).

110100100010000110010^410^610^8mean countvarianceVariance grows faster than the meanPoisson: variance = meanNB: μ + μ²/θPoisson: σ² = μnegative binomial:σ² = μ + μ²/θ050100share of the sample (%)Closure: only taxon A grewA: 20 %A 20%B: 30 %B 30%C: 50 %C 50%beforeA: 50 %A 50%B: 19 %B 19%C: 31 %C 31%after (A × 4)
Figure S4.1. Left: 160 simulated features, 12 replicates each. Counts that follow a Poisson law have variance = mean (blue); biological replicates follow a negative binomial with extra variance μ²/θ (orange, θ = 4), which is what ,  and many microbiome models assume. Right: B and C did not change in absolute amount, yet their shares fell from 30 % to 19 % and from 50 % to 31 %, because shares must add to 100 %.

Zeros

Most entries of a microbiome table are zero. A zero can mean the taxon is absent (biological zero), present but not sampled at this depth (sampling zero), or removed by a technical step. Sampling zeros are expected from the NB itself at low means; models that add a separate "zero inflation" component are needed only when there are more zeros than the count model predicts (Silverman et al., 2020). Practically: never read a zero as proof of absence, filter features that are zero almost everywhere (they cannot be tested), and be careful with any method that takes a logarithm (log 0 is undefined, so a pseudocount is added, and its size changes the results for rare taxa).

Compositional data

The total number of reads per sample is fixed by the sequencing run, so the counts only carry relative information: the data are compositions (Aitchison, 1982; Gloor et al., 2017). If one taxon blooms, the shares of all others must fall even if their absolute amounts did not change (Figure S4.1, right). Three consequences follow. Correlations between shares are biased, often negative (chapter S7). A taxon "decreasing" in relative abundance may not have decreased at all. And differences in total microbial load, which can be large between people or treatments, are invisible unless measured separately (qPCR, flow cytometry, spike-ins) (Vandeputte et al., 2017).

The remedy is to work with log-ratios, which do not depend on the total. The centred log-ratio (CLR) divides each count by the geometric mean of the sample:

clr(x)i = log( xi / g(x) ), g(x) = (x1·x2·…·xD)1/D

CLR values are centred on zero within each sample: positive means more abundant than the sample's typical taxon. The Euclidean distance between CLR vectors is the Aitchison distance, the natural distance between compositions. A difference in CLR between groups is a difference relative to the typical taxon, which is the honest statement compositional data allow.

Normalisations in MicroRes step 04

MethodIdeaGood forWatch out
TSS (proportions)divide by the sample totalcomposition plots, Bray–Curtisdominated by abundant taxa; compositional
Rarefyingsubsample every sample to the same depthrichness and other depth-sensitive diversitythrows data away; random; not for differential abundance (McMurdie & Holmes, 2014)
CSSscale by the sum up to a data-driven quantilesparse marker-gene datadesigned for metagenomeSeq (Paulson et al., 2013)
TMMtrimmed mean of M-values against a reference sampleRNA-seq-like countsassumes most features do not change (Robinson & Oshlack, 2010)
Median of ratios ()size factor = median ratio to a pseudo-referenceNB modelsneeds features present in all samples; struggles with very sparse tables
VST / rlogvariance-stabilising transformation of NB countsordination, clustering, heatmapsnot for testing; testing uses the model
CLRlog-ratio to the geometric meanAitchison distance, correlations, ML, CLR-based testspseudocount choice affects rare taxa

No normalisation is right for everything, and benchmarks show that the choice changes results, especially for differential abundance (Weiss et al., 2017). MicroRes's Compare all transformations panel shows how each behaves on your data; the methods paragraph of the report records which one you used, because a reader cannot judge a result without it.

Check your understanding

Why is variance larger than the mean across biological replicates?

The true abundance itself differs between replicates, adding biological variance on top of sampling (Poisson) variance: overdispersion.

What does a CLR value of −2 mean?

The taxon is e² ≈ 7.4 times less abundant than the geometric mean of that sample's taxa (natural log).

Key references

  1. Aitchison J. The statistical analysis of compositional data. Journal of the Royal Statistical Society B 44, 139–177 (1982).
  2. Gloor GB, Macklaim JM, Pawlowsky-Glahn V, Egozcue JJ. Microbiome datasets are compositional: and this is not optional. Frontiers in Microbiology 8, 2224 (2017).
  3. Love MI, Huber W, Anders S. Moderated estimation of fold change and dispersion for RNA-seq data with . Genome Biology 15, 550 (2014).
  4. Robinson MD, McCarthy DJ, Smyth GK. : a Bioconductor package for differential expression analysis of digital gene expression data. Bioinformatics 26, 139–140 (2010).
  5. Robinson MD, Oshlack A. A scaling normalization method for differential expression analysis of RNA-seq data. Genome Biology 11, R25 (2010).
  6. Paulson JN, Stine OC, Bravo HC, Pop M. Differential abundance analysis for microbial marker-gene surveys. Nature Methods 10, 1200–1202 (2013).
  7. McMurdie PJ, Holmes S. Waste not, want not: why rarefying microbiome data is inadmissible. PLoS Computational Biology 10, e1003531 (2014).
  8. Weiss S, Xu ZZ, Peddada S, et al. Normalization and microbial differential abundance strategies depend upon data characteristics. Microbiome 5, 27 (2017).
  9. Vandeputte D, Kathagen G, D'hoe K, et al. Quantitative microbiome profiling links gut community variation to microbial load. Nature 551, 507–511 (2017).
  10. Silverman JD, Roche K, Mukherjee S, David LA. Naught all zeros in sequence count data are the same. Computational and Structural Biotechnology Journal 18, 2789–2798 (2020).
  11. Holmes S, Huber W. Modern Statistics for Modern Biology. Cambridge University Press, 2019 (free online edition). Read: ch. 1–4 (generative models, mixtures), 6 (testing), 8 (count data), 9 (multivariate methods).
  12. Xia Y, Sun J, Chen D-G. Statistical Analysis of Microbiome Data with R. Springer, 2018. Read: ch. 3 (overview of statistics), 5–7 (compositional data, diversity), 9–10 (models for count data).
Part IX · Statistics in bioinformatics

The statistics of diversity: alpha, beta and PERMANOVA

Diversity indices are estimates, distances are modelling choices, and an ordination is a projection. This chapter treats the ecological results of MicroRes as statistics: what each number estimates, how depth biases it, and how PERMANOVA and PERMDISP test differences between communities.

Alpha diversity is an estimate

Observed richness counts the taxa seen, and the number seen grows with the number of reads (Figure S5.1, left). Comparing raw richness between samples of different depth compares sequencing effort as much as biology. Two remedies exist: compare at equal effort (rarefying, or reading the rarefaction curves at a common depth) or estimate the true richness including unseen taxa. Chao1 uses the taxa seen once (F₁, singletons) and twice (F₂, doubletons):

ŜChao1 = Sobs + F₁(F₁ − 1) / (2(F₂ + 1))

Many singletons relative to doubletons mean many taxa remain unseen (Chao, 1984). With denoised amplicon data singletons are often removed by the denoiser, which makes Chao1 unreliable: report it only when singletons are genuine. Willis argues that diversity should be estimated with a standard error and compared with models that use it, rather than computed from rarefied tables (Willis, 2019).

Shannon (H = −Σ pi ln pi) and Simpson (Σ pi²) weight abundant taxa more and depend less on depth. Hill numbers put them on one scale, the effective number of equally abundant taxa: q = 0 is richness, q = 1 is eH, q = 2 the inverse Simpson (Hill, 1973; Chao et al., 2014). A sample with Shannon H = 2.3 has the diversity of e2.3 ≈ 10 equally common taxa: a number a reader can picture.

05k10k15k20k0100200300400reads sampledtaxa observedRarefaction curvesrich community: 374 taxa in 20,000 readsrich (374)medium community: 210 taxa in 20,000 readsmedium (210)poor community: 84 taxa in 20,000 readspoor (84)rarefy to 5,000?-0.400.4-0.400.4PCoA 1 (31 %)PCoA 2 (12 %)Ordination of 50 samplesgroup A sample 1group A sample 2group A sample 3group A sample 4group A sample 5group A sample 6group A sample 7group A sample 8group A sample 9group A sample 10group A sample 11group A sample 12group A sample 13group A sample 14group A sample 15group A sample 16group A sample 17group A sample 18group A sample 19group A sample 20group A sample 21group A sample 22group A sample 23group A sample 24group A sample 25group B sample 1group B sample 2group B sample 3group B sample 4group B sample 5group B sample 6group B sample 7group B sample 8group B sample 9group B sample 10group B sample 11group B sample 12group B sample 13group B sample 14group B sample 15group B sample 16group B sample 17group B sample 18group B sample 19group B sample 20group B sample 21group B sample 22group B sample 23group B sample 24group B sample 25A: tightB: spread out
Figure S5.1. Left: expected richness when subsampling a sample (computed exactly, hypergeometric). Curves still rising mean more reads would find more taxa; comparing richness is only fair at the same depth, or with an estimator that models the unseen. Right: PERMANOVA tests whether the centroids (crosses) differ; PERMDISP tests whether the spread (dashed circles) differs. Here B is more dispersed, so a significant PERMANOVA alone could reflect spread rather than a shift. Simulated, seed 6.

Testing alpha diversity

An index per sample becomes an ordinary variable: compare two groups with Wilcoxon or a t-test, several with Kruskal–Wallis and Dunn (chapter S2), and adjust for covariates or repeated measures with a linear or mixed model (chapter S6). Diversity is often skewed and sometimes depends on depth even after rarefying; including log(depth) as a covariate is a simple check.

Beta diversity: the distance decides the question

DistanceUsesSensitive to
Jaccardpresence/absencerare taxa, membership
Bray–Curtisrelative abundancesdominant taxa
Unweighted UniFracpresence/absence + treerare lineages, deep differences (Lozupone & Knight, 2005)
Weighted UniFracabundances + treeabundant lineages
AitchisonCLR valuesratios; compositionally coherent

Different distances answer different questions, so a difference seen with Jaccard but not with Bray–Curtis says the groups differ in which rare taxa they carry rather than in their dominant members. Choose the distance for the question before testing, or report all and interpret the pattern.

Ordination

PCoA (principal coordinates) places samples in a few dimensions that best preserve the chosen distances, by an eigen-decomposition of the double-centred distance matrix; with Euclidean distances on CLR values it equals PCA. The percentage on each axis is the share of the total variation that axis shows. Two axes showing 31 % and 12 % (Figure S5.1, right) leave more than half the variation out of the picture, so a lack of visible separation is not evidence of no difference. NMDS preserves only the rank order of distances; its stress (below about 0.2) says how faithful the picture is. Ordinations are for looking; tests are for deciding (Legendre & Legendre, 2012, ch. 9; Holmes & Huber, 2019, ch. 9).

PERMANOVA and PERMDISP

PERMANOVA partitions the sum of squared distances into a part between groups and a part within, exactly as ANOVA does with variances, giving a pseudo-F statistic and R², the share of variation explained by the variable. Its P value comes from permuting the group labels (Anderson, 2001). Because it uses distances, it works with any measure above. Covariates enter in order (sequential) or by margin; repeated measures need restricted permutations within subject.

PERMANOVA can be significant because group centroids differ or because groups differ in spread. PERMDISP tests the second: it compares the distances of samples to their group centroid (Anderson, 2006). Read them together: PERMANOVA significant and PERMDISP not → a shift in composition; both significant → the shift may be partly or wholly a difference in heterogeneity, which is itself a finding (dysbiotic communities are often more variable) but a different one (Anderson & Walsh, 2013).

Check your understanding

Sample A has 200 observed taxa at 50,000 reads, sample B 150 at 8,000 reads. Is A richer?

Not necessarily; at equal depth B may be as rich or richer. Compare at a common depth or with an estimator that accounts for unseen taxa.

PERMANOVA P = 0.002 and PERMDISP P = 0.001. What can you say?

The groups differ, but at least partly in dispersion; a shift in average composition is not established.

Key references

  1. Chao A. Nonparametric estimation of the number of classes in a population. Scandinavian Journal of Statistics 11, 265–270 (1984).
  2. Willis AD. Rarefaction, alpha diversity, and statistics. Frontiers in Microbiology 10, 2407 (2019).
  3. Hill MO. Diversity and evenness: a unifying notation and its consequences. Ecology 54, 427–432 (1973).
  4. Chao A, Gotelli NJ, Hsieh TC, et al. Rarefaction and extrapolation with Hill numbers: a framework for sampling and estimation in species diversity studies. Ecological Monographs 84, 45–67 (2014).
  5. Lozupone C, Knight R. UniFrac: a new phylogenetic method for comparing microbial communities. Applied and Environmental Microbiology 71, 8228–8235 (2005).
  6. Anderson MJ. A new method for non-parametric multivariate analysis of variance. Austral Ecology 26, 32–46 (2001).
  7. Anderson MJ. Distance-based tests for homogeneity of multivariate dispersions. Biometrics 62, 245–253 (2006).
  8. Anderson MJ, Walsh DCI. PERMANOVA, ANOSIM, and the Mantel test in the face of heterogeneous dispersions: what null hypothesis are you testing? Ecological Monographs 83, 557–574 (2013).
  9. Legendre P, Legendre L. Numerical Ecology, 3rd English edition. Elsevier, 2012. Read: ch. 7 (resemblance coefficients) and ch. 9 (ordination).
  10. Holmes S, Huber W. Modern Statistics for Modern Biology. Cambridge University Press, 2019 (free online edition). Read: ch. 1–4 (generative models, mixtures), 6 (testing), 8 (count data), 9 (multivariate methods).
  11. Xia Y, Sun J. Bioinformatic and Statistical Analysis of Microbiome Data: From Raw Sequences to Advanced Modeling with QIIME 2 and R. Springer, 2023. Read: the chapters on alpha and beta diversity, differential abundance and compositional methods.
Part IX · Statistics in bioinformatics

Models, confounding and the differential abundance methods

A two-group test answers a two-group question. Real studies have ages, batches, antibiotics, repeated visits and several variables of interest at once. Regression models handle them, and every differential abundance and differential expression method in our tools is a regression model underneath.

The linear model

A linear model writes the outcome as a sum of effects plus noise:

y = β₀ + β₁·diet + β₂·age + β₃·batch + ε

Each coefficient β is the expected change in y for a one-unit change in that variable with the others held fixed. That last phrase is the whole point: the model separates effects that travel together in the data. Group comparisons are linear models with indicator variables (the t-test and ANOVA are special cases). Assumptions to check: independent errors, roughly constant variance, and residuals without strong patterns (Gelman, Hill & Vehtari, 2020, ch. 6–11).

A generalised linear model keeps the linear predictor but links it to a mean that suits the data: a log link and a negative binomial distribution for counts, a logit link and a binomial for presence/absence or cases/controls. Sequencing depth enters as an offset (log size factor), so the model compares rates, not raw counts.  and  are NB GLMs fitted to every feature (McCarthy, Chen & Smyth, 2012; Love, Huber & Anders, 2014).

Confounding

0123456782468dose of a diet fibre (g/day)log abundanceOne trend overall, the opposite trend within each batchbatch 1: 2.9 g, 3.74batch 1: 3.7 g, 3.62batch 1: 3.3 g, 3.66batch 1: 1.7 g, 4.71batch 1: 1.9 g, 5.23batch 1: 3.6 g, 3.54batch 1: 1.0 g, 5.38batch 1: 3.5 g, 4.23batch 1: 3.4 g, 3.76batch 1: 2.4 g, 4.52batch 1: 1.9 g, 4.89batch 1: 1.8 g, 4.92batch 1: 1.8 g, 4.51batch 1: 2.3 g, 4.63batch 1: 2.5 g, 4.97batch 1: 2.7 g, 3.86batch 1: 4.0 g, 3.91batch 1: 3.4 g, 4.01batch 1: 2.9 g, 4.06batch 1: 4.0 g, 4.32batch 1: 1.6 g, 5.28batch 1: 1.5 g, 4.69batch 1: 2.8 g, 4.32batch 1: 1.1 g, 5.52batch 1: 1.1 g, 5.27batch 1: 2.5 g, 4.71batch 1: 2.4 g, 4.54batch 1: 3.8 g, 3.98batch 1: 2.9 g, 4.77batch 1: 2.5 g, 4.24batch 2: 5.9 g, 5.94batch 2: 6.0 g, 6.12batch 2: 4.5 g, 6.71batch 2: 5.3 g, 6.68batch 2: 4.7 g, 6.67batch 2: 5.2 g, 6.58batch 2: 4.3 g, 6.47batch 2: 6.9 g, 5.48batch 2: 4.6 g, 6.12batch 2: 6.0 g, 5.18batch 2: 4.9 g, 6.45batch 2: 6.6 g, 5.21batch 2: 6.0 g, 5.97batch 2: 4.4 g, 7.65batch 2: 6.5 g, 5.29batch 2: 6.8 g, 5.18batch 2: 6.7 g, 5.54batch 2: 5.7 g, 6.25batch 2: 4.4 g, 6.78batch 2: 4.6 g, 6.68batch 2: 6.8 g, 5.68batch 2: 5.7 g, 6.29batch 2: 4.5 g, 6.41batch 2: 6.7 g, 5.48batch 2: 5.9 g, 5.96batch 2: 5.7 g, 5.71batch 2: 5.1 g, 6.51batch 2: 5.2 g, 6.06batch 2: 4.7 g, 7.01batch 2: 4.1 g, 7.10ignoring the batchpooled slope +0.28 (wrong sign)slope adjusted for batch -0.58batch 1batch 2
Figure S6.1. Simpson's paradox. Within each sequencing batch the taxon falls with the dose (slope ≈ −0.6), but batch 2 both received higher doses and reads higher overall, so the pooled line rises. Adding batch to the model (abundance ~ dose + batch) recovers the true slope. Simulated, seed 7.

A confounder is associated with both the variable of interest and the outcome. If cases were sequenced in one batch and controls in another, batch and disease are inseparable, and no model can fix it afterwards: that is a design failure (chapter 2). If they are only partly entangled, including the confounder as a covariate recovers the effect of interest (Figure S6.1). Antibiotics, proton-pump inhibitors, metformin, age, BMI, transit time and sequencing batch are frequent confounders in microbiome studies (Leek et al., 2010; Vujkovic-Cvijin et al., 2020). Batch-correction methods such as  remove batch means and variances when batches are balanced with the biology (Johnson, Li & Rabinovic, 2007); with unbalanced designs, model batch as a covariate instead of "correcting" the data first.

Repeated measures and mixed models

Ten visits of one person are not ten people (chapter S1). A mixed model adds a random effect for subject, a subject-specific intercept drawn from a distribution, so that within-person correlation is modelled and the effective sample size is honest. The same applies to cages of mice (mice in one cage share microbes), fields, fermenters and families. In formula notation: abundance ~ diet + time + (1 | subject). MaAsLin2 and LinDA fit such models for microbiome features (Mallick et al., 2021; Zhou et al., 2022); for distances, PERMANOVA with permutations restricted within subject does the equivalent.

The differential abundance methods, side by side

MethodModelWhat it estimatesTypical behaviour
NB GLM with size factors, Wald test, shrinkagelog₂ fold change of normalised countspowerful; FDR can inflate with very sparse, compositional data (Love et al., 2014)
ANCOM-BClog-linear model with a bias term for sample-specific sampling fractionslog fold change of absolute abundance, up to a constantgood FDR control; corrects compositional bias (Lin & Peddada, 2020)
ALDEx2Dirichlet Monte-Carlo instances → CLR → Welch/Wilcoxon, averageddifference in CLR (relative to the geometric mean)conservative; robust to sparsity (Fernandes et al., 2014)
LinDAlinear (mixed) model on CLR with a bias correctionlog fold change, compositionally correctedfast, allows covariates and random effects (Zhou et al., 2022)
Wilcoxon on CLRrank test per featurea shift in CLR distributionsimple, assumption-light; no covariates

On 38 datasets, popular methods identified very different sets of significant taxa from the same data (Nearing et al., 2022). Disagreement is not a bug in one method; it reflects different targets (counts, CLR, absolute fold change) and different assumptions. A taxon found by most methods is a robust finding; one found by a single method needs caution. That is exactly what the consensus heatmap of MicroRes shows.

LEfSe combines Kruskal–Wallis and Wilcoxon tests with a linear discriminant analysis effect size to rank biomarkers (Segata et al., 2011). It is useful for description, but it does not adjust for multiple testing in the usual way and does not model covariates, so treat its list as exploratory.

Check your understanding

All cases were sequenced in run 1 and all controls in run 2. Can a covariate for run fix this?

No. Run and disease status are perfectly confounded; their effects cannot be separated by any model.

Why can  and ALDEx2 disagree on the same taxon?

They estimate different quantities (normalised count fold change vs CLR difference) under different assumptions; compositional effects and sparsity hit them differently.

Key references

  1. Gelman A, Hill J, Vehtari A. Regression and Other Stories. Cambridge University Press, 2020. Read: ch. 4 (statistical inference), 6–11 (linear regression), 13–15 (logistic and generalised linear models).
  2. McCarthy DJ, Chen Y, Smyth GK. Differential expression analysis of multifactor RNA-Seq experiments with respect to biological variation. Nucleic Acids Research 40, 4288–4297 (2012).
  3. Love MI, Huber W, Anders S. Moderated estimation of fold change and dispersion for RNA-seq data with . Genome Biology 15, 550 (2014).
  4. Lin H, Peddada SD. Analysis of compositions of microbiomes with bias correction. Nature Communications 11, 3514 (2020).
  5. Fernandes AD, Reid JN, Macklaim JM, et al. Unifying the analysis of high-throughput sequencing datasets: characterizing RNA-seq, 16S rRNA gene sequencing and selective growth experiments by compositional data analysis. Microbiome 2, 15 (2014).
  6. Zhou H, He K, Chen J, Zhang X. LinDA: linear models for differential abundance analysis of microbiome compositional data. Genome Biology 23, 95 (2022).
  7. Mallick H, Rahnavard A, McIver LJ, et al. Multivariable association discovery in population-scale meta-omics studies. PLoS Computational Biology 17, e1009442 (2021).
  8. Nearing JT, Douglas GM, Hayes MG, et al. Microbiome differential abundance methods produce different results across 38 datasets. Nature Communications 13, 342 (2022).
  9. Segata N, Izard J, Waldron L, et al. Metagenomic biomarker discovery and explanation. Genome Biology 12, R60 (2011).
  10. Leek JT, Scharpf RB, Bravo HC, et al. Tackling the widespread and critical impact of batch effects in high-throughput data. Nature Reviews Genetics 11, 733–739 (2010).
  11. Johnson WE, Li C, Rabinovic A. Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics 8, 118–127 (2007).
  12. Vujkovic-Cvijin I, Sklar J, Jiang L, et al. Host variables confound gut microbiota studies of human disease. Nature 587, 448–454 (2020).
  13. Holmes S, Huber W. Modern Statistics for Modern Biology. Cambridge University Press, 2019 (free online edition). Read: ch. 1–4 (generative models, mixtures), 6 (testing), 8 (count data), 9 (multivariate methods).
  14. Xia Y, Sun J. Bioinformatic and Statistical Analysis of Microbiome Data: From Raw Sequences to Advanced Modeling with QIIME 2 and R. Springer, 2023. Read: the chapters on alpha and beta diversity, differential abundance and compositional methods.
Part IX · Statistics in bioinformatics

Correlation, networks and prediction

Which taxa go together? Which features predict a disease? Correlation and prediction are where compositional data and small samples cause the most convincing false results. This chapter explains the methods behind MicroRes's Associations, Co-occurrence networks and Biomarkers & prediction steps, and how to tell a robust result from an artefact.

Correlation

Pearson's r measures linear association and is sensitive to outliers; Spearman's ρ is Pearson on ranks and measures any monotonic association. Both range from −1 to +1. Correlation is not causation, and in microbiome data it is often not even association between amounts: two taxa can correlate because both respond to pH, because both are divided by the same total, or because of a batch (Altman & Krzywinski, 2015).

0.00.10.20.30.00.10.20.3share of taxon Ashare of taxon BIndependent taxa, correlated sharesabsolute: ρ = -0.02shares: ρ = +0.3200.5100.51false-positive rate (1 − specificity)true-positive rateROC curve of a classifierAUC = 0.78chance (AUC 0.5)AUC = 0.78chance
Figure S7.1. Left: taxa A and B were simulated independently (absolute Spearman ρ ≈ 0), but both are divided by the same total, which here is dominated by a third, highly variable taxon; their shares become correlated. This is why MicroRes builds networks with SparCC, SPIEC-EASI or Spearman on CLR values, never on raw shares. Right: a ROC curve plots sensitivity against 1 − specificity for every threshold of a classifier's score; the area under it (AUC) is the probability that a random case scores higher than a random control. Simulated, seed 8.

The left panel of Figure S7.1 shows the compositional trap: taxa simulated independently become correlated once divided by a shared, variable total. Pearson described this "spurious correlation" of ratios in 1897; Aitchison's log-ratio framework is the cure (Aitchison, 1982; Lovell et al., 2015). SparCC estimates correlations of the underlying log-abundances from the variances of log-ratios, assuming the network is sparse (Friedman & Alm, 2012). SPIEC-EASI goes further: from CLR data it estimates a sparse inverse covariance (graphical lasso or neighbourhood selection), so an edge means conditional dependence, two taxa associated after accounting for all others, and the sparsity is chosen by stability selection (Kurtz et al., 2015). Spearman on CLR values is a simpler, compositionally aware option.

Network edges are hypotheses about interaction, not evidence of it. With n samples and p taxa, there are p(p−1)/2 possible edges: 4,950 for 100 taxa. Edges need multiple-testing control or stability selection, and a network inferred from 30 samples is fragile; MicroRes reports the number of samples and the method next to every network.

Prediction: a different question from testing

Testing asks whether a feature differs; prediction asks whether a set of features can classify a new sample. Many features that differ significantly predict poorly (Figure S2.1: large effects still overlap), and a model can predict well from features that are individually unremarkable.

  • LASSO (logistic regression with an L1 penalty) shrinks most coefficients exactly to zero, giving a short list of features (Tibshirani, 1996).
  • Elastic net mixes L1 and L2 penalties and keeps groups of correlated features together, which suits taxa that co-occur (Zou & Hastie, 2005).
  • Random forest averages many decision trees on bootstrap samples; it captures interactions and non-linear effects and ranks features by importance (Breiman, 2001).

Overfitting and cross-validation

1234567891011120.00.10.20.30.4model complexity (polynomial degree)mean squared errorTraining error keeps falling; error on new data does noterror on the training dataerror on new databest on new data: degree 3training datanew data (test)
Figure S7.2. Overfitting. With 30 samples, ever more flexible models fit the training samples better and better, but beyond a point they fit the noise and predict new samples worse. Cross-validation estimates the orange curve from the data you have, which is how MicroRes chooses the penalty of LASSO and elastic net. Simulated, seed 9.

With more features than samples, the usual case in omics, a flexible model can fit the training data perfectly and still predict nothing (Figure S7.2). k-fold cross-validation estimates performance on new data: split the samples into k parts, train on k − 1, test on the held-out part, repeat, average. The penalty of LASSO or elastic net is chosen by an inner cross-validation; the performance must then be measured by an outer loop that never saw that choice (nested cross-validation), or the estimate is optimistic (Varma & Simon, 2006; James et al., 2021, ch. 5).

Data leakage is the classic error: selecting the "significant" taxa on all samples, then cross-validating a model on those taxa. The test folds influenced the selection, so the AUC is inflated, sometimes from 0.5 to 0.9 on pure noise. Every step that looks at the outcome, including feature selection, normalisation fitted to the data and parameter tuning, must happen inside the training folds.

Reading classification performance

Sensitivity is the share of true cases called positive; specificity the share of true controls called negative. Changing the threshold trades one for the other; the ROC curve draws all trade-offs, and the AUC (area under it) is the probability that a random case scores higher than a random control: 0.5 is chance, 1 perfect (Figure S7.1, right). AUC ignores prevalence; when cases are rare, also report positive predictive value (chapter S2) or the precision–recall curve. A model is only as general as its samples: an AUC from one cohort often falls in another, so external validation is the real test (Pasolli et al., 2016).

Check your understanding

Two taxa correlate at ρ = −0.6 in relative abundance. Do they compete?

Not necessarily. Compositional closure, a shared driver or a batch can produce the correlation; check with SparCC/SPIEC-EASI and design experiments to test interaction.

Why must feature selection happen inside cross-validation?

Otherwise the held-out samples influenced which features were chosen, and the performance estimate is biased upwards (leakage).

Key references

  1. Altman N, Krzywinski M. Points of significance: Association, correlation and causation. Nature Methods 12, 899–900 (2015).
  2. Aitchison J. The statistical analysis of compositional data. Journal of the Royal Statistical Society B 44, 139–177 (1982).
  3. Lovell D, Pawlowsky-Glahn V, Egozcue JJ, Marguerat S, Bähler J. Proportionality: a valid alternative to correlation for relative data. PLoS Computational Biology 11, e1004075 (2015).
  4. Friedman J, Alm EJ. Inferring correlation networks from genomic survey data. PLoS Computational Biology 8, e1002687 (2012).
  5. Kurtz ZD, Müller CL, Miraldi ER, et al. Sparse and compositionally robust inference of microbial ecological networks. PLoS Computational Biology 11, e1004226 (2015).
  6. Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society B 58, 267–288 (1996).
  7. Zou H, Hastie T. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society B 67, 301–320 (2005).
  8. Breiman L. Random forests. Machine Learning 45, 5–32 (2001).
  9. Varma S, Simon R. Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics 7, 91 (2006).
  10. James G, Witten D, Hastie T, Tibshirani R. An Introduction to Statistical Learning with Applications in R, 2nd edition. Springer, 2021. Read: ch. 4 (classification), 5 (cross-validation), 6 (lasso and ridge), 8 (trees and forests).
  11. Pasolli E, Truong DT, Malik F, Waldron L, Segata N. Machine learning meta-analysis of large metagenomic datasets: tools and biological insights. PLoS Computational Biology 12, e1004977 (2016).
  12. Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning, 2nd edition. Springer, 2009.
Part IX · Statistics in bioinformatics

Statistics inside genome results: quality, coverage, identity and trees

Genome tools look deterministic, but nearly every number they print is a probability or an estimate: a Phred score, a QV, a coverage depth, an E-value, an ANI, a  percentage, a genotype quality, a bootstrap support. Knowing the statistics behind them is what lets you read BactFlow, Bactomaniac, FungFlow, RNA-realm, Humaniac and Insectra results with confidence.

Phred scores and QV

A Phred quality is a log-scaled error probability: Q = −10 log₁₀ P, so Q20 is 1 error in 100 bases, Q30 1 in 1,000, Q40 1 in 10,000 (Figure S8.1, left) (Ewing & Green, 1998). The same scale describes a whole assembly: a consensus QV of 40 means about one error per 10,000 bases, so about 500 errors in a 5 Mb bacterial genome; QV 50 brings that to 50.  estimates QV without a reference, from the k-mers of the assembly that do not occur in accurate reads: with Kasm-only such k-mers among Ktotal, the per-base error rate is E = 1 − (1 − Kasm-only/Ktotal)1/k and QV = −10 log₁₀ E (Rhie et al., 2020). In the FungFlow validation the simulated-read assemblies reached QV 45–58 against their known truth.

Coverage is (nearly) Poisson

0102030405010^-510^-410^-310^-210^-11Phred quality Qerror probabilityQ = −10·log₁₀(P)P = 10^(−Q/10)Q10: 1 error in 10 basesQ10: 1 in 10Q20: 1 error in 100 basesQ20: 1 in 100Q30: 1 error in 1,000 basesQ30: 1 in 1,000Q40: 1 error in 10,000 basesQ40: 1 in 10,0000153045600.000.050.10reads covering one baseprobabilityDepth of a 30× run (Poisson)2041
Figure S8.1. Left: a Phred score is a log-scaled error probability; every 10 points is ten times fewer errors. Right: under the Lander–Waterman model the number of reads covering a base is Poisson with mean c (the average coverage): at 30× 95 % of bases lie between 20× and 41× (dashed lines), and the share of the genome missed is e^(−c), about 9e-14. Real coverage is more variable (GC bias, repeats), so pipelines report it rather than assume it.

If reads start at random positions, the number covering a base follows a Poisson distribution with mean c, the average coverage (Lander & Waterman, 1988). The fraction of the genome with zero coverage is then e−c: 5 % at 3×, 0.005 % at 10×. Real libraries deviate (GC bias, repeats, amplification), so tools report the observed distribution; a heterozygous site in a diploid needs enough reads on each allele, which is why variant callers ask for roughly 15–30× and why Humaniac reports the share of the genome above 20×.

Similarity scores: E-values and ANI

A  E-value is the number of alignments with at least this score expected by chance in a database of this size: E = K·m·n·e−λS, where S is the score and m, n are the query and database lengths (Karlin & Altschul, 1990). It grows with database size, so the same hit has a larger E in a bigger database. For small values E ≈ P; E = 1e−50 is overwhelming, E = 0.1 is noise.

768085909510005001000ANI between two genomes (%)genome pairsPairs within a species and between species (schematic)76–76.5: 4576.5–77: 4177–77.5: 5177.5–78: 5278–78.5: 6178.5–79: 8079–79.5: 7479.5–80: 9380–80.5: 12280.5–81: 12781–81.5: 13881.5–82: 14382–82.5: 18482.5–83: 20883–83.5: 18583.5–84: 21084–84.5: 23784.5–85: 25185–85.5: 24985.5–86: 25286–86.5: 23786.5–87: 25987–87.5: 24987.5–88: 27488–88.5: 23188.5–89: 24389–89.5: 20689.5–90: 19090–90.5: 21190.5–91: 16691–91.5: 16091.5–92: 15092–92.5: 9592.5–93: 9793–93.5: 6393.5–94: 6094–94.5: 3394.5–95: 3495–95.5: 2295.5–96: 1896–96.5: 2296.5–97: 4097–97.5: 8897.5–98: 19698–98.5: 35698.5–99: 63199–99.5: 96699.5–100: 68795 % ANIsame species →different speciesthe ANI gap
Figure S8.2. Schematic of what large surveys found (Jain et al. 2018): ANI values between genomes form two groups with a gap around 83–95 %, so 95 % works as a species boundary. Simulated to show the shape, not real data: a call of 94.6 % is a borderline case to report with its alignment fraction, not a clean 'different species'.

Average nucleotide identity (ANI) between two genomes clusters into within-species and between-species groups, with a gap that makes 95 % a practical species boundary (Goris et al., 2007; Jain et al., 2018). It is an estimate over the aligned fraction (AF), so an ANI of 99 % over 20 % of the genome is not the same evidence as 99 % over 90 %;  requires both an ANI and an alignment-fraction threshold. Values near the boundary (94–96 %) deserve the word "borderline", not a confident name. The Bactomaniac validation found that the remaining identity "errors" were nomenclature conflicts of exactly this kind ( placing B. cereus AH187 in B. paranthracis), not wrong measurements.

Completeness as a proportion

 completeness is the proportion of a set of near-universal single-copy genes found complete; with the 758 genes of fungi_odb10, 97.0 % is 735 of 758, and the binomial standard error √(p(1−p)/n) ≈ 0.6 points. Differences of a few tenths of a percent between assemblies are within that noise; differences of several points are not. A high duplicated share signals uncollapsed haplotypes or contamination rather than better quality (Simão et al., 2015).

Genotypes are probabilities

A variant caller computes, for each possible genotype G, the likelihood of the observed reads, P(reads | G), from base qualities and mapping qualities, and combines it with a prior by Bayes' rule (Li, 2011). In a VCF, QUAL is the Phred-scaled probability that the site is not variant, and GQ the Phred-scaled probability that the reported genotype is wrong: GQ 20 means a 1 % chance. For a true heterozygote the alternative-allele fraction is binomial around 0.5; a "heterozygous" call at 10 % alternative reads in a haploid bacterium or fungus points to contamination, a mixed culture or a collapsed repeat, not to heterozygosity. Clinical interpretation adds population frequencies and evidence categories on top (Richards et al., 2015).

Trees and their support

A phylogeny is an estimate. Bootstrap support resamples alignment columns, rebuilds the tree and counts how often a clade reappears (Felsenstein, 1985); it measures the stability of the clade given the data, not the probability that it is true. Standard bootstrap values of 70 % or more are usually read as good support; 's ultrafast bootstrap is less biased and is usually taken as strong from 95 % (Minh, Nguyen & von Haeseler, 2013; Hoang et al., 2018).

Gene sets and pathways

Over-representation analysis tests whether a gene set is more frequent among the significant genes than expected, with a hypergeometric (Fisher) test; its answer depends entirely on the background (all genes tested, not the whole genome). Gene-set enrichment analysis () uses the full ranked list and avoids an arbitrary cut-off (Subramanian et al., 2005); competitive tests that account for correlation between genes, such as camera, keep the error rate honest (Wu & Smyth, 2012). Both need multiple-testing adjustment over the sets tested.

Validation figures are estimates too

"100 % sensitivity" on 18 expected resistance genes is an estimate with an interval: the exact (Clopper–Pearson) 95 % lower bound for 18 of 18 is 81 %. Report the number of cases behind each figure, fix the pass marks before seeing the results, and separate errors of the pipeline from errors of the truth set. Bioinformaniac's validation pages follow these rules; they are described in the customer validation report.

Check your understanding

A hit has E = 0.5. Is it a homologue?

No evidence of one: a hit that good is expected about once by chance in this search.

What fraction of a genome is expected to be uncovered at 5× average coverage?

e⁻⁵ ≈ 0.7 % under the Poisson model; more in practice because of bias.

Key references

  1. Ewing B, Green P. Base-calling of automated sequencer traces using phred. II. Error probabilities. Genome Research 8, 186–194 (1998).
  2. Rhie A, Walenz BP, Koren S, Phillippy AM. : reference-free quality, completeness, and phasing assessment for genome assemblies. Genome Biology 21, 245 (2020).
  3. Lander ES, Waterman MS. Genomic mapping by fingerprinting random clones: a mathematical analysis. Genomics 2, 231–239 (1988).
  4. Karlin S, Altschul SF. Methods for assessing the statistical significance of molecular sequence features by using general scoring schemes. PNAS 87, 2264–2268 (1990).
  5. Goris J, Konstantinidis KT, Klappenbach JA, et al. DNA–DNA hybridization values and their relationship to whole-genome sequence similarities. International Journal of Systematic and Evolutionary Microbiology 57, 81–91 (2007).
  6. Jain C, Rodriguez-R LM, Phillippy AM, Konstantinidis KT, Aluru S. High throughput ANI analysis of 90K prokaryotic genomes reveals clear species boundaries. Nature Communications 9, 5114 (2018).
  7. Simão FA, Waterhouse RM, Ioannidis P, Kriventseva EV, Zdobnov EM. : assessing genome assembly and annotation completeness with single-copy orthologs. Bioinformatics 31, 3210–3212 (2015).
  8. Li H. A statistical framework for SNP calling, mutation discovery, association mapping and population genetical parameter estimation from sequencing data. Bioinformatics 27, 2987–2993 (2011).
  9. Richards S, Aziz N, Bale S, et al. Standards and guidelines for the interpretation of sequence variants. Genetics in Medicine 17, 405–424 (2015).
  10. Felsenstein J. Confidence limits on phylogenies: an approach using the bootstrap. Evolution 39, 783–791 (1985).
  11. Minh BQ, Nguyen MAT, von Haeseler A. Ultrafast approximation for phylogenetic bootstrap. Molecular Biology and Evolution 30, 1188–1195 (2013).
  12. Hoang DT, Chernomor O, von Haeseler A, Minh BQ, Vinh LS. UFBoot2: improving the ultrafast bootstrap approximation. Molecular Biology and Evolution 35, 518–522 (2018).
  13. Subramanian A, Tamayo P, Mootha VK, et al. Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. PNAS 102, 15545–15550 (2005).
  14. Wu D, Smyth GK. Camera: a competitive gene set test accounting for inter-gene correlation. Nucleic Acids Research 40, e133 (2012).
Part IX · Statistics in bioinformatics

Reading MicroRes and every Bioinformaniac report like a statistician

The previous chapters gave the tools; this one applies them. Step by step, it lists what each MicroRes result estimates, which statistics produced it, what to check before believing it, and how to write it up. The same checklist closes for the genome and RNA tools.

MicroRes, step by step

StepWhat it estimatesStatisticsCheck before believingChapter
02 Quality controlwhich features and samples carry informationprevalence and abundance filters (independent of groups)the funnel: share of reads kept; no group lost more samples than anotherS3, 6
03 Contaminantsreagent contaminantsdecontam frequency/prevalence modelsnegative controls present and sequenced; low-biomass samples flagged6
04 Normalise & transformcomparable values across samplesTSS, rarefying, CSS, TMM, median-of-ratios, VST, CLR"Compare all transformations"; the choice stated in the methodsS4
05 Compositionrelative abundance of top taxadescriptive (shares)shares are compositional; do not read a fall as a lossS4
06 Alpha diversityrichness/evenness per sampleChao1, Shannon, Simpson, Hill; Wilcoxon, Kruskal–Wallis + Dunnequal depth or rarefaction curves flat; effect size with PS5, S2
07 Beta diversitydifferences in community compositiondistances, PCoA/NMDS, PERMANOVA (R², P), PERMDISPPERMDISP not significant before calling a shift; distance namedS5
Differential abundancewhich taxa differ, by how much, ANCOM-BC, ALDEx2, LinDA, Wilcoxon-CLR; BHconsensus across methods; P-value histogram; covariates includedS3, S6
Associationstaxa related to continuous metadatarank correlation (CLR), BHoutliers, confounders (age, BMI, medication)S7
Networksconditional co-occurrenceSPIEC-EASI, SparCC, Spearman-CLRsamples ≫ taxa; stability; edges as hypothesesS7
Biomarkers & predictioncan features classify new samples?elastic net, LASSO, random forest; cross-validated AUCselection inside folds; spread of AUC across folds; external cohortS7
14 Pangenomecore and accessory taxa or functionsprevalence thresholds; accumulation curvesthresholds stated; sample size per group comparableS5
R1 Reference rangeswhere a sample sits in a reference populationquantiles of the reference cohortthe reference was measured the same way (region, pipeline)S1

From results to one story

A good analysis reads as one argument. Quality control shows the data are fit to use. Normalisation is stated. Alpha and beta diversity describe how much and how the communities differ overall, with effect sizes (Δ Shannon, PERMANOVA R²) and the dispersion check. Differential abundance names the taxa behind the difference, with fold changes, adjusted P values and agreement between methods. Associations and networks suggest mechanisms, flagged as hypotheses. Prediction says whether the difference is strong enough to classify individuals. Every number travels with its uncertainty and its n. MicroRes's narrative report is written in this order for that reason, and its methods paragraph lists every choice a reviewer needs.

Three questions keep the story honest: Could the design explain it? (batch, confounders, pseudoreplication; chapters 2 and S6). Could the composition explain it? (closure, load; S4). Could chance explain it? (multiple testing, power; S2–S3). An effect that survives all three is worth reporting with confidence.

The other tools, read the same way

  • BactFlow and Bactomaniac. Read quality (Phred), coverage (Poisson), assembly QV and  completeness are estimates with error (S8); species calls are ANI with alignment fraction against a threshold; resistance, toxin and virulence genes are calls with identity and coverage cut-offs. A call just under a threshold is a borderline call to confirm, not a negative.
  • FungFlow.  completeness is a binomial proportion; QV from ; species from ANI ( containment) and the ITS barcode; biosynthetic clusters are predictions whose boundaries can be truncated (the validation found the long aflatoxin and sterigmatocystin clusters detected and named, but only 60 % covered by the predicted region).
  • RNA-realm.  NB GLMs, shrunken log₂ fold changes, BH; VST for PCA;  and over-representation with a stated background (S3, S4, S6, S8).
  • Humaniac and Insectra. Genotype likelihoods, GQ and depth (S8); population structure by PCA; associations by regression with ancestry components as covariates (S6).

A reporting checklist

  1. Sample size per group, and what the independent unit is.
  2. Every filter, normalisation and transformation, with parameters.
  3. For every test: the test, the effect size with a 95 % CI, the P value and the adjustment (method and m).
  4. For distances: the distance, the ordination, PERMANOVA R² and P with permutations, PERMDISP.
  5. For differential abundance: the methods, covariates and the agreement between methods.
  6. For prediction: the model, the cross-validation scheme (nested), AUC with its spread, and any external validation.
  7. Software versions and databases: MicroRes's report and every pipeline's methods paragraph write these for you.

These points match the STORMS checklist for microbiome studies, which journals increasingly ask for (Mirzayi et al., 2021).

Check your understanding

A taxon is significant in  only (1 of 5 methods). How do you report it?

As a method-dependent, tentative finding; the robust findings are those most methods agree on.

What three questions should every surprising result face?

Could the design, the composition, or chance explain it?

Key references

  1. Mirzayi C, Renson A, Genomic Standards Consortium, et al. Reporting guidelines for human microbiome research: the STORMS checklist. Nature Medicine 27, 1885–1892 (2021).
  2. Nearing JT, Douglas GM, Hayes MG, et al. Microbiome differential abundance methods produce different results across 38 datasets. Nature Communications 13, 342 (2022).
  3. Holmes S, Huber W. Modern Statistics for Modern Biology. Cambridge University Press, 2019 (free online edition). Read: ch. 1–4 (generative models, mixtures), 6 (testing), 8 (count data), 9 (multivariate methods).
  4. Xia Y, Sun J. Bioinformatic and Statistical Analysis of Microbiome Data: From Raw Sequences to Advanced Modeling with QIIME 2 and R. Springer, 2023. Read: the chapters on alpha and beta diversity, differential abundance and compositional methods.
  5. Xia Y, Sun J, Chen D-G. Statistical Analysis of Microbiome Data with R. Springer, 2018. Read: ch. 3 (overview of statistics), 5–7 (compositional data, diversity), 9–10 (models for count data).
  6. Davis NM, Proctor DM, Holmes SP, Relman DA, Callahan BJ. Simple statistical identification and removal of contaminant sequences in marker-gene and metagenomics data. Microbiome 6, 226 (2018).
  7. Knight R, Vrbanac A, Taylor BC, et al. Best practices for analysing microbiomes. Nature Reviews Microbiology 16, 410–422 (2018).
  8. Gelman A, Hill J, Vehtari A. Regression and Other Stories. Cambridge University Press, 2020. Read: ch. 4 (statistical inference), 6–11 (linear regression), 13–15 (logistic and generalised linear models).
Launching MicroRes
Launching BactFlow
Launching RNA-realm
Launching FungFlow
Launching Humaniac
Launching Insectra
Launching Bactomaniac
// Your account

My data

Everything you uploaded and every result the tools wrote. Review it, download it, or delete it.

Storage…

MicroRes

Datasets you loaded and your saved analysis

Reports

Every report you open is kept here with its own report ID.

Files and results

Reads, genomes and results of BactFlow, FungFlow, RNA-realm, Humaniac, Insectra and Bactomaniac. Download a file, or a whole folder as a .zip.
// Your account

Account

Your name and picture show at the top of every page.

// Account recovery

New password

Checking your link…

// Contact

Get in touch

Questions, bug reports, collaborations or a dataset that does not load: send a message and you get an answer by e-mail.

Channel

// Error 404

Signal lost

That page does not exist.