Truth from outside
Fungal clusters are located with from , the curated database of characterised clusters. Bacterial genes are confirmed against the submitters' own annotation before they count.
Bioinformaniac is where your sequencing data becomes answers, and where you learn the science behind them along the way. Seven expert pipelines run on our servers, from microbiomes and RNA-seq to bacterial, fungal, insect and human genomes. A 53-chapter handbook and a game you can play explain every step, every statistic and every figure they produce. No installing, no code, no black box.
Pick the data you have. Each tool opens with an example study, so you can see the results before you upload anything.
You get diversity, differential abundance, networks, biomarkers and a written methods section.
You get quality control, differential expression, pathways and co-expression modules, from any organism.

You get a checked, annotated genome with taxonomy, , plasmids, variants and genome maps.
You get an assembled, repeat-masked and annotated genome with , CAZymes and gene clusters.
You get variants and structural variants on GRCh38, ranked by ACMG evidence and phenotype, with pharmacogenomics, families and tumour–normal.
You get a chromosome-level genome with its mitogenome and genes, or the structure, diversity and resistance variants of a population.
You get four verdicts — quality, identity, safety and stability — with every resistance gene placed on its plasmid or chromosome and rated for transfer risk, in an EFSA-mapped report.
Want every method and setting? Compare the seven tools in detail.
No setup, defensible methods, our servers, private files, everything downloadable, and you learn as you go.
No software to install, no servers or environments to manage. Open the page and start.
Established tools (, ANCOM-BC, , , …) with sensible defaults, plus a written methods section with references.
Heavy jobs are queued and run on our servers. Close the tab: the job keeps going, and the results wait in your account.
Your files live in your own space. Nobody else sees them, and you can download or delete everything at any time.
Figures, tables, genomes, reports, or a whole results folder as one .zip.
Each step explains what it does. The 53-chapter handbook covers every method and the statistics behind it, and the Learning Lab turns it into a game.
Try or sign up, upload your data, get your results.
Explore the example studies as a guest. A free account takes your name, e-mail and a password.
Drag in tables or reads. Large files are sent in pieces, so a dropped connection does not lose your upload.
Run step by step or start a pipeline and come back later. Download figures, tables and reports.
Guests and free accounts pay nothing; full access to every tool on request.
Microbiome, bacterial, RNA-seq and fungal analyses, each with its handbook chapter.
Coding, privacy, papers, storage and how long a genome takes.
No. Everything runs from the browser. Each step explains its settings in plain words, and the defaults follow current practice.
Yes. Your uploads and results sit in a folder only your account can open. You can download everything or delete your account and all its files from the Account page.
Yes. MicroRes, RNA-realm, FungFlow, Humaniac and Insectra write a methods section that names every tool, version and setting, with references. Please cite the underlying tools as listed.
You can use all of it. When it is full, new uploads and runs stop with a clear message; delete old files in My data or ask us for more space.
A bacterial genome usually takes under an hour, a fungal genome a few hours. Jobs run on our servers, so you can close the page and come back.
Write to us from the Contact page, or from Messages (the envelope at the top) when you are signed in. The answer arrives in the same conversation.
Free account, no card. Explore real results today, and learn the science while you are at it.
All six come with your account. Steps marked demo are open to guests.
A guided pipeline for microbiome data with two modules, taxonomic analysis and functional metagenomics. Every step writes a methods paragraph with references into your report.

A graphical workflow for bacterial isolate genomes. Upload reads to your private folder, choose the steps you need and follow each run live.
Bulk RNA-seq analysis following current Bioconductor practice, without writing R. Every step adds a methods paragraph with references.
A guided pipeline for fungal isolate genomes. Upload reads to your private folder, choose the stages and follow the run live; results open in an interactive report.
Human genomics on GRCh38, from reads to an interpreted genome. Six modules run alone or chained, each step its own job; results open in an interactive report with an IGV genome viewer, and a PDF report with methods and the run log.
Insect genomics: one insect to a chromosome-level, annotated genome, or many insects to the structure and selection of their populations. Five modules, run alone or chained, each step its own job.
The bacterial strain safety dossier: is this strain what you think it is, is its genome good enough, does it carry resistance, toxin or virulence genes and could they spread, and is it still the same strain? Four verdicts with their evidence, laid out along the EFSA requirements for whole-genome analysis of microorganisms in the food chain.
We run our pipelines on public genomes with a documented biology: strains that make toxins, strains that are safe, strains that carry resistance on a plasmid. Every answer is scored against a truth that does not come from our own software, against criteria we wrote down before the first result existed.
Loading the validation…
Fungal clusters are located with from , the curated database of characterised clusters. Bacterial genes are confirmed against the submitters' own annotation before they count.
The pass marks were fixed before the first run and are never adjusted afterwards. A miss is shown as a miss.
Toxin producers and pathogens test sensitivity. Safe strains test the opposite error: condemning something harmless.
Accessions, panels, every measured result and the software versions are in the evidence pack, so you can repeat any of it.
Yes. The evidence pack lists every strain by its public accession and every expected result. Run those genomes through our tools, or through your own, and compare.
Yes. Send us the reference strains or your acceptance criteria through Contact. We run them and give you the same kind of report, with your criteria fixed before we look at the result.
Whenever a release changes the analysis. The date of the last run and the software versions are shown above.
No. It shows that the tools find what is known to be there, and nothing that is not. Whether a strain may be released is your decision; this is evidence for it.
During the free alpha every account is free. You can explore every tool with real example results, use MicroRes on your own tables, and play every Learning Lab topic. To run a tool on your own data, ask for access from inside the tool: we open it for 7-hour windows while we budget the server load. Full capacity for every account is coming soon.
Runs are only limited by your free storage and the CPUs and memory of your account. Jobs wait in a queue when the servers are busy and start as soon as there is room.
| Included | Free account | On request |
|---|---|---|
| MicroRes, RNA-realm, BactFlow, FungFlow, Humaniac and Insectra | All steps | All steps |
| Runs (genomes, read jobs, pipelines) | No limit | No limit |
| CPUs and RAM per analysis | 4 CPUs · 8 GB | Up to 12 CPUs · 28 GB |
| Storage | 5 GB | Up to 1 TB |
| Help | Messages from your account | Messages, plus help with methods |
As many as you like. A run needs free storage for its results, and it uses at most the CPUs and memory of your account. When the servers are busy, your job waits in the queue and starts on its own.
Click Send a request above, or write from Messages (the envelope at the top). Say what you analyse and what you need; we answer in the same conversation and raise the limits of your account.
The Free account stays. Paid plans with more resources will come later; you will hear from us before anything changes, and nothing is charged without your consent.
On our server in the EU. Your files are private to your account and are never used to train models or shared. See the Privacy Policy.
Yes. Send a request for a lab or institution and tell us how many people need access.
Last updated 22 November 2026.
These terms govern the use of the Bioinformaniac website and analysis tools, provided by Bioinformaniac, a Danish company (CVR 40945024), A. F. Beyersvej 6 st. tv., Copenhagen, Denmark; e-mail [email protected], telephone +45 71 59 13 12. They apply to everyone who creates an account or uses the service: individuals, companies, research institutions and other organisations. Where a person signs up for an organisation, that person confirms they are authorised to bind it, and the organisation is the customer. A written agreement signed by Bioinformaniac (for example a master services agreement or a data processing agreement) takes precedence over these terms where the two differ.
Bioinformaniac provides web tools for bioinformatics analysis (MicroRes, BactFlow, RNA-realm, FungFlow, Humaniac, Insectra, Bactomaniac), a methods handbook, a learning game and an AI assistant. Results are produced by published methods and open-source software, the versions of which are recorded in each report. The service is for research, development and education. It is not a medical device and must not be used for diagnosis or treatment decisions. Unless agreed in writing, it has not been qualified or validated for use under GxP, GMP, ISO 17025, IVD or other regulated quality systems, and results must be checked by a qualified person before they support a regulatory submission, a product release, a safety or quality decision, a publication or a report to a third party. The validation page shows how the pipelines performed on public reference data; it describes measured performance and is not a guarantee for your samples.
Keep your password safe; the customer is responsible for everything that happens under its accounts. One account per person; accounts are not shared. You must be at least 16 years old. Some features are open to everyone on demonstration data. Which features an account can use beyond that, including uploading your own data, starting runs and downloading results or reports, depends on the access Bioinformaniac has given the account. Access can be granted for a limited time, limited to certain tools, and changed or ended when the period or purpose ends. A free account may be limited to viewing results.
The customer keeps all rights to the data it uploads and to the results it creates from them. Bioinformaniac receives only the permission needed to store and process them to provide the service, to keep it secure and to correct faults. Your data are not used to train AI models, are not sold, and are not shared except with the service providers listed in the Privacy Policy and as the law requires. Bioinformaniac treats uploaded data, results and reports as confidential and gives access to them only to staff who need it to run and support the service, or as you ask. The customer confirms it has the right to upload the data, that doing so breaks no agreement or law, and that the data contain no information identifying a person unless it has a lawful basis and a data processing agreement is in place (it is available on request; human sequence data always needs one).
The tools combine Bioinformaniac's own code with third-party open-source software and public reference databases, each under its own licence or terms of use. Some databases and tools restrict commercial use, redistribution or the use of their output. The customer is responsible for checking that its intended use of results, and of anything it downloads from a report, is allowed by those terms. The software names, versions and references in each report are there to help with this.
The platform, its design, text, handbook, figures, validation material and Bioinformaniac's own code remain the property of Bioinformaniac or its licensors. The customer receives a non-exclusive, non-transferable right to use the service and to use the reports and results it creates for its own purposes, including in publications, regulatory documents and products. It may not copy, scrape, reverse engineer, resell or provide the service to others, or use it to build a competing product.
Do not attack, overload or probe the service, bypass its limits or access controls, share accounts, mine cryptocurrency or upload unlawful or malicious material. Do not use the service for activities prohibited by export control or sanctions law. Access may be suspended at once for conduct that endangers the service or other users, and otherwise after notice.
The AI assistant and the learning game explain methods and results. Answers can be incomplete or wrong and are not scientific, medical, legal or regulatory advice. Check them against the handbook and the cited literature. What is sent to the AI provider when you use the assistant is described in the Privacy Policy.
Plans, limits and prices are shown on the Pricing page or agreed in an offer. Larger allowances, company and institution plans are given on request. Subscriptions renew each month or year until cancelled; cancellation takes effect at the end of the paid period. Card payments are handled by Stripe; business customers can pay by invoice on the terms shown on the invoice. Prices exclude VAT, which is added where required. Bioinformaniac may change prices for a new period with at least 30 days' notice. Consumers in the EU have a 14-day right of withdrawal from a new subscription; by starting to use paid features in that period they agree that the service begins immediately and that the refund is reduced for the part already used.
The service is run on a best-effort basis, with maintenance and updates from time to time. No uptime or response time is promised unless the plan or a written agreement says so. Keep your own copies of important data and download your results.
The service is provided as is. To the extent the law allows, Bioinformaniac gives no warranty that results are free of error or fit for a particular purpose, and is not liable for indirect or consequential loss, loss of profit, loss of data, or the cost of decisions taken on the basis of results. Total liability for all claims in a year is limited to the fees paid for the service in the 12 months before the claim, or, for a free account, to nil where the law permits. Nothing limits liability for intent or gross negligence, for death or personal injury, or any other liability that cannot be limited by law, and consumers keep their statutory rights.
You can delete your account at any time on the Account page; your files are deleted with it, and backups roll over as described in the Privacy Policy. Bioinformaniac may end the service or an account with 60 days' notice and will let you download your data first, and may end access at once for a serious breach of these terms. Sections that by their nature continue (data and results, intellectual property, liability, law) survive the end.
These terms may be updated, for example when features or plans change. A material change is announced on the site or by e-mail at least 30 days before it applies to existing customers. Continued use after that date means the new terms are accepted; a customer who does not accept them can end the account before then.
These terms follow Danish law. Disputes with business customers are decided by the courts of Copenhagen, Denmark (Københavns Byret as the first instance), unless the parties agree otherwise in writing. Consumers keep the rights they have under the mandatory law of the country where they live. Questions about these terms: [email protected].
Last updated 22 November 2026.
Bioinformaniac (CVR 40945024), A. F. Beyersvej 6 st. tv., Copenhagen, Denmark, is responsible for your personal data. Questions or requests: [email protected] or the Contact page.
To provide the service you signed up for (contract), to keep it secure and fix faults (legitimate interest), and to keep accounting records (legal obligation).
We use only cookies needed to keep you signed in and to remember a guest demo session. No advertising or tracking cookies, so no cookie banner is needed.
Stripe (payments), Cloudflare (network and security in front of the site, and its Turnstile check that a sign-up is made by a person and not a robot) and, if set up, our e-mail provider for password links. If you use our AI (the ✦ Ask our AI button), your question, the earlier messages of that conversation, what is on your screen and the parts of your own results it reads to answer it are sent to Anthropic (Claude API), which may process them outside the EU; Anthropic does not use this data to train its models. Nothing is sent to Anthropic unless you ask the assistant something. Nobody else. We do not sell data and do not use your data to train AI models.
Data is stored on our server in the EU. Account data is kept while your account exists; when you delete your account, your files are deleted at once and backups roll over within 14 days. Billing records are kept for 5 years as Danish bookkeeping law requires.
You can see, correct, download (Account → Download my data) and delete (Account → Delete account) your data at any time, object to processing, and complain to the Danish Data Protection Agency (Datatilsynet).
Write to the Bioinformaniac team: a question, a problem, or a request for more storage, CPUs or memory. Answers arrive below.
Every analysis in this handbook turns DNA sequence into a biological statement. Knowing where each step sits in that chain makes it easier to choose methods, and to notice when a result is coming from the pipeline rather than from the biology.
Most microbial sequencing projects ask one of two questions. The first is who is there and what are they doing in a mixed community: a stool sample, a soil core, a fermenting food. This is the domain of microbiome analysis. It uses either marker-gene (amplicon) sequencing, usually of the 16S rRNA gene for bacteria and archaea or ITS and 18S for fungi and other eukaryotes, or shotgun metagenomics, where all DNA in the sample is sequenced. The second question is what is this organism: the complete genome of a single bacterial isolate, its taxonomy, its genes, its plasmids and how it differs from its relatives. This is isolate genomics.
A third question has become routine in industry and in clinical research: which viruses are present, and is the sample safe. Viral metagenomics asks it of a stool sample, a wastewater sample, a fermentation or a phage preparation, and it needs its own methods, because viruses have no universal marker gene, their genomes are often incomplete in an assembly, and a virus that infects a bacterium can carry toxin or resistance genes into it. Two safety questions cut across all three fields: is a resistance gene present (antimicrobial resistance, AMR) and is it where it can spread.
The fields share tools and ideas, such as quality control of reads, reference databases and the need for careful controls, but their statistics are very different. A microbiome study compares many samples and has to deal with sparse, compositional count tables. A genome project usually deals with a handful of genomes, and the hard part is getting each sequence right.
For a microbiome study the chain runs: study design and sampling, DNA extraction, library preparation and sequencing, read quality control, feature construction (amplicon sequence variants or taxonomic profiles), taxonomy and function assignment, filtering and contaminant removal, normalisation and transformation, then the ecological and statistical questions: diversity, community differences, differentially abundant features, associations, networks and prediction. Chapters 2 to 13 follow this chain. For a virome the same chain holds, with two additions: a check that each viral genome is complete enough and really present (not just a few reads on one gene), and a link from each virus to its host and lifestyle. For resistance the chain adds a screen of the functional profile or of the assembled genes against a curated list of resistance determinants, followed by a verdict that says how far down it can be trusted: drug class, mechanism, pathway and gene.
For an isolate genome it runs: culture and DNA extraction, sequencing with short reads, long reads or both, read quality control, assembly, polishing, quality assessment, taxonomic placement, annotation, then targeted questions such as typing, plasmids, resistance genes and comparison with reference strains. Chapters 14 to 21 follow that chain.
Each chapter explains the reasoning behind a step, the main methods and the trade-offs between them, with references to the papers that introduced or tested them. Where the field has not settled on one answer, for example on rarefaction or on which differential abundance test to trust, the chapter says so and describes what careful authors do instead of pretending there is a single correct choice. Boxes marked In practice give concrete recommendations; boxes marked Pitfall describe mistakes that are common in published work.
The handbook and the tools follow the same chains, so you can read a chapter and then run its step. Bioinformaniac has eight tools; each answers one of the questions above.
| Tool | Question | What is in it | Chapters |
|---|---|---|---|
| MicroRes · taxonomic module | Who is there in a mixed community? | Steps 01 to 14: import, quality control, contaminants, normalisation, composition, alpha and beta diversity, phylogeny, core microbiome, differential abundance, associations, networks, biomarkers, pangenome. | 4 to 13 |
| MicroRes · functional module | What can the community do, and does it carry resistance? | Steps F1 to F8: import (HUMAnN, PICRUSt2, ), cleaning, profiles, functional beta diversity, differential functions, who carries each function, the species-function atlas, and the antimicrobial resistance screen with a verdict for each sample. | 10 to 13 |
| MicroRes · viral module | Which viruses are present, whom do they infect, are they safe? | Steps V1 to V7: import, genome quality and presence, composition, hosts and virus-host links, lifestyle and safety screen (toxin, resistance and lysogeny genes), a study report and a clinical report. | V1 to V3 |
| BactFlow | What is this isolate? | Quality control, assembly and post-assembly: typing, plasmids, resistance, comparison. | 14 to 21 |
| Bactomaniac | Is this bacterial strain safe to use? | A strain safety dossier: identity, resistance with its genomic context, toxins, quality and stability. | 14 to 21 |
| RNA-realm | Which genes change between conditions? | RNA-seq from counts or reads to differential expression, pathways and co-expression, with covariates and a random effect for repeated measures. | 24 to 29 |
| FungFlow, Humaniac, Insectra | What is this fungal, human or insect genome? | Assembly, annotation and the genome browser, with biosynthetic clusters for fungi. | 30 to 35 |
You can read the handbook on its own, but every chapter ends with a short note on where the step lives in the tools.
A result that cannot be explained or repeated is not an industrial result. Every tool therefore ends in a report, and MicroRes and RNA-realm write theirs as a story: chapters in a fixed order, the findings first, each figure with a sentence that says what it shows, the cautions next to the findings they qualify, and a methods chapter that lists every setting, every software version and the unique ID of the report. Because the text is produced from the measured results by fixed rules, a number in the report is always a number from your data. The same layout is available as a web page, a PDF and a Markdown file.
Reveal the code is the second half of the promise. Each tool can show, for the analysis you just ran, the exact code or command that produced it, as a script that you can read, run on your own machine and keep with the paper or the dossier. Reading it is also the fastest way to learn what a step really does. Reveal the code is available to administrators and to accounts with full access; in the free release the button is visible but dimmed.
The same care applies to the analysis itself. When your samples are not independent (the same person sampled before and after a treatment, animals in the same cage), say so: MicroRes and RNA-realm let you name the covariates that must be adjusted for and a random effect for the repeated subject, and the report records the choice.
Each chapter starts with what you will learn, explains the idea before the tool, works through at least one small example with real numbers, lists the common mistakes and ends with questions to test yourself. Book references in the text and in the reference list name the chapter to read next: the docs are a map, the books are the territory. A sensible order for a newcomer is Part I (foundations), then the part for your data type, then Part IV on practice. The statistics are best learned from Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R and Xia & Sun, Bioinformatic and Statistical Analysis of Microbiome Data for microbiomes, Efron & Hastie, Computer Age Statistical Inference for large-scale testing, and James et al., An Introduction to Statistical Learning for machine learning; the sequence side from Ismail, Bioinformatics: A Practical Guide to NGS Data Analysis, Low & Tammi, Practical Bioinformatics for Beginners and Christensen (ed.), Introduction to Bioinformatics in Microbiology.
Because counts are relative, noisy and shaped by every step from extraction to database. A conclusion is only as good as the weakest step of the chain.
An isolate project reconstructs one organism completely (assembly, annotation); a microbiome project measures many organisms partially and compares samples statistically.
The most important decisions in a microbiome study are made before any DNA is extracted. Design determines which questions can be answered at all; controls determine whether the answer can be trusted.
The unit of replication is the thing that was independently assigned to a treatment or sampled from a population: a person, an animal, a pen, a field plot. Several samples from the same unit (repeated time points, technical replicates, several gut sections) are not independent and must be modelled as such, usually with a random effect or by averaging. Treating them as independent inflates the apparent sample size and produces false positives, a problem known as pseudoreplication.
Microbial communities vary a lot between individuals, so effects that are clear within an individual can be small compared with the differences between individuals. Longitudinal designs, where each subject is its own control, are therefore much more powerful than cross-sectional ones for many questions. Animal studies add a well-known complication: animals housed together share microbes through coprophagy and the environment, so the cage, not the animal, is often the true unit.
Several factors are known to shift community composition strongly enough to create spurious associations: antibiotics and other medication (proton-pump inhibitors, metformin, laxatives), diet in the days before sampling, stool consistency and transit time, age, and host genotype in animal models. Technical factors matter just as much: the extraction kit, the sequencing run, the person who processed the samples, and the time a sample spent at room temperature. The safest approach is to randomise samples across extraction batches and sequencing runs, so that technical batches cannot line up with biological groups, and to record every one of these variables.
Reagents and laboratory surfaces carry their own low-level microbial DNA. In samples with high biomass, such as stool, this "kitome" is swamped by real signal. In low-biomass samples, such as skin swabs, tissue biopsies, blood, milk or air, contaminants can make up a large share of the reads and have produced headline findings that later proved to be artefacts (Salter et al., 2014; Eisenhofer et al., 2019). Every extraction batch should include an extraction blank (no sample) and every PCR plate a no-template control. These controls are sequenced like real samples and used later to identify contaminants (chapter 6).
Positive controls are mock communities: defined mixtures of known strains or of their DNA, such as the ZymoBIOMICS standards. Sequencing a mock community shows how far your workflow distorts the true composition. Extraction bias against Gram-positive cells, primer mismatches and GC bias all show up as a gap between expected and observed abundances. A mock community is also the best way to compare two pipelines on your own data.
Sequencing returns a sample of reads whose total depends on the library, not on how many microbes were in the sample. Counts are therefore relative: if one taxon doubles, every other taxon appears to shrink. Several methods recover absolute numbers. Flow cytometry measures total cell load per gram (Vandeputte et al., 2017), qPCR of the 16S gene measures total gene copies, and spike-ins add a known number of foreign cells or synthetic DNA before extraction. If the scientific question is about absolute change (did the antibiotic reduce the total bacterial load?), plan for one of these from the start.
A sample without metadata is almost worthless. Record subject, time point, treatment, the confounders above, extraction batch, sequencing run and any deviation from protocol, in a single tidy table with one row per sample. Use controlled vocabularies, units in the column names and no merged cells. Standards such as MIxS from the Genomic Standards Consortium describe the minimum information expected when data are deposited in public archives, and following them from the start saves work at publication time (Wilkinson et al., 2016).
Three. Technical replicates measure the laboratory noise, not the variation between people.
Kit and laboratory contaminants make up a larger share of the reads when there is little real DNA; without blanks they look like genuine taxa.
Short reads are cheap and accurate; long reads span repeats and reveal structure. Most choices downstream follow from which kind of read you have and from how good it is.
Illumina instruments produce reads of 100 to 300 bases, often as pairs from the two ends of a fragment. Per-base accuracy is high (typically better than 99.9 percent) and errors are mostly substitutions, which rise towards the end of each read. Short reads are the standard for amplicon sequencing, for shotgun metagenomic profiling and for polishing bacterial genomes. Their weakness is length: any repeat longer than a read or a fragment, such as rRNA operons, insertion sequences and duplicated genes, cannot be resolved, so short-read genome assemblies usually break into tens to hundreds of contigs.
Oxford Nanopore (ONT) reads DNA as it passes through a protein pore and can produce reads from a few kilobases to more than a hundred kilobases. Older chemistries had error rates of several percent, dominated by insertions and deletions in homopolymers. The R10.4.1 pores with recent basecallers reach modal accuracies above 99 percent, enough to produce near-finished bacterial genomes without short-read polishing (Sereika et al., 2022). PacBio HiFi reads are 10 to 25 kb long with accuracy above 99.9 percent, from circular consensus sequencing of the same molecule. Older PacBio CLR reads are longer but much noisier. A long read that spans every repeat in a genome allows assembly into one circular chromosome plus plasmids.
Each base comes with a Phred quality score Q, where the probability that the base is wrong is
P(error) = 10−Q/10
so Q20 means a 1 in 100 chance of error and Q30 a 1 in 1000 chance. Quality plots per position show where reads degrade; read-length histograms show whether long-read libraries contain the long molecules you paid for. Simple summary tools such as report read counts, total bases, N50 and mean quality per file (Shen et al., 2016); does the same with plots tailored to long reads (De Coster et al., 2018).
Ismail works through FASTQ encodings, quality plots and trimming in detail, and Low & Tammi give a hands-on version of the same checks (Ismail, Bioinformatics: A Practical Guide to NGS Data Analysis, ch. 1; Low & Tammi, Practical Bioinformatics for Beginners, ch. 3).
For 16S amplicons, 10,000 to 50,000 reads per sample usually captures the dominant community; rare taxa need more. For shotgun profiling of gut samples, a few million read pairs per sample are enough for species-level taxonomic profiles, while functional profiling and genome reconstruction from metagenomes benefit from 10 to 20 million or more. For isolate genomes, 50 to 100-fold short-read coverage or 30 to 50-fold long-read coverage is typical; very high long-read coverage can be subsampled to the longest, best reads.
10−30/10 = 0.001, one error in a thousand bases. Q20 is one in a hundred.
Illumina errors are mostly substitutions near read ends, easy to trim; Nanopore errors include insertions and deletions, often in homopolymers, spread along the read.
Almost every sequence question starts with a comparison: is this gene like one we know, where does this read belong, how far apart are these two strains? An alignment is the answer in its most explicit form, and is how we ask it of millions of sequences at once.
Two sequences are homologous if they descend from a common ancestor. That is a yes-or-no statement about history: a pair is homologous or it is not, and "70 % homologous" is a category error. What we can measure is similarity, the fraction of aligned positions that are identical or chemically alike. Similarity is evidence for homology, never the definition of it. Pearson puts the practical consequence plainly: statistically significant similarity lets us infer homology, and absence of significant similarity does not prove its absence (Pearson, 2013).
Homologs come in two kinds that matter for naming genes. Orthologs diverged when species diverged, so they are "the same gene" in two organisms. Paralogs diverged when a gene was duplicated inside one lineage, and often take on different jobs. Calling a paralog by its relative's name is one of the commonest sources of wrong annotation, which chapter A3 returns to.
An alignment writes two sequences in rows so that positions believed to share an ancestor sit in the same column. A column holds a match, a mismatch or, when one sequence has an inserted or deleted stretch, a gap (a dash). Many alignments are possible for the same pair, so we need a rule to say which is best: a score that adds a reward for each match, a penalty for each mismatch and a penalty for each gap, and we keep the alignment with the highest total.
DNA scoring is simple because there are only four letters: a positive score for a match and a negative one for a mismatch. The ratio sets what the alignment is tuned to find. 's megablast uses +1 and −2, which rewards long, nearly identical matches; the default uses +2 and −3, which tolerates somewhat more divergence (the defaults of +; Camacho et al., 2009).
For proteins, not every mismatch is equally bad. Replacing leucine with isoleucine, two similar hydrophobic residues, happens often in evolution and costs little; replacing tryptophan with alanine happens rarely and costs a lot. A substitution matrix stores one score for every pair of the 20 amino acids as a log-odds ratio: how often the pair is seen in real alignments of related proteins, divided by how often chance alone would put them together. The two families you will meet are PAM, built from closely related proteins and extrapolated to greater distances (Dayhoff et al., 1978), and BLOSUM, built directly from conserved blocks of real alignments (Henikoff & Henikoff, 1992). The number in a BLOSUM name is the identity at which sequences in the blocks were merged: BLOSUM62 comes from blocks clustered at 62 %, so it suits moderately related proteins; BLOSUM45 suits more distant ones and BLOSUM80 closer ones. BLOSUM62 is the default of for proteins.
| Pair | BLOSUM62 score | Why |
|---|---|---|
| W–W | +11 | tryptophan is rare, so matching it is strong evidence |
| C–C | +9 | cysteines often form structural disulfide bonds |
| A–A | +4 | alanine is common, so matching it says less |
| I–V | +3 | both small, branched and hydrophobic |
| D–E | +2 | both acidic and negatively charged |
| A–W | −3 | small against large and aromatic: rarely tolerated |
An insertion or deletion is usually one event that removes or adds several residues in one go, so a gap of length 5 is much less than five times as unlikely as a gap of length 1. Affine gap penalties capture this with a large cost to open a gap and a small cost for each further position (Gotoh, 1982). For protein searches charges 11 to open and 1 to extend with BLOSUM62. The gap penalties matter as much as the matrix: too cheap and the program stitches unrelated fragments together, too dear and it cannot cross a real insertion.
There are astronomically many ways to place gaps, yet the best one can be found without trying them all, because the best alignment of two prefixes is built from the best alignments of shorter prefixes. Fill a table whose cell (i, j) holds the best score for the first i letters of one sequence against the first j of the other. Each cell is the best of three moves: diagonal (align the two letters, add the match or mismatch score), down (a gap in the second sequence, add the gap penalty) or right (a gap in the first). When the table is full, the bottom-right cell is the optimal score and following the choices back to the top-left gives the alignment itself. This is the Needleman–Wunsch algorithm for global alignment, which aligns both sequences from end to end (Needleman & Wunsch, 1970).
Global alignment is right when the two sequences are expected to be homologous along their whole length, such as two versions of one gene. When only a region is shared, for instance a protein domain inside two different proteins, forcing the ends to align drags the score down. Local alignment, the Smith–Waterman algorithm, changes two rules: no cell may fall below zero, which lets an alignment start anywhere, and the answer is the highest cell anywhere in the table, not the corner (Smith & Waterman, 1981). Tracing back from that cell until a zero is reached recovers the best-matching region.
Dynamic programming is exact but slow: the table has m × n cells, which is trivial for two genes of a few kilobases and impossible for two human chromosomes, where aligning chromosome 1 with itself would need about 6 × 1016 cells. Everything in the next sections is a way to avoid filling the whole table.
A raw alignment score depends on the matrix and gap penalties, so a score of 120 means nothing on its own. Karlin and Altschul showed that, for local alignments without gaps, the scores of unrelated sequences follow a known distribution, so the chance of seeing a score at least this high can be calculated (Karlin & Altschul, 1990). Two numbers come out of that theory.
Remember what the E-value measures: how surprising the score is under a random model of sequences. Real proteins are not random. Low-complexity stretches (poly-Q, collagen repeats) and unusual amino-acid composition produce high scores between unrelated proteins, which is why masks low-complexity regions by default and applies a composition-based adjustment to protein scores (Schäffer et al., 2001). A significant E-value is evidence of homology, but it is evidence about the score, not a verdict on the biology.
avoids the table by looking only where similarity is likely. It cuts the query into short words (11 letters for , 28 for megablast, 3 amino acids for protein searches), finds the places in the database where a word matches exactly or, for proteins, scores above a threshold, and tries to extend each such seed into a longer alignment, stopping when the score drops too far. It is a heuristic: it can miss a true alignment that has no good seed, in return for being thousands of times faster than Smith–Waterman (Altschul et al., 1990). Gapped requires two nearby seeds on the same diagonal before it extends, which cut the run time again, and PSI- repeats the search with a profile built from the hits of the previous round, which finds more distant relatives (Altschul et al., 1997). The modern command-line program is + (Camacho et al., 2009).
| Program | Query → database | Use it to… |
|---|---|---|
| / megablast | nucleotide → nucleotide | find the same or a very close sequence; megablast for near-identical hits, dc-megablast or for more divergent ones |
| | protein → protein | find homologous proteins, including distant ones |
| | nucleotide, translated in 6 frames → protein | annotate a gene or read of unknown frame |
| | protein → nucleotide, translated in 6 frames | find where a known protein hides in an unannotated genome |
| tblastx | translated → translated | compare coding sequences when neither has been annotated (slow) |
Searching at the protein level is more sensitive for divergent sequences than searching DNA: the 20-letter alphabet carries more information per position, and the genetic code is degenerate, so third-codon positions drift faster than the amino acids they code for. Between distantly related species a DNA search often finds nothing where a protein search finds a clear homolog.
The tabular format (-outfmt 6) gives one row per aligned region. Read the columns together, never one alone.
| Column | Meaning | Common trap |
|---|---|---|
| pident | identical positions divided by alignment length | 100 % over 20 residues of a 600-residue protein is a hit to a motif, not to the gene |
| length, qcovs | aligned length; share of the query covered | a high identity on low coverage is a domain match |
| evalue | expected random hits at this score or better | depends on database size: do not compare across databases |
| bitscore | score on a common scale | use it, not the E-value, to rank hits within one search |
For transferring a function from a hit to your protein, identity and coverage both matter. Tian and Skolnick found that enzyme function is usually conserved above about 40 % identity but is increasingly unreliable below it (Tian & Skolnick, 2003). Between roughly 20 and 35 % identity over a long alignment lies what Rost called the twilight zone, where identity alone cannot separate homologs from chance and a profile or structure is needed (Rost, 1999). These are starting points, not laws: a catalytic residue lost to a single substitution can silence an enzyme at 90 % identity.
For a second explanation with exercises, read Baxevanis, Bader and Wishart, ch. 3, on pairwise similarity with and FASTA (Bioinformatics, 4th ed.); Christensen, ch. 4, which links pairwise alignment, multiple alignment and in a microbiology setting (Introduction to Bioinformatics in Microbiology); and Pevzner and Shamir, ch. 4, for dynamic programming as a general idea (Bioinformatics for Biologists).
Possibly a shared domain, but the evidence is about 8 % of the sequence. Coverage matters as much as identity: this is a domain-level hit, and the rest of the proteins may be unrelated.
A zero lets an alignment start afresh anywhere. A poor region cannot drag the score of a good one below zero, so the best-matching region is found regardless of what lies on either side.
The bit score is unchanged; it depends on the alignment, not the database. The E-value rises about tenfold, because there are ten times as many chances for a random hit.
: the protein is the query and the genome is translated in all six frames.
A pairwise alignment compares two sequences. A multiple alignment lines up many at once, and it is the raw material for almost everything that follows: conserved sites, profile models, and the family trees that tell you how genes, strains and species are related.
In a multiple sequence alignment (MSA) each column is a claim: the residues in it descend from one residue of the common ancestor. A well-built MSA shows which positions are conserved, which vary, and where insertions and deletions have happened. Conserved columns point to structure and function; variable columns carry the signal that separates strains and species. The same alignment feeds profile models (chapter A3) and phylogenetic trees (below), so a mistake here is inherited by everything built on it.
Exact multiple alignment by dynamic programming grows exponentially with the number of sequences and is not used in practice. Almost every method is progressive: align the two most similar sequences first, then add the next most similar sequence or group, following a guide tree from the closest pairs to the most distant ones. The weakness is built in. A gap placed in an early alignment is never revisited, so an early error spreads to every later step, a problem often summarised as "once a gap, always a gap". Better methods soften this with iterative refinement, which repeatedly splits the alignment and realigns the parts, as in MUSCLE (Edgar, 2004), or with a consistency score that prefers alignments supported by many pairwise comparisons.
| Program | Strength | Use it when |
|---|---|---|
| | fast, with settings from quick to accurate (Katoh & Standley, 2013) | the default choice; --auto picks a strategy for the data size |
| MUSCLE | accurate; version 5 gives an ensemble of alignments to test how much a result depends on the alignment (Edgar, 2022) | moderate numbers of sequences; checking robustness |
| Clustal Omega | scales to many thousands of protein sequences (Sievers et al., 2011) | very large protein sets |
Align proteins, then go back to DNA. A coding sequence aligned as DNA treats a deleted codon as a stretch of three nucleotides that can be misplaced by one position, and the codon structure is lost. The reliable order is to translate, align the proteins, and map the gaps back onto the nucleotides in whole codons, which preserves the reading frame and is the correct input for tests of selection. PAL2NAL does this step (Suyama et al., 2006).
Look at the alignment before using it. Scroll through it in a viewer: long gap-rich blocks, a single sequence that does not fit anywhere, or a large block of mismatches at one end usually mean a wrong sequence (a fragment, a contaminant, a different gene) or a poor alignment. Remove or fix those first. Automatic trimming of unreliable columns, for example with trimAl (Capella-Gutiérrez et al., 2009), is common practice, but it is not free: a benchmark of trimming tools found that they frequently worsened single-gene trees, because the columns removed carry signal as well as noise (Tan et al., 2015). Trim lightly and record what you did.
A phylogenetic tree is a diagram of descent. Tips are the sequences you sampled. Each internal node is a common ancestor of everything below it and each branch is a lineage. A clade is a node and all its descendants; a clade is monophyletic when it contains an ancestor and all of its descendants. Three things are easy to get wrong:
| Approach | Idea | Trade-off |
|---|---|---|
| Distance (neighbor-joining) | convert the alignment to pairwise distances and join the closest pairs, correcting for unequal rates (Saitou & Nei, 1987) | very fast, one tree, loses information when sequences are very divergent |
| Maximum parsimony | choose the tree that needs the fewest changes | no model of evolution; can mislead when rates differ between lineages |
| Maximum likelihood | choose the tree and branch lengths that make the data most probable under an explicit model (Felsenstein, 1981) | the standard for most work; slower, needs a model |
| Bayesian inference | sample trees in proportion to their posterior probability (Ronquist et al., 2012) | gives a probability for each clade and handles dating; slow, needs care that runs have converged |
For most projects use maximum likelihood. (Minh et al., 2020) and -NG (Kozlov et al., 2019) are the usual programs; is a much faster approximate method suited to very large alignments, where it trades some accuracy for speed (Price et al., 2010).
Count the share p of positions at which two sequences differ. It is tempting to use p as the distance, but a position can change twice (A to G and back, or A to G to T), and the second change may be invisible. The longer the time, the more changes are hidden. A substitution model corrects for this. The simplest, Jukes–Cantor, assumes every change is equally likely and gives the corrected distance d = −¾ ln(1 − 4p/3).
Real models add the facts of real evolution: transitions (A↔G, C↔T) are more frequent than transversions, base frequencies are not equal, and some sites change faster than others, usually modelled by a gamma distribution of rates (written +G). The general time-reversible model with gamma rates (GTR+G) is a common DNA model. Rather than guessing, let the program choose: ModelFinder, built into , compares models by an information criterion and reports the best (Kalyaanamoorthy et al., 2017).
The bootstrap resamples the columns of the alignment with replacement, builds a tree from each resample, and reports for every branch the percentage of resampled trees that contain it (Felsenstein, 1985). A branch with 87 means that 87 of 100 resampled alignments gave that grouping. It measures how consistently the data support the grouping, not the probability that the grouping is true: a consistent error, such as a wrong model or a contaminated sequence, can be supported at 100. The standard bootstrap needs many full tree searches, so offers an ultrafast version, UFBoot, whose values are read differently, with 95 or more usually taken as well supported (Hoang et al., 2018), and the SH-aLRT test, where 80 or more is the common mark (Guindon et al., 2010). Report which kind of support a figure shows, and treat branches below these marks as unresolved, not as wrong.
A tree built from one gene is a gene tree. It need not match the tree of the species. Paralogs make the gene tree describe duplications as well as speciations, so trees of gene families must be read with orthology in mind. Horizontal gene transfer moves genes between lineages, which is common in bacteria. Incomplete lineage sorting makes gene trees differ from the species tree in closely related species. The usual remedies are to build the species tree from many single-copy genes, either by joining their alignments or by reconciling their individual trees, and to check how well individual genes agree. The 16S rRNA gene is a convenient single marker for bacteria, but a tree from one marker is a tree of that marker.
Baxevanis, Bader and Wishart, ch. 8, covers multiple alignment and how to measure its quality (Bioinformatics, 4th ed.). Claverie and Notredame take the practical route, from building an alignment to building a tree, in ch. 9 and ch. 13 (Bioinformatics For Dummies). For the methods behind tree estimation read Warnow (Computational Phylogenetics), and for doing it in R, Paradis (Analysis of Phylogenetics and Evolution with R).
Not necessarily. The order of tips is arbitrary because nodes can be rotated. Close relatives are those that share a recent common ancestor: follow the branches up to the node they share.
No. Some positions have changed more than once and the extra changes are invisible. Under Jukes–Cantor the corrected distance is about 0.82.
No. It shows that the data consistently support it under the model used. A wrong model, a contaminated sequence or long-branch attraction can be supported at 100.
The gene tree contains duplication events as well as speciation events, so it does not equal the species tree. Separate orthologs from paralogs before reading it as a species phylogeny.
A genome sequence is a string of letters. Annotation turns it into a list of genes with names and functions, and almost every later analysis depends on that list. It is also the step in which the most confident-looking errors are made, so it pays to know what each kind of evidence is worth.
Bacterial and archaeal genomes are compact: the great majority of the sequence codes for protein or for a structural RNA, and genes lie close together, often on both strands. Genes with related functions are frequently transcribed together as an operon, one messenger RNA carrying several genes, so the order of genes on the chromosome is itself information. A bacterial gene is a single uninterrupted reading frame, with a ribosome-binding site, the Shine–Dalgarno sequence, a few bases upstream of the start codon.
Eukaryotic genomes are loose. In the human genome, protein-coding exons are only about 1–2 % of the sequence. Genes are split into exons separated by introns, which are removed from the transcript by splicing; introns nearly always begin with GT and end with AG. Genes also have untranslated regions (UTRs) at either end, and one gene can give several transcripts by alternative splicing. Fungi lie in between: compact genomes, but with introns, usually few and short in yeasts and more numerous in filamentous species. Chapter 33 treats fungal gene prediction in practice.
A codon is three nucleotides. Of the 64 codons, 61 specify an amino acid and three (TAA, TAG, TGA) mean stop; ATG codes for methionine and is the usual start. A sequence can be read in three frames on each strand, six in all, and a gene is read in only one. An open reading frame (ORF) is the stretch from a start codon to the next stop in one frame. Not every ORF is a gene: stop codons turn up by chance, so short ORFs are everywhere.
Organisms do not all use the same code. Bacteria and archaea use the standard amino-acid assignments but also accept GTG and TTG as starts ( translation table 11). Mycoplasma reads TGA as tryptophan instead of stop (table 4), and mitochondria have their own tables. Annotating with the wrong table truncates every gene that contains the reassigned codon.
Prokaryotes. Gene finders score candidate ORFs on how much they look like the genome's own coding sequence (codon usage and its pattern across the three codon positions), on the presence of a ribosome-binding site, and on the start codon used, then choose the set of genes that fits together best along the chromosome. learns all of these from the genome itself without being given a training set (Hyatt et al., 2010). Start positions are the hardest part: the end of a gene is fixed by its stop codon, but several in-frame start codons may be possible, and a wrong choice shortens or lengthens the protein at the N-terminus.
Eukaryotes. The problem is to find exons and join them correctly across introns. Ab initio programs such as use a hidden Markov model of exon, intron and intergenic sequence (Stanke et al., 2006). They become much more accurate when given evidence: RNA-seq reads aligned to the genome and proteins from related species. Pipelines such as combine the two and train the model on the genome being annotated (Brůna et al., 2021).
Non-coding genes. Transfer RNAs are found with (Chan et al., 2021). Other structural RNAs, from ribosomal RNA to riboswitches, are found by covariance models of their conserved secondary structure, built into and catalogued in (Nawrocki & Eddy, 2013; Kalvari et al., 2021).
compares a sequence with another sequence. When two proteins are too distant for that, a profile does better, because it summarises a whole family. A profile hidden Markov model (profile HMM) is built from a multiple alignment of a family (chapter A2). For each column it records how likely each amino acid is, so a conserved tryptophan counts for much and a variable position for little, and it has states for insertions and deletions between columns. A new sequence is scored against the model, and the score says how well it fits the family, not any one member. searches sequences against such models quickly (Eddy, 2011).
The database holds profile HMMs for thousands of protein families and domains, each with a curated score cut-off that sets where a hit is counted (Mistry et al., 2021), and combines with many other signature databases into one report per protein (Paysan-Lafosse et al., 2023). A domain hit tells you which part of a protein resembles a known module, and not what the whole protein does: a protein with a kinase domain and an unknown N-terminal region is a kinase of unknown specificity.
Free text such as "putative transporter" is hard to compare across genomes. The Gene Ontology (GO) gives controlled terms in three branches: molecular function (what the product does), biological process (what larger activity it takes part in) and cellular component (where it works) (Ashburner et al., 2000; Gene Ontology Consortium et al., 2023). Each assignment carries an evidence code saying how it was reached. Codes beginning with experimental evidence (for instance direct assay) rest on an observation about that protein; the code IEA, inferred from electronic annotation, is computed with no human review. Most of the annotations in any large database are of the second kind. Pathway resources such as KEGG add the question of which reactions a genome can carry out.
Functional annotation then adds up as a ladder of decreasing certainty:
An annotation that is transferred from one genome to the next is copied, and the copy is not checked. Schnoes and colleagues compared the function assigned in public databases with the experimentally supported function across several enzyme superfamilies and found that misannotation was substantial in a number of superfamilies and that it had spread by transfer between similar sequences (Schnoes et al., 2009). Every new genome annotated by homology to a polluted database inherits the pollution and adds to it.
Bacterial genomes are usually annotated with an integrated pipeline: 's PGAP, used for every genome submitted to (Tatusova et al., 2016), (Seemann, 2014), or , which searches a curated and versioned database of protein families and returns identifiers that can be traced back to their source (Schwengers et al., 2021). They differ in the databases they use and in how cautiously they name genes, so record the pipeline and database version with the result. Chapter 19 describes bacterial annotation step by step, and chapters 33 and 34 cover fungal gene prediction and function.
Baxevanis, Bader and Wishart, ch. 5, explains gene prediction methods and the structure of an annotation project (Bioinformatics, 4th ed.). Zvelebil and Baum, ch. 10, treat gene detection and annotation across bacteria and eukaryotes (Understanding Bioinformatics). Keith (ed.), vol. I, has a chapter on genome annotation (ch. 5) and one on working with ontologies (ch. 6) (Bioinformatics, Methods in Molecular Biology).
Sequencing or assembly errors, most likely homopolymer indels from long reads, not 30 % true pseudogenes. Polish the assembly and annotate again.
Stop codons are AT-rich, so in a GC-rich genome they occur less often by chance. About one random stop-free stretch in seven is at least 100 codons long, so long false ORFs appear.
That it probably phosphorylates something. The domain does not tell you the substrate, the pathway or whether the catalytic residues are intact. The rest of the protein and other evidence decide.
IEA is inferred electronically, with no human review of the protein. An experimental code rests on an observation made on that protein, so it is far more reliable.
Bioinformatics tools run on servers, read files too large to open in a spreadsheet, and are chained into pipelines. The command line is where that work happens. A few dozen commands and the habit of writing down what you ran are enough to do most of it, and to be able to repeat it.
Three reasons make the shell indispensable. The machines that do the work are remote servers with no screen, reached by a terminal. The data are large: a sequencing run is gigabytes of text, and a command such as head shows the start of it at once. And a command is a record. A series of clicks cannot be rerun on next year's data, but a script of commands can, and a reviewer can read it (Wilson et al., 2014).
A file system is a tree of directories. You are always in one, the working directory, and a path can be absolute (starting at the root, /home/ana/project/data/reads.fq.gz) or relative to where you are (data/reads.fq.gz).
| Command | What it does |
|---|---|
pwd, ls -lh, cd dir | where am I, what is here with sizes, move to dir (cd .. goes up) |
mkdir -p a/b, cp, mv | make nested directories, copy, move or rename |
rm file | delete a file. There is no recycle bin: it is gone |
head -n 4 f, tail -n 4 f, less -S f | the first or last lines, and a pager to scroll a large file (q quits) |
wc -l f, zcat f.gz | head | count lines, read a compressed file without unpacking it |
Two habits save most accidents. Never put spaces in file names: the shell splits a name at the space, so my reads.fq becomes two files. And keep raw data read-only in its own directory, so that every result can be regenerated from it (Noble, 2009).
Each command reads text from its standard input and writes to its standard output. The pipe | connects the output of one to the input of the next; > writes to a file (and overwrites it), >> appends; errors go to a separate stream, 2>. Every command also returns an exit status: 0 means success and anything else means failure, which scripts can test.
| Tool | Use |
|---|---|
cut -f1,3 | pick columns of a tab-separated file (-d, for commas) |
sort, sort -k2,2nr, uniq -c | order lines, order by column 2 numerically and descending, count identical lines (the input must be sorted) |
grep -E 'pattern', -c, -v | lines that match a pattern, count them, or keep the lines that do not match |
awk -F'\t' '…' | a small language for columns: sums, means, filters |
sed 's/a/b/', tr | replace text, translate characters |
Sort numbers with -n. Without it, sort compares characters, so 5386, 19282 and 48502 come out as 19282, 48502, 5386, because "1" comes before "5". With sort -n they come out in numeric order.
A FASTA record is a header line starting with > followed by the sequence, so grep -c '>' genomes.fna counts the sequences. A FASTQ record is exactly four lines: the read name (starting with @), the bases, a + line, and a quality string with one character per base. The bases are therefore on every fourth line, starting at the second:
awk 'NR%4==2 {n++; t+=length($0)} END {printf "%d reads, mean length %.2f\n", n, t/n}' reads.fq
On a file of three reads of 10, 12 and 6 bases this prints 3 reads, mean length 9.33. A common shortcut fails: grep -c '^@' reads.fq counts lines that start with @, but @ is also a valid quality character (Phred score 31), so on a file in which one read has such a quality string it returned 4 for 3 reads. Count by position, as above, or use a tool built for it such as (Shen et al., 2016). A BAM file is binary: view it with view (Danecek et al., 2021), never with less.
A regular expression describes a family of strings. ^ and $ anchor a pattern to the start and end of a line, [0-9] is any digit, {6} repeats the previous item six times, and a dot must be written \. to mean a real dot. grep -E '^NC_[0-9]{6}\.[0-9]+$' keeps lines like NC_001416.1 and rejects AB123 and NC_12.3. Test a pattern on a few lines you know should match and a few that should not, before you trust it on a million.
A script is a file of commands. Run on its own, a script carries on after a failed step and produces output from broken input, which is worse than stopping. Start every script with:
#!/usr/bin/env bash
set -euo pipefail
-e stops the script when a command fails, -u stops it on an unset variable, and pipefail makes a pipeline fail if any command in it fails, not only the last. Without pipefail the line false | true reports success; with it the script stops (tested in bash). An empty variable is the one this does not catch. If d is set to an empty string, rm -rf ${d}/* becomes rm -rf /*. Guard it with ${d:?d is empty}, which stops the script with a message, and always quote variables ("$f") so that file names with spaces survive.
A loop does the same thing to many files: for f in data/*.fastq.gz; do echo "$f"; done. Put one step in one place, parameters at the top and outputs in a directory named for the step, and the script becomes its own documentation (Buffalo, ch. 12).
A shared server has a few login nodes to connect to and many compute nodes to work on. Do not run assemblies or alignments on the login node; hand them to the scheduler, which queues them and runs each where there are resources. On , a job script starts with resource requests (#SBATCH --cpus-per-task=8, --mem=32G, --time=04:00:00), is submitted with sbatch script.sh, watched with squeue -u $USER and cancelled with scancel JOBID. A job that asks for too little memory is killed when it exceeds it, and one that asks for too much waits longer in the queue, so read the log of a finished job and adjust. Workflow managers such as write these job scripts for you and resume after a failure (Di Tommaso et al., 2017).
Anyone, including you in a year, should be able to rerun an analysis and get the same result. A few habits give most of it: record every command in a script, not in memory; keep raw data read-only and results disposable; record the versions of the tools; put the code under version control; and write a short README that says what each directory holds (Sandve et al., 2013; Noble, 2009).
Buffalo, Bioinformatics Data Skills, is the best single book for this chapter: ch. 2 on setting up a project, ch. 3 on the Unix shell, ch. 4 on working with remote machines, ch. 7 on Unix data tools and ch. 12 on shell scripting, pipelines and running tasks in parallel. Hendrix, Applied Bioinformatics, ch. 1, introduces sequences, Biopython and GNU/Linux together. The next chapter turns from the shell to Python and R.
sort lengths.txt on 5386, 19282 and 48502, and get 19282 first. Why, and what fixes it?Plain sort compares characters, and "1" comes before "5". sort -n compares numbers.
100 reads, four lines each. The bases are on lines 2, 6, 10, and so on: NR%4==2.
rm -rf ${out}/* and out was never filled in. What happens with set -u, and what is safer?If out is unset, set -u stops the script. If it is set to an empty string it does not, and the command becomes rm -rf /*. Use ${out:?}, which stops on empty as well.
set -o pipefail?Without it a pipeline reports the status of its last command only, so a failure in the first step is hidden and the script goes on with empty or broken input.
The shell moves files; Python and R analyse what is in them. Python is a general-purpose language with good tools for sequences and for machine learning. R was built for statistics and holds most of the packages for sequencing statistics. A bioinformatician reads both, and the same few ideas, tidy tables, groups and plots, work in each.
Python is the glue of bioinformatics: it reads files, calls other programs, and has NumPy for fast arrays (Harris et al., 2020), pandas for tables (McKinney, 2010), Biopython for sequences and formats (Cock et al., 2009) and scikit-learn for machine learning. R is the language of statistics, and the Bioconductor project collects thousands of R packages for genomic data, with shared data structures so that tools for counts, ranges and annotation work together (Huber et al., 2015). , and , the packages behind differential expression in the RNA-seq chapters (24 to 29), are Bioconductor packages. You do not have to choose: the habits below are the same in both.
Most time in an analysis goes into getting data into the right shape. Wickham's tidy data rules are the standard: each variable is a column, each observation is a row, and each kind of observational unit is its own table (Wickham, 2014). A count table with one row per gene and one column per sample is wide: convenient to read, and often the form a tool wants. A table with one row per gene and sample, with columns for gene, sample and count, is long: the form for grouping, plotting and modelling. Knowing which you hold, and how to convert, solves a large share of everyday problems.
The same task in both languages: read a tab-separated file, group, summarise. The file is the real output from the viral demonstration data (80 genomes, with the length and the end structure of each).
pandas
import pandas as pd
v = pd.read_csv("_virus_summary.tsv", sep="\t")
v.groupby("topology")["length"].agg(["count", "median"])
R
v <- read.delim("_virus_summary.tsv")
aggregate(length ~ topology, v, function(x) c(n = length(x), median = median(x)))
Operations you will use constantly: filter rows (v[v["virus_score"] >= 0.99] in pandas, subset(v, virus_score >= 0.99) in R), select columns, merge two tables on a shared key (pd.merge, merge), group and summarise as above, and reshape between wide and long (melt and pivot in pandas; pivot_longer and pivot_wider in the tidyr package). When you merge, check that the number of rows is what you expect: a key that is not unique on one side silently multiplies rows.
Biopython reads and writes the standard formats and gives sequences methods. SeqIO.parse("genomes.fna", "fasta") iterates over the records of a FASTA file without loading them all, which matters for large files. A Seq object can be reverse-complemented and translated:
from Bio.Seq import Seq
from Bio.SeqUtils import gc_fraction
s = Seq("ATGGCCATTGTAATGGGCCGCTGAAAGGGTGCCCGATAG")
len(s), gc_fraction(s), s.translate(to_stop=True), s.reverse_complement()
This gives a length of 39, a GC fraction of 56.4 %, the protein MAIVMGR (translation stops at the first stop codon, TGA, because of to_stop=True) and the reverse complement CTATCGGGCACCCTTTCAGCGGCCCATTACAATGGCCAT. The genetic code that translation uses must match the organism: Biopython takes a table argument, and the wrong table gives wrong proteins (chapter A3).
R data live in data frames (tables) and vectors; most operations apply to a whole vector at once, so loops are rarely needed. The tidyverse packages (dplyr for grouping and filtering, ggplot2 for plots) give a consistent grammar for the table operations above. Bioconductor adds shared containers: a SummarizedExperiment holds a matrix of counts together with the table describing the samples and the table describing the genes, so that subsetting one subsets all three together. That removes a whole class of errors in which the counts and the sample table drift out of order.
Four R habits cause silent errors. Factor levels: a grouping variable is stored as a factor, and its first level is the reference that a model compares everything with, which is alphabetical unless you set it, so with control and treated the reference is control by luck; with healthy and disease it is disease, and every fold change flips sign. Indexing: R counts from 1 and Python from 0, and a slice in Python excludes its end. Missing values: mean(x) of a vector with an NA is NA unless you ask na.rm = TRUE, and a filter silently drops rows where the test is NA. Recycling: adding a vector of length 2 to one of length 6 repeats the short one, and gives a warning only when the lengths do not divide evenly.
A plot is an analysis step, not a decoration. Show the individual data points where there are few, not only a bar of the mean. Label axes with units. Use a log scale for quantities that span orders of magnitude, and say so. Start a bar chart's axis at zero. Do not use a colour scheme that a colour-blind reader cannot separate. And report the number of observations behind every summary.
Analyse in a script, or in a notebook that you can run from top to bottom in a fresh session. The danger of notebooks is hidden state: a cell run out of order leaves a variable behind that the final code no longer creates. Set a random seed wherever something is random (set.seed(1) in R, np.random.default_rng(1) in Python), and record the versions of the language and packages (sessionInfo() in R, pip freeze or a environment file in Python) (Sandve et al., 2013).
McKinney, Python for Data Analysis, covers NumPy in ch. 4, pandas in ch. 5, loading data in ch. 6, wrangling in ch. 7 and group operations in ch. 9. For R, see Buffalo, Bioinformatics Data Skills, ch. 8, a rapid introduction to the language, and Irizarry, Introduction to Data Science (the tidyverse and data wrangling parts) and Wickham and Grolemund, R for Data Science (tidy data). Gentleman, R Programming for Bioinformatics, is the reference for Bioconductor's design.
It is a wide table, convenient to read. The tidy (long) form has one row per gene and sample, with columns gene, sample and count, so that sample can be used for grouping and plotting.
The reference level. Factor levels are ordered alphabetically, so "disease" is the reference and the effect is healthy versus disease. Set the level order, or relevel, so that "healthy" is the reference.
A key was not unique on one side, so rows were repeated for each match. Check the keys for duplicates before merging, and the row count after.
Some gene symbols, such as SEPT2 or MARCH1, are converted to dates automatically, and the original symbol is lost.
Almost every file in bioinformatics is text: FASTA, FASTQ, GFF, VCF, count tables, logs. A handful of old Unix tools read all of them, in seconds, without loading anything into memory. Learn these tools well and most questions about a file ("how many?", "which ones?", "what is in column 5?") are one line long.
Every analysis that goes wrong starts with a file that was not what its author thought. Before any tool, look. head -n 5 f shows the start; wc -l f counts lines; file f says whether it is text, gzip or something else; cut -f1-4 f | head shows the first four columns; and cat -A f | head -n 3 shows invisible characters: tabs appear as ^I and the end of every line as $. A file saved on Windows has ^M$ at each line end (a carriage return), which breaks comparisons that look identical on screen. tr -d '\r' < f > f.unix repairs it.
Most tables are tab-separated (TSV) or comma-separated (CSV). The delimiter matters: cut and sort use a tab by default, and awk splits on whitespace unless told otherwise with -F'\t'. A gene description with a space in it will be split into two fields by plain awk, so for tab-separated data always write -F'\t'.
All examples below were run on small files made for the chapter. A count table counts.tsv (genes in rows, samples in columns), a sample table meta.tsv, a gene annotation in GFF3, and a FASTA file:
counts.tsv meta.tsv a.fa
gene s1 s2 s3 sample group >seq1 Escherichia coli
geneA 10 0 5 s1 control ATGCGTACGTTAGC
geneB 200 180 210 s2 treated ATGGCC
geneC 0 0 3 s3 treated >seq2 Bacillus subtilis
GGGTTTAAACCC
| Command | Meaning |
|---|---|
grep 'text' f | lines containing the text |
grep -v '^#' f | lines that do not start with #: drops comments and headers |
grep -c '>' a.fa | count the matching lines (here, the sequences of a FASTA file) |
grep -E 'a|b' | extended regular expressions: alternation, +, {n}, grouping |
grep -w 'gyrB' | whole word only, so that gyrB does not match gyrB2 |
grep -n, -A 2, -B 2 | show line numbers, and two lines after or before each match |
grep -F -f ids.txt f | lines of f that contain any identifier listed in ids.txt, as fixed strings |
grep -f without -F reads each identifier as a pattern, where . matches any character: NC_001416.1 would also match NC_001416x1. For a list of identifiers always add -F.
A regular expression (regex) is a pattern for text. Most characters match themselves; a few are operators. These are enough for almost all work:
| Pattern | Matches |
|---|---|
. | any one character. To match a real dot, write \. |
^, $ | the start and the end of the line |
[ACGT], [^ACGT], [0-9] | one character from the set, one not in the set, one digit |
*, +, ? | the previous item zero or more times, one or more times, zero or one time |
{3}, {2,5} | exactly 3 times, between 2 and 5 times |
(a|b) | a group, with a choice. A group can be reused in sed as \1 |
Two examples, both run. Accession numbers that look like nucleotide records, six digits and a version:
$ printf 'NC_001416.1\nAB123\nNC_045512.2\n' | grep -E '^NC_[0-9]{6}\.[0-9]+$'
NC_001416.1
NC_045512.2
And a renaming with sed, which keeps part of the match with a group and writes it back as \1:
$ echo "sample_01_R1.fastq.gz" | sed -E 's/_R([12])\.fastq\.gz/_\1.fq/' sample_01_1.fq
Two warnings. Anchor your patterns. Without ^ and $ a pattern matches anywhere in the line, which is rarely what you meant. And write the dot as \.: a bare dot is a wildcard. Regular expressions are also the wrong tool for nested formats (for a tree in Newick format or a JSON file, use a parser; see the Python chapters).
cut -f1,3 f keeps columns 1 and 3 (tab by default; -d, for commas). paste a b glues files side by side. sort -k2,2nr sorts by column 2, numerically, in reverse. uniq -c counts runs of identical lines, which is why its input must be sorted first; sort | uniq -c | sort -rn is the idiom for "count each value, most common first". For a numeric sort always add -n: plain sort puts 19282 before 5386, because it compares characters.
$ cut -f1,3 counts.tsv | paste - <(cut -f4 counts.tsv) gene s2 s3 geneA 0 5 geneB 180 210 geneC 0 3
The <( ... ) is process substitution: the output of a command is offered to another command as if it were a file. It avoids temporary files, and it is how you compare the outputs of two commands: diff <(cut -f1 a | sort) <(cut -f1 b | sort) lists identifiers that are in one file and not in the other.
awk reads a file line by line, splits each line into fields $1, $2, … (and $0 is the whole line), and runs your instructions. A program is a set of pattern { action } pairs; BEGIN runs before the first line and END after the last. NR is the line number and NF the number of fields. Variables start at 0 and arrays are built as you go.
sed 's/old/new/' f replaces the first old on each line; add g for every one. sed 1d deletes the first line (the header). sed -n '5,10p' prints lines 5 to 10. sed -E turns on extended regular expressions, and -i edits the file in place, which is dangerous: test without -i first, or write to a new file. A typical use is to pull a field out of a long annotation column:
$ grep -v '^#' anno.gff3 | grep -P '\tgene\t' | sed -E 's/.*Name=([^;]+).*/\1/' dnaA gyrB recA
Read the substitution: match any text (.*), then Name=, then capture one or more characters that are not a semicolon ([^;]+), then anything; replace the whole line by the captured group. For a field in a GFF attribute or a VCF INFO column this idiom is what you need.
join merges two files on a common first column, like a database join, but both files must be sorted on that column in the same way. Add -t $'\t' to use a tab; -v1 prints the lines of the first file that found no partner. Joining the count table to an annotation of two genes:
$ sed 1d counts.tsv | sort -k1,1 | join -t $'\t' - ann.tsv geneA 10 0 5 kinase geneB 200 180 210 gyrase $ sed 1d counts.tsv | sort -k1,1 | join -t $'\t' -v1 - ann.tsv geneC 0 0 3
The second command is the one that protects you: it lists the rows that did not match. Every merge should be followed by a check of what was lost.
| Tool | Use |
|---|---|
tr 'ACGT' 'TGCA', rev | translate characters; reverse a line. echo ACGT | rev | tr ACGT TGCA gives the reverse complement ACGT (it is its own reverse complement) |
fold -w1 | sort | uniq -c | one character per line, then count: base composition (A 7, C 8, G 9, T 8 in the example FASTA) |
paste -sd, - | join lines into one comma-separated line: a quick list of identifiers |
find dir -name '*.fq' | list files by name, size or age, recursively |
xargs -P 4 -I{} cmd {} | run a command on each input item, four at a time |
zcat, gzip -c, bgzip, | read compressed files without unpacking; index a sorted, bgzip-compressed table for random access |
, , | format-aware tools: use them for FASTA/FASTQ, BAM and VCF rather than awk, because they understand the format (Shen et al., 2016; Danecek et al., 2021) |
One-line tools are best for looking at data and for quick, well-defined summaries. Switch to a script when the logic has more than two or three conditions, when you need to read the same file twice, when the format is nested (JSON, Newick, XML), or when others must read and trust it. A 10-line Python or R script is easier to review than a 300-character awk line. The next chapters are about exactly that step.
awk -F'\t' '!/^#/ && $3=="gene"' f | wc -l, or grep -v '^#' f | cut -f3 | grep -c '^gene$'. Both skip comments and test the third column exactly.
sort -k1,1 a | join - b and many rows are missing, with no error. What do you check?That b is sorted on the same key with the same locale, and that the key columns hold the same text (hidden carriage returns, different case). Run join -v1 to list what did not match.
sed -E 's/(.*)_(R[12])/\2_\1/' do to sampleA_R1?It swaps the two parts around the last underscore: R1_sampleA. The groups are written back in the order \2, \1.
grep -f ids.txt table.tsv sometimes return rows that are not in the list?Each identifier is read as a pattern: a dot matches any character, and an identifier like g1 also matches g10. Use grep -F -w -f.
A one-liner answers a question once. A script answers it every time, on every file, for everyone. This chapter builds, step by step, a small but real script with the features of a good one: arguments, checks, loops, functions, clear errors and an exit status other programs can read. The script is run on real files and every output below is its real output.
A variable is set without spaces (n=3) and read with a dollar sign ($n or ${n}). Quote every expansion: "$f", not $f. Without quotes the shell splits the value at spaces and expands wildcards in it. The difference is easy to see:
$ x="a b"; for w in $x; do echo "[$w]"; done [a] [b] $ x="a b"; for w in "$x"; do echo "[$w]"; done [a b]
Single quotes keep text exactly as written; double quotes allow $variable and $(command) to be replaced by their values. $(command) runs a command and substitutes its output: today=$(date +%F).
The shell can cut and replace parts of a variable. These are used constantly to build output names from input names:
$ name=reads_001.fastq.gz
$ echo ${name%.fastq.gz} # remove a suffix
reads_001
$ echo ${name#reads_} # remove a prefix
001.fastq.gz
$ echo ${name/reads/trimmed} # replace the first match
trimmed_001.fastq.gz
$ echo ${#name} # length
18
Also useful: ${var:-default} (use a default if unset or empty), ${var:?message} (stop with a message if unset or empty) and basename and dirname for paths.
Every command returns an exit status: 0 for success, anything else for failure. if tests it. The test [[ ... ]] compares strings and checks files: -e exists, -f is a regular file, -d is a directory, -s exists and is not empty, -z "$s" the string is empty, == compares strings, and -eq -lt -gt compare integers. Combine tests with &&, || and !.
if [[ -s "$fastq" && ! -e "$out" ]]; then
echo "running"
fi
cmd || { echo "cmd failed" >&2; exit 1; } # stop if cmd fails
A function returns a status the same way: f() { return 3; }; f; echo $? prints 3 (tested). Always send error messages to standard error (>&2), so that they do not end up in a results file.
for f in data/*.fq; do ...; done # over files
while read -r line; do ...; done < list.txt # over lines of a file
arr=(s1 s2 s3); echo ${arr[1]} # an array: prints s2
for i in "${!arr[@]}"; do echo $i:${arr[$i]}; done
declare -A len; len[s1]=3; len[s2]=2 # an associative array (a dictionary)
Run on three names, the array loop printed 0:s1, 1:s2, 2:s3; the dictionary loop printed s1=3 and s2=2. read -r reads a line without interpreting backslashes: always use -r. Looping over $(ls) breaks on spaces in names; loop over a glob (*.fq) instead.
Here is a script that reports the number of reads and the mean read length of every FASTQ file in a directory, as a table. It uses everything above. The file is screen.sh:
#!/usr/bin/env bash
# screen.sh: count reads and mean length for every FASTQ file in a directory.
set -euo pipefail
usage() { echo "usage: $0 -d DIR [-o OUT]" >&2; exit 2; }
out=summary.tsv
while getopts "d:o:h" opt; do
case $opt in
d) dir=$OPTARG ;;
o) out=$OPTARG ;;
*) usage ;;
esac
done
: "${dir:?give a directory with -d}"
[[ -d $dir ]] || { echo "not a directory: $dir" >&2; exit 1; }
printf 'file\treads\tmean_length\n' > "$out"
for f in "$dir"/*.fq; do
[[ -e $f ]] || { echo "no .fq files in $dir" >&2; exit 1; }
awk -v name="$(basename "$f")" 'NR%4==2 {n++; t+=length($0)} END {printf "%s\t%d\t%.2f\n", name, n, t/n}' "$f" >> "$out"
done
echo "wrote $out" >&2
Read it in blocks. The first line says which interpreter runs the file. set -euo pipefail makes failure visible. getopts reads options: d: means -d takes a value. : "${dir:?...}" stops with a message if -d was not given. Then the directory is checked, the header written, and each file summarised by awk. It was run on two small files, s1.fq (3 reads) and s2.fq (2 reads):
$ ./screen.sh -d data -o results/summary.tsv wrote results/summary.tsv $ cat results/summary.tsv file reads mean_length s1.fq 3 9.33 s2.fq 2 11.00 $ ./screen.sh -d nodir; echo "exit=$?" not a directory: nodir exit=1 $ ./screen.sh; echo "exit=$?" ./screen.sh: line 14: dir: give a directory with -d exit=1
The last two runs matter as much as the first. A script that fails loudly, early and with a non-zero status can be put inside a pipeline, a scheduler job or a process, which stop when it fails. A script that fails quietly lets everything after it run on wrong input.
set -euo pipefail: stop on a failed command, on an unset variable, and on a failure anywhere in a pipeline. false | true reports success without pipefail and failure with it (tested). Know the limit: set -u catches an unset variable but not an empty one; ${var:?} catches both.rm -rf "${out:?}"/tmp stops if out is empty; rm -rf $out/tmp becomes rm -rf /tmp.cmd > out.tmp && mv out.tmp out: an interrupted run then leaves no half-written result that a later step mistakes for a finished one.[[ -s "$out" ]] && continue at the top of a loop makes the script resumable.trap. trap 'rm -f "$tmp"' EXIT runs when the script ends, even on error. Tested: trap "echo cleaning up" EXIT; echo working prints working then cleaning up. --version | head -n 1) and the date at the top of the log.shellcheck on every script. It finds unquoted variables, useless cat and many other mistakes, and explains each.xargs -P N runs N copies of a command in parallel, each on one input item. ls data/*.fq | xargs -P 2 -I{} sh -c 'echo "{}: $(grep -c "^+$" {}) records"' counted the records of both files at the same time (s1.fq: 3 records, s2.fq: 2 records). GNU parallel does the same with more control. Count the cores you use: on a shared server you request them from the scheduler and then use no more than you were given. For anything with many steps and dependencies between them, a workflow manager (, ) does what a long script does badly: it knows which outputs exist, reruns only what changed, and records what was run (Di Tommaso et al., 2017).
A useful progression is: one-liners to explore; a script per step, each taking inputs and an output directory as arguments; a driver script or a workflow manager that calls the steps; and a README with the commands. At every stage, keep raw data read-only and let results be regenerable. The reproducibility rules of the previous chapter apply here without change (Sandve et al., 2013).
for f in $(ls *.fq) fails on a file called my reads.fq. Why, and what do you write?The unquoted command substitution is split at the space into my and reads.fq. Loop over the glob directly, for f in *.fq; do ... "$f" ...; done, and avoid spaces in names.
${name%.fastq.gz} do to reads_001.fastq.gz?It removes the shortest match of the suffix from the end: reads_001.
exit 0 even though a step in the middle failed. How do you prevent that?Start with set -euo pipefail, or test each important command with || { echo "failed" >&2; exit 1; }, and never end a script with an exit 0 that hides earlier failures.
Standard output is what the next command or the results file receives; an error message mixed into it corrupts the data. Standard error is shown on the terminal and kept in the log.
Python is the language in which most bioinformatics tools and pipelines are written or glued together. This chapter teaches the language itself, from the first line to functions, files and errors, using DNA sequences as the running example. Every snippet was run; the printed output is real.
A name points to a value; the value has a type. Python's built-in types cover most of what a first analysis needs:
| Type | Example | Use in bioinformatics |
|---|---|---|
int, float | 14, 0.5 | lengths, counts, fractions, p values |
str | "ATGCGT" | sequences, identifiers, file names |
list | ["r1", "r2"] | an ordered, changeable collection: reads, samples |
tuple | ("chr1", 100, 900) | a fixed record: a coordinate, a name and a length |
set | {"A", "C"} | unique items, fast membership tests, overlaps |
dict | {"seq1": 20} | look up a value by key: length by sequence name |
bool, None | True, None | conditions; "no value" |
Indexing starts at 0 and a slice s[a:b] includes a and excludes b. This is the source of most off-by-one errors, and it is also why genomic interval formats differ: BED is 0-based and half-open, GFF and VCF are 1-based and closed. On seq = "ATGCGTACGTTAGC":
>>> len(seq), seq[0:3], seq[-3:], seq[::-1] (14, 'ATG', 'AGC', 'CGATTGCATGCGTA')
A function packages a step you will repeat, with a name and a documented purpose. Write one for every idea you use twice.
comp = {"A": "T", "C": "G", "G": "C", "T": "A"}
def revcomp(s):
"""Reverse complement of an unambiguous DNA string."""
return "".join(comp[b] for b in reversed(s))
def gc(s):
s = s.upper()
return (s.count("G") + s.count("C")) / len(s)
print(revcomp(seq)) # GCTAACGTACGCAT
print(f"{gc(seq):.3f}") # 0.500
A caution that matters in biology: this revcomp fails with a KeyError on N or any ambiguity code, which is the right behaviour for a function that was written for four letters; it is better to stop than to return a wrong sequence. For real work use a library (next chapter) that handles the full alphabet.
Count bases with a dictionary, then with the standard library:
counts = {}
for b in seq:
counts[b] = counts.get(b, 0) + 1
print(counts) # {'A': 3, 'T': 4, 'G': 4, 'C': 3}
from collections import Counter
print(Counter(seq).most_common(2)) # [('T', 4), ('G', 4)]
A list comprehension builds a list in one line: [i for i in range(len(seq) - 2) if seq[i:i+3] == "CGT"] returns the positions of the motif, [3, 7]. Codons are [seq[i:i+3] for i in range(0, len(seq) - len(seq) % 3, 3)], giving ['ATG', 'CGT', 'ACG', 'TTA']; the last two bases do not fill a codon and are dropped on purpose. sorted(reads, key=lambda r: r[1], reverse=True) sorts [("r1", 10), ("r2", 12), ("r3", 6)] by length, longest first. enumerate gives an index and zip walks two lists together: for i, (n, l) in enumerate(zip(names, lengths), start=1).
Always open files with with, which closes them even when an error occurs. This reads a FASTA file, in which a record can span several lines, into a dictionary of name to sequence:
recs, name = {}, None
with open("a.fa") as fh:
for line in fh:
line = line.rstrip("\n")
if line.startswith(">"):
name = line[1:].split()[0]
recs[name] = []
else:
recs[name].append(line)
recs = {k: "".join(v) for k, v in recs.items()}
print({k: len(v) for k, v in recs.items()}) # {'seq1': 20, 'seq2': 12}
This parser is fine for a lesson and for a small file. For production work use a tested parser (Biopython, next chapter): it handles blank lines, Windows line endings and the many little ways real files deviate.
Python stops at an error and tells you which line and why. Read the last line of the message first: it names the error and says what went wrong. Catch an error only when you know how to recover:
try:
n = int("12x")
except ValueError as e:
print("ValueError:", e) # ValueError: invalid literal for int() with base 10: '12x'
Never write a bare except: that swallows everything. It turns a clear failure into silent wrong results.
| Trap | What happens |
|---|---|
A mutable default argument: def add(x, acc=[]) | The list is created once, so add(1), add(2) returns [1, 2] [1, 2]: the calls share state. Use acc=None and make the list inside: [1] [2] |
Floating point: 0.1 + 0.2 == 0.3 | False. Floats are binary approximations. Compare with a tolerance (math.isclose), never with == |
round(2.675, 2) | 2.67, not 2.68, for the same reason |
7 / 2 and 7 // 2 | 3.5 and 3; 7 % 2 is 1. Division with / always gives a float |
| Changing a list while looping over it | Items are skipped. Loop over a copy, or build a new list with a comprehension |
A file of Python code is a module. Another program can import it and use its functions, and the same file can be run on its own. The condition if __name__ == "__main__": separates the two: its block runs only when the file is executed, not when it is imported. That is how a script becomes a library without losing its command-line use. The file below, seqtools.py, holds three helpers (a FASTA reader, a reverse complement and a k-mer counter) and is used in the rest of this chapter. It reads the genome of chapter P12.
"""seqtools: a few sequence helpers. Import it, or run it as a script."""
import sys
from collections import Counter
COMP = str.maketrans("ACGTacgt", "TGCAtgca")
def read_fasta(path):
"""Yield (name, sequence) for each record, reading one line at a time."""
name, parts = None, []
with open(path) as fh:
for line in fh:
line = line.rstrip()
if line.startswith(">"):
if name is not None:
yield name, "".join(parts)
name, parts = line[1:].split()[0], []
elif line:
parts.append(line)
if name is not None:
yield name, "".join(parts)
def revcomp(seq):
return seq.translate(COMP)[::-1]
def kmer_counts(seq, k):
return Counter(seq[i:i + k] for i in range(len(seq) - k + 1))
if __name__ == "__main__": # runs only when the file is executed, not when it is imported
for name, seq in read_fasta(sys.argv[1]):
print(name, len(seq))
$ python3 seqtools.py ../data/phix174.fa
gi|9626372|ref|NC_001422.1| 5386
$ python3 -c "import seqtools; print(seqtools.__name__, [n for n in dir(seqtools) if not n.startswith('_')][:4])"
seqtools ['COMP', 'Counter', 'kmer_counts', 'read_fasta']
Run as a script, it prints the name and length of each record. Imported, it prints nothing and offers its functions. Two habits follow. Put reusable code in modules and keep scripts thin. And never give a module a name that an installed package already has (random.py, csv.py): Python imports your file instead of the library, and the error that follows is baffling.
The function read_fasta uses yield instead of return. A function with yield is a generator: it produces one value, pauses, and resumes where it stopped when the next value is asked for. A for loop over it holds a single record in memory, however large the file. This is the difference between a program that handles a 100 GB FASTQ file and one that stops at the first 8 GB. The test below reads the 6,000 reads of chapter P12 twice, once into a list and once through a generator, and measures the peak memory of each with tracemalloc (records is the four-line FASTQ reader of the command-line tool in chapter P6).
import sys, tracemalloc
sys.path.insert(0, ".")
from fqtool import records
def mean_q(qual):
return sum(ord(c) - 33 for c in qual) / len(qual)
tracemalloc.start()
with open("../data/S1.fq") as fh:
everything = list(records(fh)) # all 6,000 reads held in memory at once
a = tracemalloc.get_traced_memory()[1]
del everything
tracemalloc.reset_peak()
with open("../data/S1.fq") as fh:
total = sum(mean_q(q) for _, _, q in records(fh)) # one read at a time
b = tracemalloc.get_traced_memory()[1]
print(f"peak memory holding the list: {a / 1e6:.2f} MB; with a generator: {b / 1e6:.3f} MB; same answer: {total / 6000:.2f}")
# output
peak memory holding the list: 2.53 MB; with a generator: 0.159 MB; same answer: 33.55
Holding the list takes 2.53 MB for a 1.3 MB file, because every Python string carries overhead, while the generator needs 0.159 MB, and the answer is the same. The rule: stream when you can, load when you must. A generator can be used only once; to go over the data twice, call the function again. Open files with with, which closes them even when an error occurs, and open gzip files with gzip.open(path, "rt"), which reads text from a compressed file without unpacking it.
Many questions in bioinformatics are questions of counting: bases, codons, k-mers, reads per gene. A dictionary maps a key to a number, and collections.Counter is a dictionary built for counting (it returns 0 for a key it has not seen, and most_common(n) sorts). A k-mer is a substring of length k. A sequence of length L has L − k + 1 of them, which a generator expression counts directly. The tetranucleotides (k = 4) of a genome form a signature that differs between organisms, and metagenome binning tools use it to group contigs that belong together.
import sys
sys.path.insert(0, ".")
from seqtools import read_fasta, kmer_counts
name, seq = next(read_fasta("../data/phix174.fa"))
c = kmer_counts(seq, 4)
print(len(seq), "bases;", sum(c.values()), "tetranucleotides;", len(c), "of 256 possible occur")
print("most common:", c.most_common(4))
print("GATC:", c["GATC"], "| expected about", round((len(seq) - 3) / 256, 1))
for k, v in sorted(c.items(), key=lambda kv: kv[1])[:3]:
print("rarest:", k, v)
# output
5386 bases; 5383 tetranucleotides; 255 of 256 possible occur
most common: [('TGCT', 67), ('TGGT', 57), ('TTTT', 55), ('GTTT', 51)]
GATC: 0 | expected about 21.0
rarest: GGGG 3
rarest: CGGG 3
rarest: CTAG 3
The 5,386-base genome holds 5,383 tetranucleotides, and 255 of the 256 possible ones occur. The most common, TGCT, appears 67 times. The one that is absent is GATC, the Dam methylation site that chapter P12 found missing with grep and discussed. Two independent tools reaching the same zero is the kind of check that gives confidence in both.
The re module is the same regular expression language as grep -E and sed -E (P3), inside a program. re.finditer gives the position of every match (0-based, like grep -ob), and re.findall a list of matches. Ambiguity codes are turned into character classes with a dictionary. One trap deserves care: matches do not overlap. Counting AA in AAAA gives 2 with a normal pattern but 3 positions, and a look-ahead (?=AA), which matches a position without consuming the characters, counts all three. Use a look-ahead whenever a motif can overlap itself.
import re
seq = "".join(l.strip() for l in open("../data/phix174.fa") if not l.startswith(">"))
iupac = {"R": "[AG]", "Y": "[CT]", "W": "[AT]", "S": "[CG]", "N": "[ACGT]"}
pattern = "".join(iupac.get(c, c) for c in "AGGAGNNNNATG") # exact bases stay, ambiguity codes become classes
print(pattern)
print([m.start() for m in re.finditer("AGGAG", seq)], "<- AGGAG, 0-based, as grep -ob gave")
print("non-overlapping 'AA' in 'AAAA':", len(re.findall("AA", "AAAA")), "| overlapping with a look-ahead:", len(re.findall("(?=AA)", "AAAA")))
print([m.start() for m in re.finditer("(?=(" + "AGGAG[ACGT]{4,10}ATG" + "))", seq)], "<- ribosome-binding site then ATG")
# output
AGGAG[ACGT][ACGT][ACGT][ACGT]ATG
[232, 835, 2382, 3540, 5062] <- AGGAG, 0-based, as grep -ob gave
non-overlapping 'AA' in 'AAAA': 2 | overlapping with a look-ahead: 3
[835, 2382, 5062] <- ribosome-binding site then ATG
The five AGGAG positions are the offsets that grep -ob printed in P12, and the three ribosome-binding sites followed by ATG are the same three. Positions that two methods agree on need no further argument.
A class bundles data and the functions that belong to it. When you find yourself passing the same three values (a start, an end and a strand) around a program, a class is the cleaner design. dataclasses.dataclass writes the boring parts (the constructor, __repr__ and comparison) for you, and order=True makes objects sortable by their fields in order. The class below stores an interval with BED-style 0-based half-open coordinates, converts from a GFF line (the conversion that chapter P12 did with awk) and tests for overlap: two half-open intervals overlap when each starts before the other ends.
from dataclasses import dataclass
@dataclass(order=True)
class Interval:
start: int # 0-based, half-open, like BED
end: int
name: str = ""
strand: str = "+"
def __len__(self):
return self.end - self.start
@classmethod
def from_gff(cls, line):
f = line.rstrip("\n").split("\t")
return cls(int(f[3]) - 1, int(f[4]), f[8].split(";")[0].replace("ID=", ""), f[6])
def overlaps(self, other):
return self.start < other.end and other.start < self.end
genes = sorted(Interval.from_gff(l) for l in open("../data/genes.gff") if not l.startswith("#"))
print(genes[0])
print("coding bases:", sum(len(g) for g in genes), "| overlapping pairs:", sum(a.overlaps(b) for i, a in enumerate(genes) for b in genes[i + 1:]))
# output
Interval(start=50, end=221, name='1_1', strand='+')
coding bases: 4950 | overlapping pairs: 1
The seven genes cover 4,950 bases, as found, and exactly one pair overlaps: genes 1_2 and 1_3 share one base at position 848. Overlaps of one or a few bases between neighbouring genes are common in compact genomes, where the two proteins are made in a coupled way (translational coupling). The half-open convention is what makes the test self.start < other.end and other.start < self.end correct without a special case for touching intervals.
Every sequence-comparison tool rests on dynamic programming (chapter A1): fill a table in which each cell holds the best score of aligning the first i bases of one sequence with the first j of the other, by choosing the best of a diagonal step (match or mismatch) and the two gap steps, then trace back from the corner. It takes 25 lines, and writing it once explains why alignment time grows with the product of the lengths. The function below is the Needleman–Wunsch algorithm with a linear gap penalty. It is checked against Biopython's PairwiseAligner, which does the same job in compiled code.
def needleman_wunsch(a, b, match=1, mismatch=-1, gap=-2):
"""Global alignment by dynamic programming: fill a score table, then trace back."""
n, m = len(a), len(b)
S = [[0] * (m + 1) for _ in range(n + 1)]
for i in range(1, n + 1):
S[i][0] = i * gap
for j in range(1, m + 1):
S[0][j] = j * gap
for i in range(1, n + 1):
for j in range(1, m + 1):
diag = S[i - 1][j - 1] + (match if a[i - 1] == b[j - 1] else mismatch)
S[i][j] = max(diag, S[i - 1][j] + gap, S[i][j - 1] + gap)
i, j, x, y = n, m, [], []
while i or j:
if i and j and S[i][j] == S[i - 1][j - 1] + (match if a[i - 1] == b[j - 1] else mismatch):
x.append(a[i - 1]); y.append(b[j - 1]); i -= 1; j -= 1
elif i and S[i][j] == S[i - 1][j] + gap:
x.append(a[i - 1]); y.append("-"); i -= 1
else:
x.append("-"); y.append(b[j - 1]); j -= 1
return S[n][m], "".join(reversed(x)), "".join(reversed(y))
a, b = "GATTACAGATTACA", "GATCACAGTTACCA"
score, x, y = needleman_wunsch(a, b)
print(score); print(x); print(y)
from Bio.Align import PairwiseAligner
al = PairwiseAligner(mode="global", match_score=1, mismatch_score=-1, open_gap_score=-2, extend_gap_score=-2)
print("Biopython score:", al.score(a, b), "| same:", al.score(a, b) == score)
# output
7
GATTACAGATTA-CA
GATCACAG-TTACCA
Biopython score: 7.0 | same: True
Both give a score of 7: twelve matches, one mismatch and two gaps (one in each sequence), which is 12 − 1 − 4 with a score of 1 for a match, −1 for a mismatch and −2 for a gap. Keep the hand-written version for understanding; use the library for work, because it is much faster, supports affine gaps, local alignment and substitution matrices, and has been tested by thousands of users. The match between the two is the test, and writing a slow reference to check a fast tool is a habit worth keeping.
Create one directory per project with data/ (raw, read-only), results/, scripts/ and a README. Create an isolated environment so that the project keeps its own library versions: python -m venv .venv && source .venv/bin/activate && pip install biopython pandas, then pip freeze > requirements.txt. Or use ( env create -f env.yml), which also installs non-Python tools such as and . The file that lists the versions is part of the result: a paper that says "analysed in Python" and nothing more cannot be reproduced (Wilson et al., 2014). In a Jupyter notebook, restart the kernel and run all cells from the top before you trust a result, since cells run out of order leave hidden state behind.
seq[2:5] return for "ATGCGTACG", and how many characters?"GCG", three characters: index 2 is the third letter, and the end index 5 is excluded.
def f(x, acc=[]) give surprising results?The default list is created once, when the function is defined, and every call that does not pass acc appends to the same list. Use None and create the list inside the function.
0.05 is computed as 0.1 + 0.2 - 0.25. Why is p == 0.05 unreliable?Floating-point arithmetic is approximate, so the result differs from 0.05 by a tiny amount. Compare with math.isclose or with a tolerance.
with open(...)?The file is closed when the block ends, even if an error occurs, so no file is left open and no buffered data is lost.
Two libraries cover most daily work: Biopython for sequence files and biological objects, and pandas for tables. This chapter uses both on small files whose answers can be checked by hand, then turns the result into a command-line tool with arguments and tests, which is the step that separates a script you run from software others can use.
Biopython (Cock et al., 2009) parses the formats and knows the biology: the genetic code, ambiguity codes, reverse complements. SeqIO.parse(file, format) yields one record at a time, so it works on files of any size.
from Bio import SeqIO
from Bio.Seq import Seq
from Bio.SeqUtils import gc_fraction
for rec in SeqIO.parse("a.fa", "fasta"):
print(rec.id, len(rec), round(gc_fraction(rec.seq), 3), rec.description)
# seq1 20 0.55 seq1 Escherichia coli
# seq2 12 0.5 seq2 Bacillus subtilis
s = Seq("ATGGCCATTGTAATGGGCCGCTGA")
print(s.reverse_complement()) # TCAGCGGCCCATTACAATGGCCAT
print(s.translate()) # MAIVMGR*
print(s.translate(to_stop=True)) # MAIVMGR
For FASTQ, rec.letter_annotations["phred_quality"] holds the quality of each base. On the three-read example file, read r2 has a minimum quality of 2 and a mean of 36.8 (it contains the character #, Phred 2), whereas r1 is 40 throughout. A Phred score Q means an error probability of 10−Q/10: Q20 is 1 error in 100, Q30 is 1 in 1,000.
Check three things whenever you translate: that the sequence is in the right frame, that you use the right genetic code table (table=11 for bacteria; table=4 for Mycoplasma), and that an in-frame stop codon in the middle (a *) means a pseudogene, a frameshift or a wrong frame, not a protein.
A DataFrame is a table with labelled rows and columns (McKinney, 2010). Read, look, then check:
import pandas as pd, numpy as np
c = pd.read_csv("counts.tsv", sep="\t", index_col=0) # genes x samples
m = pd.read_csv("meta.tsv", sep="\t", index_col=0) # samples x variables
print(c.sum(axis=0).to_dict()) # {'s1': 210, 's2': 180, 's3': 218}
print((c > 0).sum(axis=1).to_dict()) # {'geneA': 2, 'geneB': 3, 'geneC': 1}
assert list(c.columns) == list(m.index) # the same samples, in the same order
That last line is the most useful line in the chapter. Samples in the count table and in the metadata must be the same, in the same order; a silent mismatch assigns each sample to the wrong group and produces results that look normal. Assert it every time.
rel = c / c.sum(axis=0) * 100 # percent of each sample print(rel.round(1)) # s1 s2 s3 # geneA 4.8 0.0 2.3 # geneB 95.2 100.0 96.3 # geneC 0.0 0.0 1.4 cpm = c.div(c.sum(axis=0), axis=1) * 1e6 print(np.log2(cpm + 1).round(2)) # the +1 is a pseudocount: log(0) is undefined # s1 s2 s3 # geneA 15.54 0.00 14.49 # geneB 19.86 19.93 19.88 # geneC 0.00 0.00 13.75
Look at geneA: its relative abundance is 4.8% in s1 and 0% in s2, and in the log scale a zero stays zero. Two cautions follow. Relative abundance is compositional: in s2, geneB is 100% because nothing else was counted, and if one gene rises the others must fall. And a pseudocount is a choice that changes the result for rare features; report it.
Analyses ask "for each group, what is the mean?". Reshape from wide (one column per sample) to long (one row per gene and sample), attach the metadata, and group (Wickham, 2014):
long = (c.reset_index()
.melt(id_vars="gene", var_name="sample", value_name="count")
.merge(m, left_on="sample", right_index=True))
print(long.groupby(["gene", "group"])["count"].mean().unstack().round(1))
# group control treated
# gene
# geneA 10.0 2.5
# geneB 200.0 195.0
# geneC 0.0 1.5
A merge is where tables silently go wrong. If the key is not unique on one side, rows multiply: merging a table of 3 rows with a second table in which the key 1 appears twice gave 5 rows, not 3 (tested). After every merge, compare the number of rows before and after, check for missing values (df.isna().sum()) and check keys for duplicates (df["k"].is_unique). Read a merge's validate="one_to_one" argument as a free assertion.
A script becomes a tool when it takes its inputs as arguments, does one thing, reports errors clearly and is tested. This tool, fqstats.py, reads FASTQ files (also gzip-compressed) and reports the number of reads, the mean read length and the mean quality, optionally ignoring low-quality reads. The core is a generator that reads four lines at a time:
def read_fastq(handle):
"""Yield (name, bases, quals) from a FASTQ file, four lines at a time."""
while True:
name = handle.readline().rstrip("\n")
if not name:
return
bases = handle.readline().rstrip("\n")
handle.readline()
quals = handle.readline().rstrip("\n")
if len(bases) != len(quals):
raise ValueError(f"{name}: {len(bases)} bases but {len(quals)} quality characters")
yield name[1:], bases, quals
The summary function converts each quality character to a number (ord(c) - 33, the Sanger/Illumina 1.8+ offset), averages, and skips reads below a threshold. The main function uses argparse, which builds the help text, checks the arguments and gives a clear error:
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("files", nargs="+", help="FASTQ files, optionally gzipped")
ap.add_argument("--min-quality", type=float, default=0,
help="skip reads with a lower mean quality (default: %(default)s)")
a = ap.parse_args(argv)
print("file\treads\tmean_length\tmean_quality")
for f in a.files:
s = summarise(f, a.min_quality)
print(f"{f}\t{s['reads']}\t{s['mean_length']:.2f}\t{s['mean_quality']:.1f}")
return 0
Run on the two example files, and again with a quality filter:
$ python fqstats.py data/s1.fq data/s2.fq file reads mean_length mean_quality data/s1.fq 3 9.33 38.4 data/s2.fq 2 11.00 40.0 $ python fqstats.py --min-quality 37 data/s1.fq data/s1.fq 2 8.00 39.2
The filter removed read r2 (mean quality 36.8), so the remaining two reads have lengths 10 and 6 and a mean length of 8. The numbers can be checked by hand, which is what a test is. With pytest, each test is a function whose name starts with test_:
def test_reads_are_parsed_four_lines_at_a_time():
fh = io.StringIO("@r1\nACGT\n+\nIIII\n@r2\nAC\n+\nII\n")
assert [(n, b) for n, b, _ in read_fastq(fh)] == [("r1", "ACGT"), ("r2", "AC")]
def test_mismatched_quality_length_is_an_error():
with pytest.raises(ValueError):
list(read_fastq(io.StringIO("@r1\nACGT\n+\nII\n")))
def test_quality_filter_removes_low_quality_reads():
assert summarise("data/s1.fq", min_q=37)["reads"] == 2
All four tests of the suite passed (4 passed). Test the cases that matter most: a normal file, a file with a defect, an empty file and the edge of a threshold. A tool that has tests can be changed without fear, and a result that comes from a tested tool is worth more than one that does not (Wilson et al., 2014).
The tool above takes one command. Real tools do several things, and the pattern for that is subcommands, the way git commit and view work. argparse builds them with add_subparsers; each subcommand has its own arguments, types (type=float) and defaults, and --version and --help come for free. Three more habits separate a tool from a script. It reports a failure on standard error with a short message and returns a non-zero exit status, so that a pipeline or a scheduler can see it failed (P4). It uses the logging module, not print, for progress messages, so that the user decides with -v how much to see. And its main(argv) takes the arguments as a parameter, so that a test can call it directly without starting a new process. The next version, fqtool.py, has two subcommands, stats and filter:
#!/usr/bin/env python3
"""fqtool: summarise or filter a FASTQ file by quality."""
import argparse
import gzip
import logging
import sys
__version__ = "0.1.0"
log = logging.getLogger("fqtool")
def open_text(path):
"""Open a plain or gzipped text file."""
return gzip.open(path, "rt") if str(path).endswith(".gz") else open(path)
def records(fh):
"""Yield (header, sequence, quality) from an open FASTQ file."""
while True:
head = fh.readline().rstrip()
if not head:
return
seq, _, qual = fh.readline().rstrip(), fh.readline(), fh.readline().rstrip()
if not head.startswith("@") or len(seq) != len(qual):
raise ValueError(f"malformed record starting {head[:30]!r}")
yield head, seq, qual
def mean_quality(qual, offset=33):
return sum(ord(c) - offset for c in qual) / len(qual)
def cmd_stats(args):
n = bases = 0
total_q = 0.0
with open_text(args.fastq) as fh:
for _, seq, qual in records(fh):
n += 1
bases += len(seq)
total_q += mean_quality(qual)
print(f"reads\t{n}\nbases\t{bases}\nmean_quality\t{total_q / n:.2f}" if n else "reads\t0")
return 0
def cmd_filter(args):
kept = total = 0
with open_text(args.fastq) as fh, open(args.out, "w") as out:
for head, seq, qual in records(fh):
total += 1
if mean_quality(qual) >= args.min_quality and len(seq) >= args.min_length:
out.write(f"{head}\n{seq}\n+\n{qual}\n")
kept += 1
log.info("kept %d of %d reads", kept, total)
return 0
def build_parser():
p = argparse.ArgumentParser(prog="fqtool", description=__doc__)
p.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
p.add_argument("-v", "--verbose", action="store_true", help="log progress to standard error")
sub = p.add_subparsers(dest="command", required=True)
s = sub.add_parser("stats", help="count reads and bases, mean quality")
s.add_argument("fastq")
s.set_defaults(func=cmd_stats)
f = sub.add_parser("filter", help="keep reads that pass a quality and length threshold")
f.add_argument("fastq")
f.add_argument("-o", "--out", required=True)
f.add_argument("-q", "--min-quality", type=float, default=30, help="minimum mean Phred quality (default 30)")
f.add_argument("-l", "--min-length", type=int, default=50, help="minimum read length (default 50)")
f.set_defaults(func=cmd_filter)
return p
def main(argv=None):
args = build_parser().parse_args(argv)
logging.basicConfig(level=logging.INFO if args.verbose else logging.WARNING, format="%(levelname)s %(message)s")
try:
return args.func(args)
except (OSError, ValueError) as e:
print(f"fqtool: error: {e}", file=sys.stderr)
return 1
if __name__ == "__main__":
sys.exit(main())
$ python3 fqtool.py --help | head -14
usage: fqtool [-h] [--version] [-v] {stats,filter} ...
fqtool: summarise or filter a FASTQ file by quality.
positional arguments:
{stats,filter}
stats count reads and bases, mean quality
filter keep reads that pass a quality and length threshold
options:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbose log progress to standard error
$ python3 fqtool.py stats ../data/S1.fq
reads 6000
bases 600000
mean_quality 33.55
$ python3 fqtool.py -v filter ../data/S1.fq -o /tmp/pyproj_q34.fq -q 34 -l 100 2>&1; grep -c '^@S1' /tmp/pyproj_q34.fq || true
INFO kept 229 of 6000 reads
229
$ python3 fqtool.py stats missing.fq 2>&1; echo "exit=$?"; python3 fqtool.py 2>&1 | tail -1; echo "exit=${PIPESTATUS[0]}"
fqtool: error: [Errno 2] No such file or directory: 'missing.fq'
exit=1
fqtool: error: the following arguments are required: command
exit=2
The mean quality, 33.55, is the figure that awk and Biopython gave on the same file. The filter at a mean quality of 34 kept 229 of the 6,000 reads and logged that at the INFO level because of -v. Without -v it is silent. A missing file gives exit status 1 and a one-line message; a missing subcommand gives status 2, which is what argparse uses for a usage error. Such conventions are what let other programs build on yours.
A test is a small program that runs your code on an input whose answer you know and checks the result. pytest finds functions named test_... and reports which pass. Its three most useful tools are shown in the suite below: parametrize runs one test on several inputs, tmp_path gives a temporary directory so that tests never touch real data, and capsys captures what the program prints. Test the cases that matter: normal input, a defective file, a missing file and the edge of a threshold. The values in test_mean_quality are checked by hand: the characters +, 5, ? and I are Q10, Q20, Q30 and Q40, whose mean is 25.
import pytest
import fqtool
FQ = "@r1\nACGT\n+\nIIII\n@r2\nACGT\n+\n!!!!\n" # one read of Q40, one of Q0
@pytest.mark.parametrize("qual, expected", [("IIII", 40.0), ("!!!!", 0.0), ("5555", 20.0), ("+5?I", 25.0)])
def test_mean_quality(qual, expected):
# '+' = Q10, '5' = Q20, '?' = Q30, 'I' = Q40, so "+5?I" has the mean (10 + 20 + 30 + 40) / 4 = 25
assert fqtool.mean_quality(qual) == pytest.approx(expected)
def test_records_reads_four_lines_at_a_time(tmp_path):
p = tmp_path / "x.fq"
p.write_text(FQ)
with open(p) as fh:
recs = list(fqtool.records(fh))
assert [r[0] for r in recs] == ["@r1", "@r2"] and recs[0][1] == "ACGT"
def test_truncated_record_is_an_error(tmp_path):
p = tmp_path / "bad.fq"
p.write_text("@r1\nACGT\n+\nII\n") # quality shorter than sequence
with open(p) as fh, pytest.raises(ValueError):
list(fqtool.records(fh))
def test_stats_command(tmp_path, capsys):
p = tmp_path / "x.fq"
p.write_text(FQ)
assert fqtool.main(["stats", str(p)]) == 0
out = capsys.readouterr().out
assert "reads\t2" in out and "mean_quality\t20.00" in out
def test_filter_keeps_only_good_reads(tmp_path):
src, dst = tmp_path / "x.fq", tmp_path / "y.fq"
src.write_text(FQ)
assert fqtool.main(["filter", str(src), "-o", str(dst), "-q", "30", "-l", "4"]) == 0
assert dst.read_text().startswith("@r1") and "@r2" not in dst.read_text()
def test_missing_file_gives_exit_status_1(tmp_path, capsys):
assert fqtool.main(["stats", str(tmp_path / "nope.fq")]) == 1
assert "error" in capsys.readouterr().err
$ python3 -m pytest -q test_fqtool.py 2>&1 | tail -6 ......... [100%] 9 passed in 0.02s
A passing suite proves little until you have seen it fail. To check that these tests have teeth, we planted a plausible bug: the quality offset set to 64 (the old Illumina convention) instead of 33. A copy of the tool with that one change was run against the same tests:
$ rm -rf /tmp/bugdemo && mkdir /tmp/bugdemo && cp test_fqtool.py /tmp/bugdemo/ && sed 's/offset=33/offset=64/' fqtool.py > /tmp/bugdemo/fqtool.py
$ cd /tmp/bugdemo && python3 -m pytest -q 2>&1 | grep -E '^FAILED|^[0-9]+ (passed|failed)' | cut -c1-140
FAILED test_fqtool.py::test_mean_quality[IIII-40.0] - assert 9.0 == 40.0 ± 4....
FAILED test_fqtool.py::test_mean_quality[!!!!-0.0] - assert -31.0 == 0.0 ± 1....
FAILED test_fqtool.py::test_mean_quality[5555-20.0] - assert -11.0 == 20.0 ± ...
FAILED test_fqtool.py::test_mean_quality[+5?I-25.0] - assert -6.0 == 25.0 ± 2...
FAILED test_fqtool.py::test_stats_command - AssertionError: assert ('reads\t2...
FAILED test_fqtool.py::test_filter_keeps_only_good_reads - AssertionError: as...
6 failed, 3 passed in 0.06s
Six of the nine tests fail, and the messages show the symptom (a mean quality of 9.0 where 40.0 was expected, and negative qualities). A suite that stays green after a change of that size would not be worth having. Writing the test before the code (test-driven development) and running the whole suite after every change are the habits that make a tool safe to modify (Youens-Clark, Ch. 4; Wilson et al. 2017).
Three rules cover most speed problems. Vectorise: operate on whole columns (c.sum(axis=0)), not row by row in a Python loop. Stream: process a large file one record at a time rather than loading it. Measure before optimising: time a run, and if a step is slow, check whether a compiled tool (, ) already does it. Then think about parallelism.
Measure before you optimise, and measure with the right tool. timeit runs a function several times and reports the average. The example computes the mean quality of the 6,000 reads in two ways: a Python loop over the characters, and NumPy, which treats the whole quality block as one array of bytes and does the arithmetic in compiled code:
import sys, timeit
import numpy as np
sys.path.insert(0, ".")
from fqtool import records
with open("../data/S1.fq") as fh:
quals = [q for _, _, q in records(fh)]
def pure():
return sum(sum(ord(c) - 33 for c in q) / len(q) for q in quals) / len(quals)
arr = np.frombuffer("".join(quals).encode(), dtype=np.uint8).reshape(len(quals), -1)
def vector():
return float((arr.astype(np.float64) - 33).mean())
t1 = timeit.timeit(pure, number=3) / 3
t2 = timeit.timeit(vector, number=3) / 3
print(f"pure Python {t1 * 1000:.0f} ms, NumPy {t2 * 1000:.1f} ms ({t1 / t2:.0f} times faster); same answer: {abs(pure() - vector()) < 1e-9}")
# output
pure Python 33 ms, NumPy 3.1 ms (11 times faster); same answer: True
NumPy was about 11 times faster here (33 ms against 3.1 ms; your numbers will differ with the machine) and gave the same answer. The condition is that the data have a regular shape: all reads here have 100 bases, so the block reshapes into 6,000 rows and 100 columns. For reads of unequal length you would pad or work per read. The lesson is general: a loop over millions of items in Python is the first place to look when a program is slow, and a vectorised expression is often the cure.
That the sample order in the metadata equals that of the count table. Assert list(counts.columns) == list(meta.index), or reindex the metadata by the count columns.
A key appears more than once on at least one side, so rows were multiplied. Check df["key"].duplicated().sum() on both tables and use validate="one_to_one".
1 in 1,000 (10−30/10), an accuracy of 99.9%.
They record what the tool is supposed to do, catch a regression when you change it, and let a reader trust the numbers without redoing the work.
R is the language of statistics and of Bioconductor, the collection of packages in which most expression, variant and microbiome methods were first published. It thinks in vectors and tables, which suits biological data. This chapter teaches the language with the same small count table used in the Python chapters, so that you can compare the two. All code was run in R 4.6 and the printed output is real.
Everything in R is a vector or built from vectors. Operations work on every element at once, and elements can have names:
> x <- c(10, 0, 5); names(x) <- c("s1", "s2", "s3")
> x[x > 0]
s1 s3
10 5
> x / sum(x)
s1 s2 s3
0.6666667 0.0000000 0.3333333
> 1:6 + c(10, 20) # the shorter vector is recycled
[1] 11 22 13 24 15 26
Indexing starts at 1 (unlike Python), and x[a:b] includes both ends. Recycling is convenient and dangerous: if the lengths do not divide evenly, R warns, but when they do, it silently repeats. NA is a missing value, and it is contagious: mean(c(1, NA, 3)) is NA; mean(c(1, NA, 3), na.rm = TRUE) is 2. Decide what a missing value means before you remove it.
A factor is a categorical variable with a fixed set of levels. Statistical models use the first level as the reference and report every other level against it, so the order decides the direction of every coefficient and fold change. By default levels are alphabetical:
> g <- factor(c("treated", "control", "treated")); levels(g)
[1] "control" "treated"
> levels(relevel(g, ref = "treated"))
[1] "treated" "control"
With control as the reference a positive coefficient means "higher in treated"; relevel it, and the same data give the opposite sign. If your fold changes have the sign opposite to the paper, check this first.
A data frame is a table whose columns are vectors of equal length. Read, look at its structure, check it:
> counts <- read.delim("counts.tsv", row.names = 1, check.names = FALSE)
> meta <- read.delim("meta.tsv", row.names = 1)
> str(meta)
'data.frame': 3 obs. of 1 variable:
$ group: chr "control" "treated" "treated"
> colSums(counts)
s1 s2 s3
210 180 218
> rowSums(counts > 0)
geneA geneB geneC
2 3 1
> all(colnames(counts) == rownames(meta))
[1] TRUE
As in Python, the last line is the one to keep: samples in the count table and in the metadata must be the same and in the same order. Relative abundance, filtering and subsetting use the same ideas as before:
> rel <- sweep(counts, 2, colSums(counts), "/") * 100 # percent of each sample
> round(rel, 1)
s1 s2 s3
geneA 4.8 0 2.3
geneB 95.2 100 96.3
geneC 0.0 0 1.4
> counts[rowSums(counts) > 5, ] # genes with more than 5 reads in total
s1 s2 s3
geneA 10 0 5
geneB 200 180 210
> subset(meta, group == "treated")
group
s2 treated
s3 treated
> tapply(as.numeric(counts["geneA", ]), meta$group, mean)
control treated
10.0 2.5
apply(counts, 1, f) applies a function to every row (2 for columns), and sapply(counts, sum) to every column of a data frame. They return vectors, so they are the natural way to compute a statistic per gene or per sample. The tidyverse packages (dplyr, tidyr, readr) offer a different, very readable grammar (filter, mutate, group_by, summarise) built on the tidy-data idea that every variable is a column and every observation a row (Wickham, 2014). Base R is enough for everything in this chapter and is always installed; learn tidyverse once you are comfortable with it.
gc <- function(s) {
b <- strsplit(toupper(s), "")[[1]]
mean(b %in% c("G", "C"))
}
round(gc("ATGCGTACGTTAGC"), 3) # 0.5
for (i in 1:3) if (i %% 2 == 1) cat("odd", i, "\n") # odd 1 / odd 3
A function returns its last value. Give arguments sensible defaults, check inputs with stopifnot(), and avoid loops over rows when a vectorised operation exists: colSums(counts) is both shorter and much faster than a loop.
Compare two groups with a t test (assumes roughly normal values) or the Wilcoxon rank-sum test (compares ranks; use it for skewed data or small groups). With eight values per group drawn at random from two distributions whose means differ by 1, one run gave:
> set.seed(1); a <- rnorm(8, 10, 1); b <- rnorm(8, 11, 1) > round(t.test(a, b)$p.value, 4) [1] 0.0938 > round(wilcox.test(a, b)$p.value, 4) [1] 0.1049
A true difference of one standard deviation was not significant with eight samples per group: the test lacks power. That is the lesson of every small study: a result that is not significant is not evidence that there is no difference. Plan the sample size before the experiment, not after.
When you test thousands of genes or taxa at once, about 5% of the true nulls will have p < 0.05 by chance. Correct for it. p.adjust(p, "BH") applies the Benjamini–Hochberg procedure, which controls the false-discovery rate (Benjamini and Hochberg, 1995):
> p <- c(0.001, 0.02, 0.04, 0.3) > round(p.adjust(p, "BH"), 4) [1] 0.0040 0.0400 0.0533 0.3000
The third gene, at p = 0.04, is no longer below 0.05 after the correction (adjusted 0.0533). Report the adjusted value and say which correction you used.
The reference level of the factor: levels are alphabetical, so "disease" came first and became the reference. Use relevel(x, ref = "healthy").
No. It shows that this study could not detect a difference of this size. Power depends on the sample size, the variability and the effect.
Not without correction: at a 5% threshold, about 1,000 null genes are expected below it by chance. Use the adjusted value.
1:6 + c(10, 20) return, and why is it risky?11 22 13 24 15 26: the shorter vector is recycled. If the lengths were mismatched by mistake, you could get a wrong answer without an error.
This chapter applies R to a typical community or expression dataset: a samples-by-features table, a grouping, and the questions "are the groups different overall?", "which features differ?" and "how do I show it?". The code that was run here uses base R, vegan and ggplot2; the Bioconductor workflows are shown as a map, and the parts that were not run are marked.
The data are simulated so that the truth is known: 12 samples, 8 taxa, two groups of 6, and three taxa made three times more abundant in the treated group. Samples are rows here, as vegan expects.
library(vegan); library(ggplot2)
set.seed(7)
n <- 12
grp <- factor(rep(c("control", "treated"), each = n / 2))
otu <- matrix(rpois(n * 8, lambda = 20), nrow = n)
otu[grp == "treated", 1:3] <- otu[grp == "treated", 1:3] * 3
rownames(otu) <- paste0("S", 1:n); colnames(otu) <- paste0("taxon", 1:8)
rel <- otu / rowSums(otu) # relative abundance
d <- vegdist(rel, "bray") # Bray-Curtis dissimilarity
adonis2(d ~ grp, permutations = 999) # PERMANOVA
The output of the PERMANOVA in this run:
Df SumOfSqs R2 F Pr(>F) Model 1 0.18742 0.79231 38.148 0.002 ** Residual 10 0.04913 0.20769 Total 11 0.23655 1.00000
Read the table. R2 is the share of the total variation in the distances explained by the grouping: here 79%, which is large because the effect was planted and strong. F compares the variation between groups with the variation within them, and Pr(>F) is the share of 999 random relabelings of the samples that gave an F at least this large; with 999 permutations the smallest possible p is 0.001. In real microbiome data a single factor usually explains much less, often 2–15%.
PERMANOVA assumes the groups have similar spread. A significant result can come from one group being more variable, not from a different centre. Test it with betadisper(d, grp) and permutest(), and look at the ordination before believing the p value.
Principal coordinates analysis (PCoA) places the samples in two dimensions so that distances are preserved as well as possible. cmdscale(d, k = 2) does it in base R; ggplot2 draws it:
pc <- cmdscale(d, k = 2)
df <- data.frame(PC1 = pc[, 1], PC2 = pc[, 2], group = grp)
p <- ggplot(df, aes(PC1, PC2, colour = group)) + geom_point(size = 3) + theme_minimal()
ggsave("pcoa.pdf", p, width = 5, height = 4)
Save figures as PDF or SVG (vector graphics) for publication, so that they stay sharp at any size, and as PNG for the web. In the R installation used for this chapter the default PNG device failed with a "graphics API version mismatch" error, while the PDF device worked; if you meet the same, install the ragg package and use ggsave(..., device = ragg::agg_png), or save a PDF. Colour choices matter: do not use red and green together, and label the axes with units. Show the data points, not only a mean, so that readers see the spread.
Diversity within a sample is summarised by richness (how many taxa), the Shannon index (richness and evenness) and others. diversity(otu, "shannon") gives the Shannon index per sample; tapply(sh, grp, median) summarises by group. In this run the medians were 2.063 (control) and 1.919 (treated): the treatment made three taxa more abundant, so the community became less even and its Shannon index fell, even though no taxon was lost. Diversity indices answer different questions, so choose them for the question, report more than one, and do not compare richness between samples with very different sequencing depths without rarefying or standardising by coverage.
The overall test says that groups differ; the next question is which features differ. The methods fall into two families. For count data from RNA-seq, models for counts with a dispersion estimate: and (Love et al., 2014). For compositional microbiome data, methods that respect the fact that abundances are proportions, such as ANCOM-BC, ALDEx2 or a model on centred log-ratios. These methods often disagree on microbiome data, so use more than one and report what several agree on. The shape of a analysis, shown here but not run in this chapter because the package is not installed on the machine where the examples were tested:
# not run here: requires Bioconductor's
library()
dds <- DESeqDataSetFromMatrix(countData = counts, # genes x samples, raw integer counts
colData = meta, # samples x variables, same order
design = ~ group)
dds$group <- relevel(dds$group, ref = "control") # the reference level decides the sign
dds <- DESeq(dds)
res <- results(dds, alpha = 0.05) # adjusted p values (Benjamini-Hochberg)
summary(res)
Three rules for such a workflow. Give the model raw counts, not normalised values or percentages: it normalises internally and needs the counts to estimate noise. Put the factor of interest last in the design, and any batch or covariate before it. And look at the results before trusting them: a plot of log fold change against mean abundance (an MA plot), the distribution of p values (flat with a peak near zero is healthy), and a PCA of the samples to find outliers and batch effects.
Bioconductor is a repository of R packages for genomic data that share common data structures, so that packages can work together (Huber et al., 2015). Install it with BiocManager::install(), and record the version, since packages and R versions move together. The structures to know:
| Structure | Holds | Used for |
|---|---|---|
SummarizedExperiment | a count matrix, row annotations (genes) and column annotations (samples) kept together | RNA-seq and most omics; subsetting samples subsets the annotations too, so they cannot drift apart |
GRanges | genomic intervals with strand and metadata | peaks, genes, variants; overlaps and distances |
DNAStringSet (Biostrings) | many DNA sequences | sequence work, pattern matching, translation |
phyloseq | an OTU table, taxonomy, sample data and a tree | microbiome analysis (McMurdie and Holmes, 2013) |
The value of these structures is the same as that of the sample-match assertion earlier: the data and their annotations travel together, so that a subset of the samples is a subset of the metadata as well.
Put the analysis in a script or a Quarto or R Markdown document that reads the raw files and produces every figure and table, so that the whole analysis reruns with one command. Fix the random seed (set.seed()) wherever randomness is used (permutations, subsampling), and finish the document with sessionInfo(), which records the versions of R and of every package. Save the figures as files, not as screenshots. A reviewer, or you in a year, can then check every number (Sandve et al., 2013).
The groups differ in their community composition, and the grouping explains about 8% of the variation between samples: a modest but reliable effect. Check the dispersion before concluding that the centres differ.
It models counts with a noise (dispersion) estimate that depends on the count scale, and normalises internally by size factors. Percentages or transformed values break these assumptions.
betadisper is significant. What does that mean?The groups differ in their spread, so the PERMANOVA result may reflect different variability and not different centres. Report both, and interpret the figure with that in mind.
SummarizedExperiment over a bare matrix?The counts, the gene annotations and the sample table are one object: subsetting samples subsets the sample table too, so they cannot become misaligned.
Biopython is the toolbox for the biological objects of Python: a sequence that knows its alphabet and its genetic code, a record that carries annotations and features, parsers for every common file format, pairwise aligners, motif models and phylogenetic trees. This chapter goes through it object by object. Every snippet was run, mostly on the real files of the MicroRes demonstration data (377 representative 16S sequences of 253 bases and their tree), so the numbers quoted are real.
A Seq behaves like a Python string (it has a length, slices, find, count) but also knows what it is: it can be complemented, reverse-complemented, transcribed and translated. Two properties matter in daily work. A Seq is immutable, like a string; use MutableSeq if you must change it in place. And its methods do not guess: a character outside the alphabet is an error or an ambiguity code, never silently changed.
from Bio.Seq import Seq
s = Seq("ATGGCCATTGTAATGGGCCGCTGA")
s.reverse_complement() # Seq('TCAGCGGCCCATTACAATGGCCAT')
s.translate() # Seq('MAIVMGR*')
s.translate(to_stop=True) # Seq('MAIVMGR')
s.transcribe() # DNA to RNA: T becomes U
Seq("ATGNNNCGT").translate() # Seq('MXR') NNN is an unknown codon, translated as X
Seq("ACGTRYN").reverse_complement() # Seq('NRYACGT') ambiguity codes are complemented correctly
Ambiguity codes (R = A or G, Y = C or T, N = any) are real: they appear in assemblies and in consensus sequences. A hand-made complement table that only knows A, C, G and T fails on them; Seq does not. The reverse complement matters because DNA is double-stranded: a gene can be on either strand, and a read from the other strand is the reverse complement of the gene.
Translation depends on four things that must be right. The frame: a coding sequence starts at a start codon and is read in triplets; starting one base off gives an unrelated protein. The genetic code table: most bacteria use table 11 (the standard code, with alternative start codons such as TTG and CTG); Mycoplasma reads TGA as tryptophan (table 4); mitochondria use their own tables. The stop codon: use to_stop=True to cut the protein at the first stop. And the length: a length that is not a multiple of three raises a warning, because a frameshift or a truncated record is likely.
from Bio.Data import CodonTable
t = CodonTable.unambiguous_dna_by_id[11]
t.start_codons[:3] # ['TTG', 'CTG', 'ATT']
t.stop_codons # ['TAA', 'TAG', 'TGA']
Seq("ATGGCCTGA").translate(table=4) # Seq('MAW'): in table 4, TGA is tryptophan, not a stop
The last line is a classic trap: the same DNA gives a 2-residue protein with a stop under table 11 and a 3-residue protein under table 4. Always state the table in a methods section. An internal stop codon (*) in a gene that should be intact means one of: a pseudogene, a sequencing or assembly error (a frameshift), the wrong frame, or the wrong table.
A SeqRecord wraps a Seq with an identifier (id), a name, a description, free-form annotations, per-letter annotations (the quality scores of a FASTQ read) and a list of features. A SeqFeature is one annotated region: a type (CDS, gene, rRNA), a FeatureLocation with start, end and strand, and qualifiers (product, locus_tag). This is exactly what a file holds, and what a genome annotation produces.
from Bio.SeqRecord import SeqRecord
from Bio.SeqFeature import SeqFeature, FeatureLocation
r = SeqRecord(Seq("ATGGCCATTGTAATGGGCCGCTGAAAGGGTGCCCGATAG"), id="demo1", description="toy gene")
r.features.append(SeqFeature(FeatureLocation(0, 24, strand=1), type="CDS", qualifiers={"product": "toy protein"}))
cds = r.features[0].extract(r.seq) # Seq('ATGGCCATTGTAATGGGCCGCTGA')
cds.translate(table=11, to_stop=True) # Seq('MAIVMGR')
f = SeqFeature(FeatureLocation(10, 31, strand=-1), type="gene")
f.extract(r.seq) # Seq('CACCCTTTCAGCGGCCCATTA'): the reverse complement of bases 10 to 30
Coordinates are the main source of errors. Biopython uses Python's convention: 0-based and half-open, so FeatureLocation(0, 24) covers the first 24 bases, positions 1 to 24 in the 1-based, closed convention of and GFF. When you read coordinates from a GFF or a table, convert them: a GFF feature at 101 to 200 is FeatureLocation(100, 200). On the minus strand, extract returns the reverse complement for you, which is why a gene on the minus strand is read correctly without extra code. Writing the record out as (SeqIO.write(r, handle, ""), after setting r.annotations["molecule_type"] = "DNA") produced a file starting LOCUS demo1 39 bp DNA.
SeqIO.parse(file, format) returns an iterator of records: one record in memory at a time, so a file of any size can be processed. SeqIO.read expects exactly one record. SeqIO.write(records, file, format) writes, and SeqIO.convert converts between formats. The formats you will meet: fasta, fastq (Sanger/Illumina 1.8+), , embl, gff (through a helper library), stockholm and clustal for alignments.
On the 377 representative sequences of the demonstration data:
from Bio import SeqIO
from Bio.SeqUtils import gc_fraction
recs = list(SeqIO.parse("sequences.fasta", "fasta"))
len(recs), recs[0].id, len(recs[0]) # (377, 'ASV0001', 253)
lens = [len(r) for r in recs]
min(lens), max(lens) # (253, 253): every sequence has the same length
round(sum(gc_fraction(r.seq) for r in recs) / len(recs), 3) # 0.519: mean GC content
Three techniques make large files manageable:
SeqIO.write((r for r in SeqIO.parse(f, "fasta") if len(r) >= 500), out, "fasta") never holds the file in memory.idx = SeqIO.index("sequences.fasta", "fasta") gives random access by identifier without loading the sequences: len(idx) was 377 and idx["ASV0002"] returns that record. Use it when you need to pull a few thousand sequences out of a genome-sized file. Close it with idx.close().gzip.open(path, "rt") and pass the handle to parse.For FASTQ, each record carries per-base quality in rec.letter_annotations["phred_quality"], a list of integers. A Phred quality Q is an error probability of 10−Q/10. Trimming by quality, filtering by mean quality and counting bases below Q20 are all list operations on that field; for large datasets do this in a compiled tool (, ) and use Biopython for the analysis that follows.
Bio.SeqUtils has the common calculations: gc_fraction(seq), molecular_weight(seq) (for the 14-base example, 4358.8), seq1 and seq3 to convert between one- and three-letter amino-acid codes, and MeltingTemp.Tm_NN for the melting temperature of a primer by the nearest-neighbour method (40.8 °C for ATGCGTACGTTAGC with default salt conditions). Treat a computed Tm as a guide: the value depends on salt and primer concentration, which you must set to your reaction (Na=, dnac1=).
A pairwise alignment finds the arrangement of two sequences that maximises a score (see the alignment chapter of this handbook). PairwiseAligner does global (Needleman–Wunsch) and local (Smith–Waterman) alignment with an affine gap penalty, and for proteins accepts a substitution matrix.
from Bio.Align import PairwiseAligner, substitution_matrices
al = PairwiseAligner(mode="global", match_score=1, mismatch_score=-1, open_gap_score=-2, extend_gap_score=-0.5)
a = al.align("GATTACA", "GCATGCA")[0]
a.score # 1.0
print(a)
# target 0 GATTACA 7
# 0 |..|.|| 7
# query 0 GCATGCA 7
bl = PairwiseAligner(substitution_matrix=substitution_matrices.load("BLOSUM62"), open_gap_score=-10, extend_gap_score=-0.5)
bl.align("HEAGAWGHEE", "PAWHEAE")[0].score # 4.0
Read the output: bars mark identical letters, dots mismatches, and the numbers are positions. The scores are only comparable under the same scoring scheme; a score of 4.0 under BLOSUM62 is not "better" than 1.0 under a match/mismatch scheme. For two protein sequences always use a substitution matrix (BLOSUM62 for moderately distant, BLOSUM80 for close, BLOSUM45 for distant sequences), not match/mismatch scores. For many sequences use a dedicated multiple-alignment program (, MUSCLE) and read the result with AlignIO; Biopython aligns pairs, it is not a multiple-alignment engine.
A motif such as a promoter box or a binding site is a family of similar short sequences. Bio.motifs counts the letters at each position, builds a position weight matrix (PWM) and scores new sequences against it.
from Bio import motifs
m = motifs.create([Seq("TATAAT"), Seq("TATGAT"), Seq("TACAAT"), Seq("TATAAT")])
print(m.counts)
# 0 1 2 3 4 5
# A: 0.00 4.00 0.00 3.00 4.00 0.00
# C: 0.00 0.00 1.00 0.00 0.00 0.00
# G: 0.00 0.00 0.00 1.00 0.00 0.00
# T: 4.00 0.00 3.00 0.00 0.00 4.00
m.consensus # Seq('TATAAT')
pwm = m.counts.normalize(pseudocounts=0.5)
round(pwm.log_odds()["T"][0], 2) # 1.58 (log2 odds of T at position 0)
Two cautions. Counts from four sites are a very poor estimate: the pseudocount keeps a letter that was never seen from getting probability zero, and with so few examples every score is uncertain. And a PWM scan of a genome finds many matches by chance; a threshold must be justified against the background (pwm.pssm and its distribution), and a hit is a candidate, not a binding site.
Bio.Phylo reads Newick, NEXUS and PhyloXML trees and offers traversal, pruning, rooting and drawing. On the tree that comes with the MicroRes demonstration data:
from Bio import Phylo
tree = Phylo.read("tree.nwk", "newick")
tree.count_terminals() # 377: one leaf per sequence
round(tree.total_branch_length(), 3) # 19.465
sorted(t.name for t in tree.get_terminals())[:3] # ['ASV0001', 'ASV0002', 'ASV0003']
tree.root_at_midpoint() # root the tree
Phylo.draw_ascii(subtree) # a text drawing, useful on a server
Typical tasks: tree.common_ancestor("a", "b") gives the smallest clade that contains two leaves; tree.distance("a", "b") the path length between them; tree.prune(leaf) removes a leaf; Phylo.write(tree, "out.nwk", "newick") saves it. A tree is a hypothesis built from an alignment under a model; do not read branch lengths between distantly related leaves as time without a calibrated model.
Bio.Entrez talks to 's E-utilities: search (esearch), fetch (efetch), link (elink). asks every user to give an e-mail address and to respect a limit of three requests per second without an API key. A complete, live example, with its output, is in the section on records below.
For , run the program on your machine or a server with -outfmt 6 (tab-separated) and read the table with pandas (columns: query, subject, percent identity, alignment length, mismatches, gap opens, query start and end, subject start and end, e-value, bit score). Bio.SearchIO parses XML and many other search formats into query, hit and HSP objects when you need the alignments themselves (an example follows). Calling 's web from a script is slow and rate-limited; it is not a way to search thousands of sequences.
This section and those that follow continue with the files of chapter P12: the phiX174 genome, the simulated reads S1.fq, the gene table, the hits. The first question is the one asked in the shell: how long is the genome, what is its GC content, and how good are the reads? gc_fraction is the current name of the function that older books call GC (it returns a fraction between 0 and 1, not a percentage).
from Bio import SeqIO
from Bio.SeqUtils import gc_fraction
rec = SeqIO.read("data/phix174.fa", "fasta")
print(rec.id, len(rec.seq), f"GC {100 * gc_fraction(rec.seq):.2f}%")
print(rec.description)
# output
gi|9626372|ref|NC_001422.1| 5386 GC 44.71%
gi|9626372|ref|NC_001422.1| Coliphage phiX174, complete genome
For a FASTQ file, SeqIO decodes the quality string into a list of integers in letter_annotations["phred_quality"], with the Sanger offset of 33. That makes the arithmetic of chapter P12 a few lines of Python, and gives a way to settle a question left open there: which quantity does a tool report as "average quality"? The program below computes the mean of the read means (the number our awk gave), the share of bases at Q30 or better, and the quality that corresponds to the mean error probability.
from math import log10
from Bio import SeqIO
n = q_sum = bases = q30 = 0
perr = 0.0
for r in SeqIO.parse("data/S1.fq", "fastq"):
q = r.letter_annotations["phred_quality"] # decoded with the Sanger offset of 33
n += 1
q_sum += sum(q) / len(q)
bases += len(q)
q30 += sum(x >= 30 for x in q)
perr += sum(10 ** (-x / 10) for x in q)
print(f"{n} reads; mean of the read means {q_sum / n:.2f}")
print(f"bases at Q30 or better: {100 * q30 / bases:.2f}%")
print(f"quality from the mean error probability: {-10 * log10(perr / bases):.2f}")
# output
6000 reads; mean of the read means 33.55
bases at Q30 or better: 80.87%
quality from the mean error probability: 31.24
The first value, 33.55, is what awk printed. The last, 31.24, is exactly what stats -a reports as AvgQual. The two differ because a quality score is a logarithm: a few bad bases (Q10, error 0.1) raise the mean error probability far more than they lower the mean score. The error-probability average is the better summary of how many errors to expect, and it is the one to quote. Seeing two tools disagree and then reproducing both numbers is the quickest way to learn what each of them means.
A gene table becomes proteins in four steps: convert the 1-based GFF coordinates to Biopython's 0-based half-open ones, build a SeqFeature with a FeatureLocation that carries the strand, extract the sequence (which reverse-complements a minus-strand feature), and translate with the right genetic code. The code below does it for the seven genes of chapter P12 and prints three checks for each: the length in bases, whether the protein ends with a stop, and whether there is a stop inside.
from Bio import SeqIO
from Bio.Seq import Seq
from Bio.SeqFeature import SeqFeature, FeatureLocation
genome = SeqIO.read("data/phix174.fa", "fasta").seq
for line in open("data/genes.gff"):
if line.startswith("#"):
continue
f = line.rstrip("\n").split("\t")
start, end, strand = int(f[3]), int(f[4]), 1 if f[6] == "+" else -1
gid = f[8].split(";")[0].replace("ID=", "")
feat = SeqFeature(FeatureLocation(start - 1, end, strand=strand), type="CDS") # GFF is 1-based, Biopython 0-based
dna = feat.extract(genome)
prot = dna.translate(table=11)
stop = prot.endswith("*")
print(gid, f"{len(dna):5d} bp", f"{len(prot) - stop:4d} aa", "ends with stop" if stop else "NO final stop", "internal stops:", str(prot[:-1]).count("*"), str(prot[:8]))
# output
1_1 171 bp 56 aa ends with stop internal stops: 0 MSRKIILI
1_2 459 bp 152 aa ends with stop internal stops: 0 MSQVTEQS
1_3 117 bp 38 aa ends with stop internal stops: 0 MSKGKKRS
1_4 1284 bp 427 aa ends with stop internal stops: 0 MSNIQTGA
1_5 528 bp 175 aa ends with stop internal stops: 0 MFQTFISR
1_6 987 bp 328 aa ends with stop internal stops: 0 MFGAIAGG
1_7 1404 bp 468 aa NO final stop internal stops: 0 MVRSYYPS
Six genes end with a stop and have no internal stop, as a complete gene should. The seventh, 1_7, has no final stop. Chapter P12 showed the same gene with and put it down to a gene that is cut by the end of the linear sequence. The next section, with the record, shows that this reading was right.
An open reading frame (ORF) is a stretch with no stop codon in one of the six reading frames (three per strand). The simplest gene finder takes every long ORF and starts it at the first methionine after the previous stop. It needs ten lines of code, and it shows both how far a rule goes and where it stops.
from Bio import SeqIO
genome = SeqIO.read("data/phix174.fa", "fasta").seq
found = []
for strand, s in ((1, genome), (-1, genome.reverse_complement())):
for frame in range(3):
prot = str(s[frame:len(s) - (len(s) - frame) % 3].translate(table=11))
pos = 0
for stretch in prot.split("*"):
m = stretch.find("M") # an ORF starts at the first Met after a stop
if m >= 0 and len(stretch) - m >= 100:
aa_start = pos + m
found.append((strand, frame, frame + 3 * aa_start, len(stretch) - m))
pos += len(stretch) + 1
print(len(found), "ORFs of at least 100 amino acids, six frames")
for strand, frame, nt, aa in sorted(found, key=lambda x: -x[3])[:5]:
where = nt + 1 if strand == 1 else len(genome) - nt
print(f"strand {strand:+d} frame {frame} starts at {where} {aa} aa")
# output
10 ORFs of at least 100 amino acids, six frames
strand +1 frame 2 starts at 3954 477 aa
strand +1 frame 1 starts at 1001 427 aa
strand +1 frame 2 starts at 2931 328 aa
strand +1 frame 0 starts at 3076 202 aa
strand -1 frame 0 starts at 1651 185 aa
The longest ORFs coincide with real genes (the ORFs at 1,001 and 2,931 are genes 1_4 and 1_6), but the rule starts the one at 3,954 at the first methionine after the stop, 27 bases before the start that chose (3,981). A rule cannot tell which of several in-frame ATGs is the true start; a gene finder uses the ribosome-binding site and the codon statistics of the organism to decide. The rule also reports ORFs that are only chance stretches without stops, such as the one on the minus strand. Use the rule to understand; use a predictor for results, and a curated annotation when there is one.
Bio.SeqUtils.ProtParam.ProteinAnalysis computes properties from a protein sequence: molecular weight, isoelectric point (the pH at which the protein carries no net charge), the GRAVY score (the mean hydropathy of Kyte and Doolittle: positive for hydrophobic, negative for hydrophilic) and the instability index (Guruprasad et al.: above 40 predicts an unstable protein in a test tube). The values are predictions from sequence, useful for choosing a purification pH or spotting a membrane protein, not measurements.
from Bio import SeqIO
from Bio.SeqFeature import SeqFeature, FeatureLocation
from Bio.SeqUtils.ProtParam import ProteinAnalysis
genome = SeqIO.read("data/phix174.fa", "fasta").seq
print("gene aa MW(kDa) pI GRAVY instability")
for line in open("data/genes.gff"):
if line.startswith("#"):
continue
f = line.split("\t")
gid = f[8].split(";")[0].replace("ID=", "")
cds = SeqFeature(FeatureLocation(int(f[3]) - 1, int(f[4]), strand=1), type="CDS").extract(genome)
prot = str(cds.translate(table=11)).rstrip("*").replace("*", "")
pa = ProteinAnalysis(prot)
print(f"{gid:7s} {len(prot):4d} {pa.molecular_weight() / 1000:7.1f} {pa.isoelectric_point():5.2f} {pa.gravy():6.2f} {pa.instability_index():6.1f}")
# output
gene aa MW(kDa) pI GRAVY instability
1_1 56 6.4 7.89 0.07 53.1
1_2 152 16.9 4.91 0.11 50.3
1_3 38 4.2 12.00 -1.40 65.6
1_4 427 48.5 6.39 -0.42 41.4
1_5 175 19.0 6.24 0.14 29.9
1_6 328 34.4 5.14 -0.43 14.2
1_7 468 53.8 9.14 -0.45 37.8
Gene 1_3 is only 38 residues, has an isoelectric point of 12.00 and the most negative GRAVY: a short, very basic, hydrophilic protein, which fits a DNA-binding protein (the record names it a DNA condensation protein). The instability index is not reliable for very short sequences; read it for the larger proteins only.
Bio.Restriction holds the enzymes of the REBASE database as objects. enzyme.search(seq) returns the cut positions, and Analysis answers questions about a whole set: which enzymes cut once, which not at all, for a linear or a circular molecule. For cloning, the enzymes that cut a vector exactly once are the interesting ones.
from Bio import SeqIO
from Bio.Restriction import AllEnzymes, Analysis, CommOnly, EcoRI, PstI
genome = SeqIO.read("data/phix174.fa", "fasta").seq
print("EcoRI sites:", EcoRI.search(genome), "| PstI sites:", PstI.search(genome))
an = Analysis(CommOnly, genome, linear=False) # commercially available enzymes, circular genome
once = sorted(str(e) for e, cuts in an.with_N_sites(1).items())
print(len(once), "commercial enzymes cut the circular genome exactly once, for example:", ", ".join(once[:8]))
print(len(an.without_site()), "commercial enzymes do not cut it at all")
# output
EcoRI sites: [] | PstI sites: []
81 commercial enzymes cut the circular genome exactly once, for example: AanI, AasI, AatII, Acc16I, AccBSI, AdeI, AhdI, AleI
235 commercial enzymes do not cut it at all
This agrees with chapter P12, where grep found no EcoRI, BamHI, HindIII or PstI sites in this genome. Eighty-one commercially available enzymes cut the circular genome once, and 235 do not cut it at all. The two methods are independent, and their agreement is the check.
The tabular output of P12 can be read as a table with pandas. Bio.SearchIO reads it (format blast-tab), the XML output (blast-xml) and many other search formats into three levels: a query result, its hits, and for each hit its high-scoring pairs (HSPs), the actual local alignments. Use it when you need the alignment itself or the coordinates in a consistent, 0-based form: note that query_start is 0-based, so we add one to compare with the tabular output.
from Bio import SearchIO
for qres in SearchIO.parse("results/hits.tsv", "blast-tab"):
best = qres[0][0] # best hit, best HSP
print(f"{qres.id:14s} -> {best.hit_id.split('|')[3]} identity {best.ident_pct:.1f}% over {best.aln_span} bp "
f"query {best.query_start + 1}-{best.query_end} subject {best.hit_start + 1}-{best.hit_end} e={best.evalue:.1e}")
# output
q1_from_301 -> NC_001422.1 identity 97.3% over 150 bp query 1-150 subject 301-450 e=6.6e-72
q2_from_1501 -> NC_001422.1 identity 97.3% over 150 bp query 1-150 subject 1501-1650 e=6.6e-72
q3_from_2801 -> NC_001422.1 identity 98.0% over 150 bp query 1-150 subject 2801-2950 e=1.4e-73
q4_from_3901 -> NC_001422.1 identity 97.3% over 150 bp query 1-150 subject 3901-4050 e=6.6e-72
q5_from_4801 -> NC_001422.1 identity 98.0% over 150 bp query 1-150 subject 4801-4950 e=1.4e-73
The identities, the aligned lengths and the subject coordinates agree with the table of chapter P12. XML output (-outfmt 5) carries the alignment text and is larger; tabular output with chosen columns is the right default for large searches.
Bio.Entrez sends requests to 's E-utilities. efetch returns a record in the format you ask for, and SeqIO parses it. The code below was run live: it downloads the record of phiX174, counts its coding sequences and prints each with its location. asks for a contact address with every request (use your own) and limits requests to three per second without an API key.
from Bio import Entrez, SeqIO Entrez.email = "[email protected]" # asks for a contact address; use your own with Entrez.efetch(db="nuccore", id="NC_001422.1", rettype="gb", retmode="text") as h: rec = SeqIO.read(h, "") cds = [f for f in rec.features if f.type == "CDS"] print(rec.id, rec.annotations["organism"], len(rec.seq), "bases;", len(cds), "annotated CDS") for f in cds: q = f.qualifiers print(f" {q['locus_tag'][0]:>9s} {str(f.location):<28s} {len(q['translation'][0]):4d} aa {q['product'][0]}") SeqIO.write(rec, "results/phix_ncbi.gb", "") # output NC_001422.1 Escherichia phage phiX174 5386 bases; 11 annotated CDS phiX174p01 join{[3980:5386](+), [0:136](+)} 513 aa DNA replication initiation phiX174p02 join{[4496:5386](+), [0:136](+)} 341 aa DNA replication initiation phiX174p03 join{[5074:5386](+), [0:51](+)} 120 aa head morphogenesis phiX174p04 [50:221](+) 56 aa K phiX174p05 [132:393](+) 86 aa terminase phiX174p07 [389:848](+) 152 aa head morphogenesis phiX174p08 [567:843](+) 91 aa endolysin phiX174p09 [847:964](+) 38 aa DNA condensation phiX174p06 [1000:2284](+) 427 aa major head protein phiX174p10 [2394:2922](+) 175 aa major spike protein phiX174p11 [2930:3917](+) 328 aa pilot protein for DNA ejection
Three things stand out. First, the record has eleven coding sequences, against the seven predicted in P12: the predictor, as P12 warned, reports one gene per region and misses the genes that lie inside others, such as phiX174p08, the 91-residue endolysin at 568 to 843, which lies entirely inside phiX174p07 at 390 to 848. Second, the first three genes have a compound location, join{[3980:5386](+), [0:136](+)}: gene A (513 amino acids) starts at base 3,981, runs to the end of the sequence and carries on from the start, because the genome is circular. This is exactly why reported 1_7 as partial and why its protein had no stop: the stop lies on the other side of the origin. A feature extracted with feature.extract(seq) handles a join correctly; slicing by start and end does not. Third, the record has no gene qualifiers; the locus_tag and product carry the names. Read a record before you write code that assumes its fields.
The record also lets us test a statement from P12: the BBTools copy of the genome differs from the current record.
from Bio import SeqIO
from Bio.Align import PairwiseAligner
local = SeqIO.read("data/phix174.fa", "fasta").seq
ncbi = SeqIO.read("results/phix_ncbi.gb", "").seq
diff = [i + 1 for i, (a, b) in enumerate(zip(local, ncbi)) if a != b]
print(len(local), len(ncbi), "bases; positions that differ:", diff)
for i in diff:
print(f" {i}: local {local[i - 1]}, {ncbi[i - 1]}")
# output
5386 5386 bases; positions that differ: [587, 833, 2731, 2793, 2811]
587: local A, G
833: local A, G
2731: local G, A
2793: local T, C
2811: local T, C
Five bases differ, so two things that carry the same accession are not the same sequence. Always record the source, the version and the date of a reference, and if a result depends on single bases (variant calling, primer design), use the sequence that your reads actually come from.
Biopython reads and writes alignments with Bio.AlignIO (FASTA, Clustal, Stockholm, PHYLIP, Nexus, MAF and more) and treats an alignment as a table: aln[:, 100] is column 100, aln[0] is a row, len(aln) counts the sequences. It does not align sequences with more than two sequences itself; for that use , MUSCLE or Clustal Omega (chapter A2) and read the result. The example aligns the genomes of three related microviruses (phiX174, G4 and alpha3, downloaded from ), measures the share of identical columns, builds a distance matrix and a neighbour-joining tree.
import subprocess from Bio import AlignIO, Entrez, SeqIO from Bio.Phylo.TreeConstruction import DistanceCalculator, DistanceTreeConstructor from Bio import Phylo Entrez.email = "[email protected]" ids = ["NC_001422.1", "NC_001420.2", "NC_001330.1"] # phiX174, G4, alpha3: three related phages with Entrez.efetch(db="nuccore", id=",".join(ids), rettype="fasta", retmode="text") as h: recs = list(SeqIO.parse(h, "fasta")) SeqIO.write(recs, "results/microvirus.fa", "fasta") out = subprocess.run(["", "--quiet", "results/microvirus.fa"], capture_output=True, text=True).stdout open("results/microvirus.aln.fa", "w").write(out) aln = AlignIO.read("results/microvirus.aln.fa", "fasta") print(len(aln), "sequences,", aln.get_alignment_length(), "alignment columns;", [f"{r.id}:{len(str(r.seq).replace('-', ''))}" for r in aln]) ident = sum(1 for i in range(aln.get_alignment_length()) if len(set(aln[:, i])) == 1 and "-" not in aln[:, i]) print(f"columns identical in all three and gap-free: {ident} ({100 * ident / aln.get_alignment_length():.1f}%)") dm = DistanceCalculator("identity").get_distance(aln) print(dm) tree = DistanceTreeConstructor().nj(dm) Phylo.draw_ascii(tree) # output 3 sequences, 7915 alignment columns; ['NC_001422.1:5386', 'NC_001420.2:5577', 'NC_001330.1:6087'] columns identical in all three and gap-free: 1876 (23.7%) NC_001422.1 0.000000 NC_001420.2 0.585850 0.000000 NC_001330.1 0.677448 0.398231 0.000000 NC_001422.1 NC_001420.2 NC_001330.1 _________________________________________________________________ NC_001422.1 | _|______________________ NC_001420.2 | |____________________________________ NC_001330.1
The three genomes (5,386, 5,577 and 6,087 bases) align into 7,915 columns, of which 23.7 % are identical in all three, and the genome of G4 is closer to alpha3 than either is to phiX174 (an identity distance of 0.40 against 0.59 and 0.68). Read this as a sketch. Whole-genome alignments of rearranged or very divergent genomes are unreliable, an identity distance counts every gap column, and neighbour joining on three sequences cannot be rooted. A real tree needs a protein or codon alignment of conserved genes, a substitution model and support values (chapter A2). The point is the mechanics: AlignIO.read, DistanceCalculator, DistanceTreeConstructor and Phylo.draw_ascii take four lines.
Bio.PDB parses structure files into the hierarchy structure, model, chain, residue, atom, with coordinates as NumPy arrays, so geometry is ordinary array arithmetic. The example downloads crambin (PDB entry 1CRN, a 46-residue plant protein), counts its parts, builds the sequence from the atoms, and computes distances between the Cα atoms. Consecutive Cα atoms are about 3.8 Å apart in every protein, which is a useful test that the coordinates were read correctly; the number of residue pairs that are close in space but far apart in sequence is a crude measure of how compact the fold is.
import urllib.request
from Bio.PDB import PDBParser, PPBuilder
import numpy as np
urllib.request.urlretrieve("https://files.rcsb.org/download/1CRN.pdb", "results/1crn.pdb")
s = PDBParser(QUIET=True).get_structure("crambin", "results/1crn.pdb")
model = s[0]
chain = next(model.get_chains())
res = [r for r in chain if r.id[0] == " "]
print("chains:", len(list(model.get_chains())), "| residues:", len(res), "| atoms:", sum(1 for _ in model.get_atoms()))
print("sequence:", str(PPBuilder().build_peptides(chain)[0].get_sequence()))
ca = np.array([r["CA"].coord for r in res])
d = np.linalg.norm(ca[:, None] - ca[None], axis=-1)
print(f"consecutive CA-CA distance: {np.mean([d[i, i + 1] for i in range(len(res) - 1)]):.2f} A")
print("residue pairs closer than 8 A but more than 4 apart in sequence:", int(sum(1 for i in range(len(res)) for j in range(i + 5, len(res)) if d[i, j] < 8)))
rg = np.sqrt(((ca - ca.mean(axis=0)) ** 2).sum(axis=1).mean())
print(f"radius of gyration of the CA atoms: {rg:.1f} A")
# output
chains: 1 | residues: 46 | atoms: 327
sequence: TTCCPSIVARSNFNVCRLPGTPEAICATYTGCIIIPGATCPGDYAN
consecutive CA-CA distance: 3.82 A
residue pairs closer than 8 A but more than 4 apart in sequence: 67
radius of gyration of the CA atoms: 9.7 A
The consecutive distance of 3.82 Å is the expected value. Sixty-seven pairs of residues more than four apart in sequence lie within 8 Å of each other, and the radius of gyration of the Cα atoms is 9.7 Å, small for 46 residues: a compact protein. Secondary-structure assignment needs the external program DSSP (Bio.PDB.DSSP calls it); superposition of two structures uses Bio.PDB.Superimposer; for models predicted by AlphaFold the same parser reads the mmCIF or PDB file, and the per-residue confidence is stored in the B-factor column.
Biopython has changed more than most libraries, and code in books and answers written before 2021 often fails or warns. The changes below are the ones you will meet; the first two we checked on the version used for this chapter (1.88).
Seq("ATG", IUPAC.unambiguous_dna) raised an error from version 1.78: delete the alphabet argument. A Seq prints as plain text and does not remember whether it is DNA or protein; you keep track of that.GC is now gc_fraction and returns a fraction; from Bio.SeqUtils import GC fails with an ImportError.Bio.pairwise2 is deprecated and warns on import; use Bio.Align.PairwiseAligner (see above).Seq is now close to str: it can be compared, hashed and searched; seq.tostring() and seq.tomutable() no longer exist.For the rest, the Biopython Tutorial and Cookbook is the manual, and it is kept current with every release: sequence objects (Ch. 3), sequence annotation objects (Ch. 4), Sequence Input/Output (Ch. 5), multiple alignments (Ch. 6), search tools (Ch. 8), Entrez (Ch. 9), Bio.PDB (Ch. 11), Bio.Phylo (Ch. 13), motifs (Ch. 14), clustering and supervised learning (Ch. 15 and 16), graphics including GenomeDiagram (Ch. 17), KEGG (Ch. 18) and the Cookbook (Ch. 20), which holds worked examples such as finding the longest ORFs, plotting GC along a genome and working with the file formats that SeqIO does not cover.
It is not fast for bulk work: counting reads, trimming adapters or sorting alignments belongs to compiled tools (, , ). It does not read BAM or VCF files well; use pysam for those. It is not a statistics library; hand the data to pandas, SciPy or R. Its strength is correctness on biological objects: coordinates, strands, genetic codes, ambiguity codes, file-format quirks. Use it for exactly that.
ATGGCCTGA, are translated with table 11 and with table 4. What do you get, and why?Under table 11, MA followed by a stop (TGA). Under table 4 (Mycoplasma), TGA codes for tryptophan: MAW. Always state the table.
r?SeqFeature(FeatureLocation(100, 200, strand=-1), type="gene").extract(r.seq). Convert the start to 0-based; the end stays; extract reverse-complements for the minus strand.
Seq("ACGTRYN").reverse_complement() give NRYACGT?The sequence is reversed and each letter complemented, including the ambiguity codes: R (A or G) becomes Y (T or C), and N stays N.
SeqIO.index for random access by identifier, or a streaming filter with SeqIO.parse; or grep -f ids.txt on the command line. Not list(SeqIO.parse(...)).
No. Four sites are a poor estimate and a motif is short, so many matches arise by chance. Set a threshold against a background model, and treat hits as candidates to test.
This chapter is one complete analysis in Python, from the raw count table to a figure and a statement you can defend. It uses the demonstration study that ships with MicroRes: 72 stool samples from three diets (Control, Fibre, Antibiotic), 4 negative-control blanks, 377 features, and a table of measurements per sample. Every number below was produced by the code shown, run with pandas, NumPy, SciPy, statsmodels, scikit-learn and Matplotlib. The same analysis is repeated in R in the next chapter, and the two agree.
import numpy as np, pandas as pd
from scipy import stats
counts = pd.read_csv("counts.tsv", sep="\t", index_col=0) # features x samples
meta = pd.read_csv("metadata.tsv", sep="\t", index_col=0) # samples x variables
counts.shape, meta.shape # ((377, 76), (76, 14))
assert list(counts.columns) == list(meta.index) # the same samples in the same order
meta["Sample_type"].value_counts().to_dict() # {'Sample': 72, 'Blank': 4}
meta["Diet"].value_counts().to_dict() # {'Control': 24, 'Fibre': 24, 'Antibiotic': 24, 'Blank': 4}
Before any statistics, look at what you have: the shape, the types of the columns (meta.dtypes), missing values (meta.isna().sum()), the design (three groups of 24, balanced), and what is not a sample (four blanks, which are controls and must not enter the group comparison). A balanced design is easy to analyse; with unbalanced groups, many methods need extra care.
bio = meta.index[meta["Sample_type"] == "Sample"] c, m = counts[bio], meta.loc[bio] c.shape, int(c.values.sum()) # ((377, 72), 2109697) depth = c.sum(axis=0) int(depth.min()), int(depth.median()), int(depth.max()) # (950, 25055, 80357)
The sequencing depth varies by a factor of 85 between samples (950 to 80,357 reads). That is the first thing to deal with: a feature that has 10 reads in a deep sample and 10 in a shallow one is not equally abundant. Counts must be put on a common scale before samples are compared.
Features seen in very few samples carry little information and add noise and tests. A prevalence filter keeps those present in at least 10% of the samples:
prev = (c > 0).mean(axis=1) c = c[prev >= 0.10] c.shape[0] # 300 of the 377 features kept
Choose the threshold before looking at the results, and report it. A filter that depends on the groups (keep features that are present in at least 10% of one group) is a legitimate choice, but it must be applied to the pooled data without looking at the outcome, otherwise it biases the test.
Two common starting points, each with a purpose:
rel = c / c.sum(axis=0) clr = np.log(c + 0.5) clr = clr - clr.mean(axis=0) # centre every sample
The pseudocount (0.5 here) is a choice that matters for rare features. State it, and check that the main conclusion does not depend on it by repeating the analysis with 0.1 and 1. Sequencing data are compositional: they tell you the share of each feature, not its absolute amount, so a feature that rises makes all the others fall (Gloor et al., 2017).
The Shannon index summarises how many features there are and how evenly they share the reads:
p = rel
shannon = -(p * np.log(p.where(p > 0))).sum(axis=0)
shannon.groupby(m["Diet"]).median().round(3).to_dict()
# {'Antibiotic': 3.771, 'Control': 4.252, 'Fibre': 4.143}
Is the difference between the three diets larger than chance? The values are not normally distributed and the groups are small, so use rank-based tests: Kruskal–Wallis for all three groups, then Mann–Whitney for a pair.
g = [shannon[m["Diet"] == d].values for d in ["Control", "Fibre", "Antibiotic"]] stats.kruskal(*g) # statistic 22.67, p = 1.2e-05 a, b = shannon[m["Diet"] == "Antibiotic"], shannon[m["Diet"] == "Control"] stats.mannwhitneyu(a, b) # U = 85, p = 3.0e-05
Report an effect size with the p value: the antibiotic group has a median Shannon of 3.77 against 4.25 in the control group, a drop of 0.48, which is the number a reader can judge. A very small p value with a tiny difference is not interesting; a large difference with p = 0.06 in six samples per group may be. Both numbers belong in the sentence.
Which features differ between antibiotic and control? Test each of the 300 features, on the CLR values, and correct for the number of tests:
from statsmodels.stats.multitest import multipletests x = clr.loc[:, m["Diet"] == "Antibiotic"] y = clr.loc[:, m["Diet"] == "Control"] pv = [stats.mannwhitneyu(x.loc[i], y.loc[i]).pvalue for i in clr.index] q = multipletests(pv, method="fdr_bh")[1] (np.array(pv) < 0.05).sum(), (q < 0.05).sum() # (119, 72)
119 features have a raw p below 0.05, and 72 survive the Benjamini–Hochberg correction at a false-discovery rate of 5%. With 300 tests, about 15 features would have p < 0.05 by chance even if nothing differed, so the raw count overstates the finding by a lot. The corrected value says: of the features reported at this threshold, about 5% are expected to be false. Always report the corrected number and name the method (Benjamini and Hochberg, 1995). Microbiome methods built for compositional counts (ANCOM-BC, ALDEx2, -type models) are better than a rank test on a transformed table when effects are small; use more than one and report what several agree on.
A group difference might be explained by something else that differs between the groups. A linear model estimates the diet effect while holding other variables fixed:
import statsmodels.formula.api as smf
df = pd.DataFrame({"shannon": shannon, "fibre": m["Fibre_g_day"], "diet": m["Diet"], "bmi": m["BMI"]})
fit = smf.ols("shannon ~ fibre + C(diet, Treatment('Control')) + bmi", data=df).fit()
fit.params.round(3)
# Intercept 4.222 | diet[Antibiotic] -0.480 | diet[Fibre] -0.067 | fibre -0.006 | bmi 0.005
round(fit.rsquared, 3) # 0.374
Read the coefficients against the reference level, Control (set explicitly with Treatment('Control')): antibiotic lowers Shannon by 0.48 units relative to control after adjusting for fibre intake and BMI, while fibre intake and BMI do not matter once diet is in the model. R2 = 0.37 says the model explains 37% of the variation in Shannon. Look at the residuals (fit.resid) before you trust the standard errors; a model for repeated measures from the same person (this study has three time points per subject) needs a random effect or a robust standard error, and the p values above ignore that.
Principal component analysis on the CLR values (which gives an Aitchison-distance view) projects the samples onto the directions of greatest variation:
from sklearn.decomposition import PCA pca = PCA(n_components=2).fit(clr.T.values) pca.explained_variance_ratio_.round(3) # [0.135 0.037]
The first two axes explain 13.5% and 3.7% of the variation: small shares, as is usual for microbiome data with hundreds of features, so a separation of groups on these two axes is a strong signal, and the absence of one is not proof of no difference. A PERMANOVA on a distance matrix (skbio.stats.distance.permanova) is the test that goes with the picture; see the R chapter for its output on the same data.
Can the community tell whether a sample is from the antibiotic group? A random forest is a reasonable first model. The danger is overfitting: a model with hundreds of features and 72 samples can memorise them. The only honest estimate of performance is on samples the model has not seen, by cross-validation:
from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import StratifiedKFold, cross_val_score y_ = (m["Diet"] == "Antibiotic").astype(int) cv = StratifiedKFold(5, shuffle=True, random_state=1) auc = cross_val_score(RandomForestClassifier(300, random_state=1), clr.T.values, y_, cv=cv, scoring="roc_auc") auc.round(2), round(auc.mean(), 2) # [0.7 0.92 0.98 0.94 1.0] 0.91
The area under the ROC curve is 0.91 on average, but it ranges from 0.70 to 1.0 across the five folds: with 72 samples, the estimate is noisy, and a single number hides that. Two more rules. Everything that learns from the data (a filter on prevalence, a normalisation fitted to the data, a feature selection) must happen inside each fold, or the test samples leak into the training. And these samples come from 24 subjects with three time points each; if the three samples of a subject fall in both training and test sets, the model can recognise the person, not the diet, and the AUC is inflated. Group the folds by subject (GroupKFold) for such data.
import matplotlib; matplotlib.use("Agg")
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(5, 3.5))
for i, (d, col) in enumerate(zip(["Control", "Fibre", "Antibiotic"], ["#4477aa", "#228833", "#cc3311"])):
v = shannon[m["Diet"] == d]
ax.scatter(np.random.default_rng(0).normal(i, 0.06, len(v)), v, s=14, color=col) # jittered points
ax.set_xticks([0, 1, 2], ["Control", "Fibre", "Antibiotic"]); ax.set_ylabel("Shannon index")
fig.tight_layout(); fig.savefig("shannon.png", dpi=300)
Show every sample as a point, so that the reader sees the spread and the sample size; add a median line if you wish; use a colour-blind-safe palette (the three colours above are from Paul Tol's set); label the axes with units; and save at 300 dpi for print or as SVG or PDF for vector output. A bar of means with error bars hides everything that matters in 24 values.
The first nine sections used pandas without explaining it. Most of a data scientist's day is spent here, so the main operations are collected in one place. A DataFrame is a table with named columns and a labelled row index; a Series is one column. Every example below ran on the demonstration study.
meta.loc["P01_Day0", ["Diet", "Age"]].to_dict() # by label: {'Diet': 'Control', 'Age': 22.0}
meta.iloc[0:2, 3:5] # by position: the first two rows, columns 3 and 4
old = meta[(meta.Sample_type == "Sample") & (meta.Age > 40)] # a boolean mask; each condition in brackets, joined by & or |
len(old) # 48 samples from people older than 40
meta.query("Diet == 'Fibre' and BMI > 28").shape[0] # the same idea, written as a string: 9 rows
loc uses labels and includes both ends of a slice; iloc uses positions and excludes the end, like a Python list. Writing and instead of & in a mask is the most common error message a beginner meets: pandas needs the element-wise operator.
samples = meta[meta.Sample_type == "Sample"]
samples.groupby("Diet").agg(n=("Subject", "size"), subjects=("Subject", "nunique"),
age=("Age", "mean"), butyrate=("Butyrate_mM", "median")).round(2)
# n subjects age butyrate
# Antibiotic 24 8 43.12 8.74
# Control 24 8 46.38 20.26
# Fibre 24 8 44.00 33.38
This table already contains a design fact that matters later: 24 samples per diet, but only 8 subjects per diet, because every person was sampled three times (days 0, 14 and 28). The samples are not independent, and sections 13 and 14 return to this. groupby follows one pattern in all its uses: split the table by a key, apply a function to each piece, combine the results.
meta["Diet"] = pd.Categorical(meta["Diet"], categories=["Control", "Fibre", "Antibiotic", "Blank"]) meta["Diet"].cat.categories.tolist() # ['Control', 'Fibre', 'Antibiotic', 'Blank'] (the order of plots and tables) meta["Diet"].memory_usage(deep=True) # 6947 bytes, against 10748 as plain strings
A categorical column stores each label once and every row as a small integer. It sorts and plots in the order you gave, not alphabetically, and in a model the first category is the reference level.
tax = pd.read_csv("taxonomy.tsv", sep="\t", index_col=0, header=None) # one column: the lineage
t = tax[1].str.split(";", expand=True)
t.columns = ["kingdom", "phylum", "class", "order", "family", "genus", "species"]
t = t.apply(lambda col: col.str[3:]) # drop the 'k__', 'p__' prefixes
t["phylum"].value_counts().head(4).to_dict()
# {'Firmicutes': 196, 'Bacteroidota': 92, 'Actinobacteriota': 31, 'Proteobacteria': 29}
The .str accessor applies a string method to a whole column, and expand=True turns the pieces into columns. For patterns use .str.extract(r"(Day\d+)") and .str.contains; they take regular expressions (the earlier chapters on the shell and on Python explain them).
reads = counts.sum(axis=1).rename("reads") # total reads per feature
t.join(reads).groupby("phylum")["reads"].sum().sort_values(ascending=False).head(4).to_dict()
# {'Bacteroidota': 928866, 'Firmicutes': 844813, 'Proteobacteria': 147642, 'Actinobacteriota': 128908}
join aligns on the index; merge(a, b, on="key", how="left") aligns on columns and chooses what happens to unmatched rows: inner keeps only matches, left keeps every row of the left table (the safe default when the left table is your samples), outer keeps everything. After every merge, check the number of rows: an unexpected increase means duplicated keys, an unexpected decrease means keys that did not match (often a stray space or a different case).
small = counts.iloc[:3, :3].reset_index() long = small.melt(id_vars="ASV", var_name="sample", value_name="count") # one row per feature and sample # ASV sample count # 0 ASV0001 P01_Day0 4049 # 1 ASV0002 P01_Day0 5851 ... (9 rows) long.pivot(index="ASV", columns="sample", values="count") # back to wide
Wide tables (features by samples) suit matrix mathematics; long tables (one measurement per row) suit plotting with seaborn and models with a formula. Wickham's rule of "tidy data" is that each variable is a column, each observation a row, each kind of unit a table (Wickham, 2014). Moving between the two with melt and pivot is a daily task.
x = pd.Series([1.0, np.nan, 3.0]) x.isna().sum(), x.mean(), x.fillna(x.mean()).tolist() # (1, 2.0, [1.0, 2.0, 3.0]) meta.isna().sum().sum() # 32 missing cells in the metadata
pandas skips NaN when it averages, which is convenient and sometimes dangerous. Find where the missing cells are before deciding what to do (here they are the metadata of the four blanks, which is expected). Filling with the mean hides uncertainty and shrinks the variance; deleting rows loses samples; a model that handles missingness explicitly is better when much is missing. Whatever you choose, count what you removed and say so.
pandas is built on NumPy arrays: blocks of numbers of one type, stored together, and operated on all at once (Harris et al., 2020).
a = np.arange(12).reshape(3, 4) a.shape, a.sum(axis=0).tolist(), a.sum(axis=1).tolist() # ((3, 4), [12, 15, 18, 21], [6, 22, 38])
axis=0 collapses the rows (one result per column); axis=1 collapses the columns (one per row). The rule to remember: the axis you name is the one that disappears.
v = counts.to_numpy(float) # 377 features x 76 samples rel = v / v.sum(axis=0) # broadcasting: a (377, 76) array divided by a (76,) vector, column by column rel.shape, np.allclose(rel.sum(axis=0), 1) # ((377, 76), True)
Broadcasting stretches the smaller array across the larger one when their shapes are compatible, comparing shapes from the last dimension backwards. It replaces loops, and it is also the main source of silent mistakes: dividing a (377, 76) array by a vector of length 377 would fail, but dividing a square matrix by the wrong vector would run and give nonsense. Check shapes after every step.
import time t0 = time.perf_counter(); out = [sum(v[i, j] for j in range(v.shape[1])) for i in range(v.shape[0])] t1 = time.perf_counter(); out2 = v.sum(axis=1); t2 = time.perf_counter() np.allclose(out, out2), round((t1 - t0) / (t2 - t1)) # (True, 189) the array call was 189 times faster
The loop and the array call give the same numbers; the array call runs in compiled code. The ratio depends on the machine (this one measured 189), but the order of magnitude does not. Write the clear version first, measure, and vectorise only the part that is slow.
X = clr.T.to_numpy(); X = X - X.mean(axis=0) # samples x features, centred U, S, Vt = np.linalg.svd(X, full_matrices=False) (S**2 / (S**2).sum())[:3].round(4).tolist() # [0.1346, 0.0366, 0.0353] from sklearn.decomposition import PCA PCA(3).fit(X).explained_variance_ratio_.round(4).tolist() # [0.1346, 0.0366, 0.0353]: the same scores = U[:, :2] * S[:2] # the coordinates of the samples on the first two axes
PCA is not magic: it is the singular value decomposition of the centred table. The squared singular values give the share of variance of each axis, Vt holds the loadings (which features define an axis) and U * S the sample coordinates. Knowing this makes it clear why PCA on centred log-ratios (the Aitchison distance) is the standard for compositional data, and why scaling every feature to unit variance gives rare features as much weight as dominant ones.
Random numbers belong here too: rng = np.random.default_rng(0) creates a generator whose sequence is fixed by the seed (rng.integers(0, 10, 5) gave [8, 6, 5, 2, 3]), and every function that shuffles, samples or initialises should take it or a seed, so that the analysis can be repeated exactly.
A test is a recipe, not a ritual. Writing it from the definition once removes much of the mystery, and it gives you a method that works when no ready-made test fits. The question here: is the Shannon index of the Antibiotic group (n = 24) lower than that of the Control group (n = 24)?
from scipy import stats a = shannon[m.Diet == "Control"].to_numpy(); b = shannon[m.Diet == "Antibiotic"].to_numpy() stats.ttest_ind(a, b, equal_var=False) # Welch: t = 5.01, df = 27.1, p = 2.9e-05 stats.mannwhitneyu(a, b) # rank-sum: U = 491 (85 for the other group), p = 3.0e-05
The Welch test compares means without assuming equal variances (the antibiotic group's standard deviation, 0.45, is three times the control group's, 0.13, so this matters; use it as the default instead of the classical t-test). The rank-sum test uses only the order of the values. Both say the groups differ. Now the same question without any distribution assumption:
rng = np.random.default_rng(1)
obs = a.mean() - b.mean(); pool = np.r_[a, b]; n = len(a)
perm = np.empty(10000)
for i in range(10000):
rng.shuffle(pool) # if the diet did not matter, the labels are exchangeable
perm[i] = pool[:n].mean() - pool[n:].mean()
obs.round(3), (np.sum(np.abs(perm) >= abs(obs)) + 1) / 10001 # (0.478, 0.0001)
The permutation test asks how often shuffling the labels produces a difference as large as the observed 0.478. With 10,000 shuffles none did, so the p value is reported as its smallest possible value, 1/10,001 (the "+1" counts the observed arrangement as one of the permutations; a p value of exactly 0 is never correct). The same logic, with the test statistic replaced, is what PERMANOVA does for whole communities.
bs = np.array([rng.choice(a, len(a)).mean() - rng.choice(b, len(b)).mean() for _ in range(5000)]) np.percentile(bs, [2.5, 97.5]).round(3) # [0.305, 0.667]: a 95% bootstrap interval for the difference
The bootstrap resamples each group with replacement to see how much the estimate would vary from study to study. A p value says "not zero"; the interval says how large the difference plausibly is, which is what a reader needs. An interval from 0.31 to 0.67 Shannon units is a clear loss of diversity.
sp = np.sqrt((a.var(ddof=1) + b.var(ddof=1)) / 2) round((a.mean() - b.mean()) / sp, 2) # Cohen's d = 1.45: a large standardised difference (np.sign(a[:, None] - b[None, :])).sum() / (len(a) * len(b)) # Cliff's delta = 0.70
Cohen's d expresses the difference in units of the pooled standard deviation. Cliff's delta is the probability that a random control sample is higher than a random antibiotic sample, minus the probability of the opposite: 0.70 means that in about 85% of random pairs the control is higher. Report an effect size next to every p value, because with enough samples a trivial difference is "significant" and with too few a large one is not.
from statsmodels.stats.power import TTestIndPower TTestIndPower().solve_power(effect_size=0.8, alpha=0.05, power=0.8) # 25.5 per group TTestIndPower().solve_power(effect_size=0.5, alpha=0.05, power=0.8) # 63.8 per group
To detect a large effect (d = 0.8) with 80% probability at the 5% level, you need about 26 samples per group; for a medium effect (d = 0.5) about 64. The effect observed here (d = 1.45) is larger than either, which is why 24 per group was enough. Power must be calculated before the study from an effect size you can justify (a pilot, the literature, or the smallest difference that matters), never from the result afterwards. For microbiome-specific designs, where the unit is a community and the outcome a distance or a taxon proportion, see the power chapter of Xia and Sun's book (Read: ch. 5).
import statsmodels.api as sm, statsmodels.formula.api as smf
m["shannon"] = shannon # from section 4, now a column of the sample table
mod = smf.ols("shannon ~ C(Diet, Treatment('Control')) + Age + C(Sex)", data=m).fit()
mod.params.round(3) # Intercept 4.052; Antibiotic -0.466; Fibre -0.104; Sex[M] 0.007; Age 0.004
round(mod.rsquared, 3) # 0.376
stats.shapiro(mod.resid).pvalue.round(3) # 0.003: residuals are not normal
sm.stats.diagnostic.het_breuschpagan(mod.resid, mod.model.exog)[1] # p = 0.002: the spread differs between groups
The coefficients are differences from the reference (Control): the antibiotic group is 0.47 Shannon units lower, adjusted for age and sex, while age and sex add nothing here. But both diagnostic tests fail. The cause is the one seen above: the antibiotic group is far more variable. The estimates remain unbiased, but the standard errors are not trustworthy. Three honest responses: use heteroscedasticity-robust standard errors (.fit(cov_type="HC3")), use a model with a separate variance per group, or use a permutation or rank-based method as in section 12. Plot the residuals against the fitted values first; a plot shows how the assumption fails, which a test does not.
dd = m.assign(y=c.loc["ASV0001"].values, depth=c.sum(axis=0).values)
pg = smf.glm("y ~ C(Diet, Treatment('Control'))", data=dd, family=sm.families.Poisson(),
offset=np.log(dd.depth)).fit()
round(pg.pearson_chi2 / pg.df_resid) # 1290: far above 1, so the Poisson model is wrong for these data
nb = smf.negativebinomial("y ~ C(Diet, Treatment('Control'))", data=dd, offset=np.log(dd.depth)).fit(disp=0)
np.exp(nb.params).round(3) # Intercept 0.046; Antibiotic 2.730; Fibre 0.377 (rate ratios against Control)
1 / nb.params["alpha"] # theta = 0.94 This feature is the first of the table, and it is a good example of why counts are not analysed with a plain regression. The Poisson model assumes that the variance equals the mean; the Pearson dispersion of 1,290 says the variance is more than a thousand times larger. The negative binomial model adds a dispersion parameter. Its coefficients, exponentiated, are rate ratios: after accounting for sequencing depth (the offset, which makes the model about reads per read sequenced), the feature is 2.73 times as abundant in the antibiotic group (p = 0.0007) and 0.38 times as abundant in the fibre group (p = 0.001) as in the controls. The same model in R (MASS::glm.nb, next chapter) gave the same rate ratios and p values to three decimals, which is the check that matters: two implementations agreeing means the code and the model are understood. Dedicated tools for sequencing counts are built on this idea: and fit the negative binomial model and shrink the dispersion across features, which stabilises the estimates when each feature has few samples, and - takes another route, a linear model on log counts with a weight for each observation (Love et al., 2014; Law et al., 2014).
Repeated measures: the same person three times
mm = smf.mixedlm("shannon ~ C(Diet, Treatment('Control')) * Day", data=m, groups=m["Subject"]).fit()
mm.summary().tables[1]
# Coef. Std.Err. z P>|z|
# Intercept 4.230 0.087 48.60 0.000
# Antibiotic -0.255 0.123 -2.07 0.038
# Fibre -0.002 0.123 -0.02 0.987
# Day 0.000 0.004 0.06 0.951
# Antibiotic:Day -0.016 0.006 -2.82 0.005
# Fibre:Day -0.008 0.006 -1.39 0.165
mm.cov_re.iloc[0, 0], mm.scale # variance between subjects 0.019, residual variance 0.050
Each of the 24 subjects contributes three samples, so the 72 rows are not 72 independent observations. A mixed model gives each subject its own baseline (the random intercept, groups=m["Subject"]) and estimates the effects of diet and time across subjects. The result tells a story the single-time comparison could not: at day 0 the antibiotic group is already 0.26 lower, and its Shannon index then falls by 0.016 per day relative to the controls (p = 0.005), while the fibre group does not differ detectably. About 27% of the variance (0.019 of 0.069) lies between people, the rest within a person over time.
Treating the 72 samples as independent, as in section 6, gives a standard error that is too small, because the information in 24 people is less than in 72 strangers. Always ask: what is the unit that was randomised or sampled independently? Here it is the subject. The R version (nlme::lme) gave the same estimates; it gave p = 0.051 for the antibiotic main effect where statsmodels gave 0.038, because R uses a t distribution with 21 degrees of freedom for a between-subject effect and statsmodels uses a normal approximation. With 8 subjects per diet the t distribution is the careful choice, and the difference between 0.038 and 0.051 is a reminder that p values near 0.05 do not deserve a verdict, only an interval.
A binary outcome: logistic regression
dl = m[m.Diet.isin(["Control", "Antibiotic"])].copy(); dl["ab"] = (dl.Diet == "Antibiotic").astype(int)
lg = smf.logit("ab ~ shannon", data=dl).fit(disp=0)
lg.params.round(3).to_dict(), np.exp(lg.params["shannon"]).round(3) # slope -6.158; odds ratio per Shannon unit 0.002
from sklearn.metrics import roc_auc_score
roc_auc_score(dl.ab, lg.predict(dl)).round(3) # 0.852
Logistic regression models the log-odds of the outcome. The slope of -6.16 means that each additional Shannon unit multiplies the odds of being an antibiotic sample by 0.002. This number looks extreme because one Shannon unit is a huge change in this data (the standard deviation is about 0.3), so express such effects per standard deviation. The area under the ROC curve, 0.85, is the probability that a random antibiotic sample has a lower diversity than a random control sample. Note that it was computed on the same data the model was fitted to; section 14 shows what it becomes when it is estimated honestly.
Faraway's Extending the Linear Model with R (Read: ch. 2 binary response, ch. 5 count regression, ch. 10 random effects, ch. 11 repeated measures) and ISLP (ch. 3 and 4) treat the same models with more mathematics.
14. Machine learning without fooling yourself
Machine learning in bioinformatics fails far more often through the evaluation than through the algorithm. This section predicts, from the 300 CLR features, whether a sample belongs to the antibiotic group (24 positives of 72), and spends most of its length on how to know how well that works (James et al., 2023; Pedregosa et al., 2011).
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, StratifiedGroupKFold, cross_val_score
X = clr.T.to_numpy(); y = (m.Diet == "Antibiotic").to_numpy().astype(int); groups = m.Subject.to_numpy()
lr = make_pipeline(StandardScaler(), LogisticRegression(C=0.1, max_iter=5000)) # scaling is inside the pipeline
The split must follow the independence of the data
cross_val_score(lr, X, y, cv=StratifiedKFold(5, shuffle=True, random_state=1), scoring="roc_auc").round(3)
# [0.98 0.92 0.925 0.978 0.978] mean 0.956
sg = StratifiedGroupKFold(5, shuffle=True, random_state=1)
cross_val_score(lr, X, y, cv=sg, groups=groups, scoring="roc_auc").round(3)
# [0.852 0.889 0.833 0.889 0.806] mean 0.854
The two lines differ in one thing: in the first, the three samples of one person can be split between the training and the test folds; in the second, a subject is entirely in one or the other. The first gives AUC 0.96, the second 0.85. The model in the first case is probably recognising people in part (each person has a stable microbiome), and the better score is an illusion. When the model will be applied to new patients, only the second number estimates what you will get. Any time a data set has repeated samples, families, batches, hospitals or sequencing runs, the unit of splitting must be that group.
Leakage: the other way to fool yourself
from sklearn.feature_selection import SelectKBest, f_classif
yperm = np.random.default_rng(0).permutation(y) # the labels are now pure noise
Xl = SelectKBest(f_classif, k=20).fit_transform(X, yperm) # choose the 20 best features using ALL samples...
cross_val_score(LogisticRegression(max_iter=5000), Xl, yperm, cv=StratifiedKFold(5, shuffle=True, random_state=1), scoring="roc_auc").mean() # 0.885
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=5000)) # ...or inside each training fold
cross_val_score(pipe, X, yperm, cv=StratifiedKFold(5, shuffle=True, random_state=1), scoring="roc_auc").mean() # 0.659
With random labels, selecting the features first and cross-validating afterwards yields an AUC of 0.89, a model with a spectacular and entirely false performance, because the test samples helped choose the features. Selecting inside the pipeline gives 0.66, which is what noise looks like on 72 samples (the true value of a useless model is 0.5; with so few samples the estimate wanders, and error bars on a cross-validated score are wide (Varoquaux, 2018)). Every step that learns something from the data (scaling, imputation, feature selection, dimension reduction, tuning) belongs inside the cross-validation loop. A Pipeline object makes this automatic (Ambroise and McLachlan, 2002).
Tuning needs its own loop
from sklearn.model_selection import GridSearchCV
inner = GridSearchCV(make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)),
{"logisticregression__C": [0.001, 0.01, 0.1, 1]},
cv=StratifiedKFold(3, shuffle=True, random_state=2), scoring="roc_auc")
nested = cross_val_score(inner, X, y, cv=sg, groups=groups, scoring="roc_auc")
nested.round(3), nested.mean().round(3) # [0.852 0.87 0.926 0.889 0.806] 0.869
If you tune a setting (here the regularisation strength C) and report the best cross-validated score, that score is optimistic, because the setting was chosen to maximise it. Nested cross-validation tunes inside each training fold and scores on a fold that tuning never saw: the honest estimate is 0.87. Regularisation is the central idea of ISLP's chapter 6: with 300 features and 72 samples, an unpenalised model would fit the training data perfectly and predict poorly, and the penalty C controls the trade-off.
Other models, and what they say
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(500, random_state=1, n_jobs=2)
cross_val_score(rf, X, y, cv=sg, groups=groups, scoring="roc_auc").mean().round(3) # 0.865 (sg: the grouped splitter above)
l1 = make_pipeline(StandardScaler(), LogisticRegression(penalty="l1", solver="liblinear", C=0.1))
cross_val_score(l1, X, y, cv=sg, groups=groups, scoring="roc_auc").mean().round(3) # 0.641
l1.fit(X, y); int((l1[-1].coef_[0] != 0).sum()) # 10 of 300 features kept
The random forest (an ensemble of decision trees that vote) reaches 0.87 and the L1-penalised model, which also selects features by setting coefficients to zero, only 0.64 with this value of C. The lasso was not tuned here, and a stronger or weaker penalty would change the result: a lesson in why every method must be tuned in a nested loop before it is compared with another. Simple models are strong baselines; try the logistic regression before the forest, and report both.
The cost of a threshold, and importance
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import confusion_matrix, balanced_accuracy_score
pred = cross_val_predict(lr, X, y, cv=sg, groups=groups, method="predict_proba")[:, 1]
confusion_matrix(y, pred > 0.5).tolist(), balanced_accuracy_score(y, pred > 0.5).round(3) # [[45, 3], [8, 16]] 0.802
At the 0.5 threshold the model finds 16 of the 24 antibiotic samples (sensitivity 67%) and wrongly flags 3 of the 48 others (specificity 94%). Accuracy alone would be misleading when the classes are unbalanced; report sensitivity, specificity, the AUC, and the precision if the outcome is rare. Which features matter can be asked from a held-out set with permutation_importance: shuffle one feature and see how much the held-out AUC drops. Here the top feature was ASV0155 (AUC drop 0.066), but the held-out AUC of that forest was only 0.73 and the set small, so a ranking of features from 24 test samples is a hint for follow-up, not a finding.
y3 = m.Diet.to_numpy() # three classes
cross_val_score(lr, X, y3, cv=sg, groups=groups, scoring="balanced_accuracy").mean().round(3) # 0.611
Telling the three diets apart, with a chance level of 33%, gives a balanced accuracy of 0.61: the antibiotic samples are recognisable, but Control and Fibre are not easily separated by this feature set, in line with the small difference in Shannon seen earlier. State what the model cannot do as well as what it can.
Clustering: finding groups nobody labelled
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score
Z = PCA(5).fit_transform(X - X.mean(0))
adjusted_rand_score(m.Diet, KMeans(3, n_init=10, random_state=1).fit(Z).labels_).round(3) # 0.438
from scipy.cluster.hierarchy import linkage, fcluster
from scipy.spatial.distance import pdist
adjusted_rand_score(m.Diet, fcluster(linkage(pdist(X), "ward"), 3, "maxclust")).round(3) # 0.502
The adjusted Rand index compares two partitions: 1 means identical, 0 means what chance would give. Both methods recover the diets only moderately (0.44, 0.50). Clustering always returns clusters, even in noise, so judge them against something (labels, replicates, stability on resampling), and never name the groups it found as if they were biological categories.
15. Figures with matplotlib and seaborn
Matplotlib draws figures from objects: a Figure holds one or more Axes, and everything is a method of an Axes. Write figures this way, not with the stateful plt.plot shortcuts, and they stay controllable when they grow (Hunter, 2007; Waskom, 2021).
import matplotlib.pyplot as plt, seaborn as sns
order = ["Control", "Fibre", "Antibiotic"]
pal = {"Control": "#4477aa", "Fibre": "#228833", "Antibiotic": "#cc3311"} # a colour-blind-safe set
m["shannon"] = shannon
fig, ax = plt.subplots(1, 2, figsize=(8, 3.4), constrained_layout=True)
sns.boxplot(data=m, x="Diet", y="shannon", order=order, palette=pal, showfliers=False, ax=ax[0])
sns.stripplot(data=m, x="Diet", y="shannon", order=order, color="k", size=3, ax=ax[0]) # every sample, on top
ax[0].set(xlabel="", ylabel="Shannon index")
for d in order: # change over time, one line per diet
s = m[m.Diet == d].groupby("Day")["shannon"].mean()
ax[1].plot(s.index, s.values, marker="o", color=pal[d], label=d)
ax[1].set(xlabel="Day", ylabel="mean Shannon"); ax[1].legend(frameon=False)
fig.savefig("shannon.pdf") # a vector file for a paper
fig.savefig("shannon.png", dpi=300) # 2400 x 1020 pixels at 8 x 3.4 inches
This is the figure of the section 13 story: left, the distribution of every sample by diet, with the points shown so that nobody has to trust the box; right, the group means over time, where the falling antibiotic line is the interaction. Rules that make a figure honest: show the individual points when there are few; label axes with units; use colour for the variable that matters and keep it the same across figures; choose a colour-blind-safe palette; never let the y axis start at a value that exaggerates a small difference in a bar chart; save as PDF or SVG for print and at 300 dpi or more for bitmaps. sns.clustermap(np.log10(top + 1), cmap="viridis") draws a clustered heat map of the most abundant features, and sns.pairplot a matrix of all pairs of variables.
On a server without a screen, call matplotlib.use("Agg") before importing pyplot, and the figure is written to a file without trying to open a window.
16. Larger data and faster code
The demonstration table has 28,000 cells; a real shotgun study can have a hundred million. Four ideas cover most of the problem.
scipy.sparse.csr_matrix(x) stores only the non-zeros: 146,220 bytes against 217,152 for the dense 64-bit array here, a saving of a third. The saving grows quickly with sparsity (a table that is 95% zeros needs less than a tenth of the dense memory) and pays off for shotgun and single-cell data. Smaller number types help too: x.astype(np.int32) halves the memory of a table of counts.for chunk in pd.read_csv(f, chunksize=50000): total += chunk["a"].sum() reads 200,000 rows 50,000 at a time and gives the same sum as reading it whole.df.to_parquet("x.parquet") stores types and compresses; the 200,000-row test table was smaller than the CSV. Use HDF5 or Zarr for large arrays, and keep CSV or TSV for tables that people will open in a spreadsheet.big.apply(lambda r: r["a"] + r["b"], axis=1) loops in Python over 200,000 rows and was more than fifty times slower than big["a"] + big["b"] with identical results. cProfile (pr = cProfile.Profile(); pr.enable(); ...; pr.disable()) shows where the time goes; optimise only what the profile points to.When one machine is not enough, the same code moves to a cluster: Dask and Polars give a pandas-like interface over data larger than memory, and a workflow manager (, ) runs one script per sample in parallel. The scientific logic does not change; only where it runs.
A script that only you can run is a draft. Three habits turn analysis code into something reusable (Wilson et al., 2017).
# diversity.py
import numpy as np
def shannon(counts):
"""Shannon index (natural log) of a vector of counts; zeros are ignored."""
x = np.asarray(counts, dtype=float)
if x.ndim != 1 or (x < 0).any():
raise ValueError("counts must be a one-dimensional, non-negative vector")
total = x.sum()
if total == 0:
return float("nan")
p = x[x > 0] / total
return float(-(p * np.log(p)).sum())
A function that states what it expects, refuses what it cannot handle and returns something definite for the edge case (an empty sample) is a function you can reuse in the next project without rereading it.
# tests/test_diversity.py
import math, pytest
from diversity import shannon
def test_one_feature_has_zero_diversity():
assert shannon([10, 0, 0]) == 0.0
def test_even_community_is_log_richness():
assert shannon([5, 5, 5, 5]) == pytest.approx(math.log(4))
def test_scaling_does_not_change_it():
assert shannon([1, 2, 3]) == pytest.approx(shannon([10, 20, 30]))
def test_empty_sample_is_nan():
assert math.isnan(shannon([0, 0]))
def test_negative_counts_are_refused():
with pytest.raises(ValueError):
shannon([1, -1])
$ python -m pytest -q ..... [100%] 5 passed in 0.01s
Each test states a fact you know independently of the code: a community with one feature has zero diversity; four equally abundant features have a Shannon index of ln 4; multiplying all counts by ten changes nothing. If the code is later "improved" and one fact stops being true, the test says so at once. Test the edges (empty, one feature, negative numbers, missing values), because that is where analysis code breaks.
# summarise.py
import argparse
import pandas as pd
from diversity import shannon
def main():
ap = argparse.ArgumentParser(description="Shannon index of every sample in a count table (features x samples).")
ap.add_argument("counts", help="tab-separated count table, features in rows")
ap.add_argument("-o", "--out", default="shannon.tsv")
a = ap.parse_args()
t = pd.read_csv(a.counts, sep="\t", index_col=0)
t.apply(shannon).rename("shannon").to_csv(a.out, sep="\t")
print(f"{t.shape[1]} samples written to {a.out}")
if __name__ == "__main__":
main()
$ python summarise.py counts.tsv -o shannon.tsv 76 samples written to shannon.tsv $ python summarise.py -h # prints the usage, the arguments and the description
argparse gives a script a documented interface, so it can be called from a shell loop, a Makefile or a workflow manager with different inputs, which is how a one-off analysis becomes a step of a pipeline. Keep the logic in importable functions and only the argument handling in main(); the if __name__ == "__main__" guard lets the tests import the module without running it. Add git for history and a requirements.txt with the versions, and the project can be reproduced.
Put the code in a script, not in notebook cells run out of order. Fix random seeds (random_state=1, default_rng(0)). Record the versions (pip freeze). Read inputs from data/, write to results/, and keep the raw table untouched. At the top, assert the things that must be true (assert list(counts.columns) == list(meta.index)), so that a wrong file fails loudly instead of producing a plausible wrong answer.
You can check every claim in the statistics sections by generating data where you know the truth. This is the best way to understand p values, power and error rates, and the way to test an analysis before trusting it on real data. Everything below ran with a fixed seed (rng = np.random.default_rng(7)): 10,000 simulated experiments of two groups of 24 measurements each.
def experiment(delta, n=24, B=10000):
a = rng.normal(0, 1, (B, n)); b = rng.normal(delta, 1, (B, n))
return stats.ttest_ind(a, b, axis=1).pvalue # B p values, one per experiment
(experiment(0.0) < 0.05).mean() # 0.050: with no real difference, 5% of experiments are "significant"
(experiment(0.5) < 0.05).mean() # 0.394: a medium effect (d = 0.5) is found in only 39% of experiments
(experiment(0.8) < 0.05).mean() # 0.776: a large effect (d = 0.8) in 78%
The first line is the meaning of a 5% significance level: it is the false-positive rate when nothing is going on. The next two are statistical power. A study of 24 per group that looks for a medium effect misses it six times in ten, and that is a property of the design and not of bad luck, which is why section 12 calculates the sample size beforehand. A "negative" result from an underpowered study says little.
a = rng.normal(0, 1, (300, 24)); b = rng.normal(0, 1, (300, 24)); b[:60] += 1.0 # 60 features truly differ by one SD p = stats.ttest_ind(a, b, axis=1).pvalue (p < 0.05).sum(), (p[60:] < 0.05).sum() # 61 "hits", 9 of them false (the 240 null features) q = multipletests(p, method="fdr_bh")[1] (q < 0.05).sum(), (q[60:] < 0.05).sum() # 48 hits, 2 false: a false-discovery proportion of 4% (multipletests(p, method="bonferroni")[1] < 0.05).sum() # 16 hits, none false: safe but it finds few of the 60 real ones
This is section 5's problem with the answer known. Without correction, 9 of the 61 hits are false (15%, close to the 12 that 240 null tests × 5% would predict). The Benjamini–Hochberg procedure keeps the proportion of false discoveries near the 5% it promises (2 of 48), while Bonferroni, which controls the chance of any false hit, is so strict that it finds only 16 of the 60 real differences. FDR control is the standard for exploratory screens, and Bonferroni for a handful of pre-specified tests (Benjamini and Hochberg, 1995).
x = rng.normal(5, 2, (10000, 24)); mean = x.mean(1); se = x.std(1, ddof=1) / np.sqrt(24); t = stats.t.ppf(.975, 23) ((mean - t * se <= 5) & (5 <= mean + t * se)).mean() # 0.954
A 95% interval is a procedure that captures the true value in 95% of repetitions: 0.954 here. It does not mean that the true value lies in this particular interval with 95% probability, and it does not mean that 95% of the data are inside it. The correct reading, "the data are compatible with true values in this range", is what makes an interval more informative than a p value.
true = rng.normal(0, 1, 2000); o1 = true + rng.normal(0, 1, 2000); o2 = true + rng.normal(0, 1, 2000) top = o1 > np.quantile(o1, 0.9) o1[top].mean().round(2), o2[top].mean().round(2) # 2.40, then 1.13
Take the top 10% on a noisy first measurement and measure them again: the mean falls from 2.40 to 1.13 with no intervention at all, because part of what put them on top was luck. The features with the most extreme effects in a discovery study are, on average, overestimated, and a replication shows smaller effects. Whoever picks "the best" biomarkers from a small study must expect that, and must replicate in new samples.
stats.skew(rng.negative_binomial(1, 0.05, 100000)).round(2) # 2.0: raw counts are strongly skewed stats.skew(rng.negative_binomial(1, 0.05, (10000, 24)).mean(1)).round(2) # 0.40: means of 24 of them much less so
Means of many values are close to normal even when the values themselves are not, which is why t-tests on means of reasonable samples are often acceptable, and why the same test applied to a single heavily skewed variable with 5 observations per group is not. The skewness falls from 2.0 for single counts to 0.4 for means of 24. This is not a licence to ignore the data: the previous sections found that the variances differed between groups, and the central limit theorem does nothing for that.
The practical habit is to simulate before the experiment (what would my design find if the effect were as I hope?), and after the analysis (does my method return the answer when I plant it?). Rosner's Fundamentals of Biostatistics (Read: the chapters on estimation, hypothesis testing and multiple comparisons) and Petrie and Watson's Statistics for Veterinary and Animal Science explain the theory behind these experiments.
Do these on the demonstration files (the Data menu of MicroRes has them) before opening the answers. Each one uses only what this chapter showed.
pielou(counts) (Shannon divided by the logarithm of the number of observed features) with a test for an even community and for a community of one feature.1. One feature is present in all 72 samples, and 166 are present in at least 36: (c > 0).all(axis=1).sum() and ((c > 0).mean(axis=1) >= 0.5).sum(). Almost no feature is universal, which is why prevalence filters are chosen by the question and not by a single "core" rule.
2. c.sum(axis=0).groupby(m.Diet).median(): Antibiotic 27,718, Control 22,098, Fibre 28,377. The groups are within 30% of each other, much narrower than the 85-fold range between samples, so depth is not confounded with diet; the normalisation of section 3 still has to be done, because it is the spread between samples that matters.
3. Bacteroides with 490,232 reads, then Faecalibacterium (203,095) and Phocaeicola (160,443). Join the taxonomy table to the row sums and group by the genus column.
4. 1,999, 1,267, 1,468 and 1,227 reads, about 5% to 8% of a typical sample (the median depth is 25,055). Reads in a blank are contamination from reagents or from the laboratory; features that are as abundant in the blanks as in the samples are suspect, and tools such as decontam use exactly this comparison. Never subtract blank counts by hand without recording it.
5. (c > 0).sum(axis=0).groupby(m.Diet).median(): Antibiotic 141, Fibre 172.5, Control 185.5. The antibiotic group has the fewest features, in line with its lower Shannon index. Use stats.kruskal on the three arrays, and a permutation test for the pair, and state the effect size in features.
6. def pielou(x): h = shannon(x); n = (np.asarray(x) > 0).sum(); return float("nan") if n < 2 else h / math.log(n). Tests: pielou([5, 5, 5]) == pytest.approx(1.0) (perfectly even), math.isnan(pielou([10, 0, 0])) (undefined for one feature, as the logarithm of 1 is zero), and pielou([1, 2, 3]) == pytest.approx(pielou([10, 20, 30])).
A feature has more reads in a deep sample simply because more was sequenced. Convert to relative abundance or a log-ratio scale, or use a model that estimates size factors, before comparing.
The number that survives false-discovery correction, 72 at a 5% FDR. With 300 tests about 15 would pass 0.05 by chance.
That the mean AUC was 0.91 (range 0.70–1.0 over five folds), and that with 72 samples the estimate is imprecise. Also say whether subjects were kept within one fold.
With hundreds of features the variation is spread across many axes. A group separation on the first two axes is a strong signal even though they explain little in total.
R is where statistics is done first, and where ecological and genomic methods are published. This chapter repeats the analysis of the previous chapter in R, on the same files, so that you can compare the two languages line by line, and adds what R does best: formal community statistics (PERMANOVA and dispersion), tidy modelling and graphics. Everything shown was run in R 4.6 with the packages vegan and ggplot2; the packages of the tidyverse and Bioconductor are described but not run here, and marked as such.
suppressPackageStartupMessages({library(vegan); library(ggplot2)})
counts <- read.delim("counts.tsv", row.names = 1, check.names = FALSE) # features x samples
meta <- read.delim("metadata.tsv", row.names = 1, stringsAsFactors = FALSE)
stopifnot(all(colnames(counts) == rownames(meta))) # same samples, same order
dim(counts) # 377 76
table(meta$Sample_type) # Blank 4, Sample 72
table(meta$Diet) # Antibiotic 24, Blank 4, Control 24, Fibre 24
Two R-specific cautions. check.names = FALSE stops R from "repairing" sample names with symbols, which would make them differ from the metadata and fail the check. And factor levels are alphabetical by default, so set the reference level yourself: m$Diet <- factor(m$Diet, levels = c("Control", "Fibre", "Antibiotic")). In every model, the first level is the reference and the other coefficients are differences from it.
bio <- rownames(meta)[meta$Sample_type == "Sample"]
cnt <- counts[, bio]; m <- meta[bio, ]
m$Diet <- factor(m$Diet, levels = c("Control", "Fibre", "Antibiotic"))
c(nrow(cnt), ncol(cnt), sum(cnt)) # 377 72 2109697
summary(colSums(cnt)) # depth per sample: min 950, median about 25,000, max about 80,000
The depth varies about 85-fold between samples, so counts must be put on a common scale before samples are compared, exactly as in the Python chapter.
prev <- rowMeans(cnt > 0) cnt <- cnt[prev >= 0.10, ]; nrow(cnt) # 300 features kept rel <- sweep(cnt, 2, colSums(cnt), "/") # relative abundance clr <- log(as.matrix(cnt) + 0.5) clr <- sweep(clr, 2, colMeans(clr)) # centred log-ratio
sweep(x, 2, v, "/") applies an operation between every column of x and the vector v: here, dividing each sample by its total. The result is identical to the NumPy version: the same 300 features, the same depth range (949 to 80,278 after filtering), the same transformation.
sh <- diversity(t(cnt), "shannon") # vegan wants samples in rows round(tapply(sh, m$Diet, median), 3) # Control Fibre Antibiotic # 4.252 4.143 3.771 kruskal.test(sh ~ m$Diet) # chi-squared = 22.666, df = 2, p = 1.197e-05 pairwise.wilcox.test(sh, m$Diet, p.adjust.method = "BH") # Control Fibre # Fibre 0.0045 - # Antibiotic 3e-05 0.0028
The medians (4.252, 4.143, 3.771) and the Kruskal–Wallis test are the same as in Python (22.67, p = 1.2e-05), which is what should happen: two correct implementations of the same method agree. The pairwise table gives Benjamini–Hochberg adjusted p values for every pair, so all three diets differ in Shannon diversity at the 5% level, with the antibiotic group lowest. vegan expects samples in rows (hence t(cnt)); and most expression tools expect genes in rows. Check the orientation of every function you use.
x <- clr[, m$Diet == "Antibiotic"]; y <- clr[, m$Diet == "Control"] pv <- sapply(rownames(clr), function(i) wilcox.test(x[i, ], y[i, ])$p.value) q <- p.adjust(pv, "BH") c(sum(pv < 0.05), sum(q < 0.05)) # 119 74
119 raw p values below 0.05, as in Python; 74 after correction in R against 72 in Python. The difference is real and instructive: the two programs handle tied values in the Wilcoxon / Mann–Whitney test differently (an exact test or a normal approximation with a continuity correction), which moves a few borderline p values across the line. Neither is wrong. It is a reminder that a count of "significant features" is a property of the method as well as the data, so report the method and look at the effect sizes, not only the count.
fit <- lm(sh ~ Diet + Fibre_g_day + BMI, data = m) round(coef(summary(fit)), 3) # Estimate Std. Error t value Pr(>|t|) # (Intercept) 4.222 0.251 16.845 0.000 # DietFibre -0.067 0.106 -0.626 0.534 # DietAntibiotic -0.480 0.081 -5.915 0.000 # Fibre_g_day -0.006 0.005 -1.228 0.224 # BMI 0.005 0.009 0.575 0.567
The coefficients match the Python model (antibiotic −0.480 against control). Read the table by row: Estimate is the change in the outcome per unit (or against the reference level), Std. Error its uncertainty, t value the ratio, and Pr(>|t|) the p value of the null hypothesis that the coefficient is zero. Plot the residuals with plot(fit) before trusting the table. For repeated measures on the same subject, use a mixed model (nlme::lme(sh ~ Diet, random = ~ 1 | Subject), run in section 12), which models the correlation between samples of one person that this simple model ignores.
Principal components on the CLR values, and a PERMANOVA on Bray–Curtis distances of the relative abundances:
pc <- prcomp(t(clr)) round(summary(pc)$importance[2, 1:2], 3) # PC1 0.135 PC2 0.037 (as in Python) d <- vegdist(t(rel), "bray"); set.seed(1) adonis2(d ~ Diet, data = m, permutations = 999) # Df SumOfSqs R2 F Pr(>F) # Model 2 2.580 0.14936 6.0576 0.001 *** # Residual 69 14.694 0.85064 # Total 71 17.274 1.00000 bd <- betadisper(d, m$Diet) round(anova(bd)[1, "Pr(>F)"], 4) # 0.1187
Read it. Diet explains 14.9% of the variation in community composition (R2 = 0.149), a moderate effect by microbiome standards, and it is significant: only 1 of 999 random relabellings gave an F as large (the smallest possible p value). The test of dispersion is not significant (p = 0.12), so the groups do not differ detectably in how spread out they are, which means the PERMANOVA result reflects a difference in the centre of the groups and not only in their spread. If the dispersion test had been significant, you would have to say that the groups differ in variability and treat the PERMANOVA with caution.
df <- data.frame(PC1 = pc$x[, 1], PC2 = pc$x[, 2], Diet = m$Diet)
p <- ggplot(df, aes(PC1, PC2, colour = Diet)) +
geom_point(size = 2.5) +
scale_colour_manual(values = c("#4477aa", "#228833", "#cc3311")) +
labs(x = "PC1 (13.5%)", y = "PC2 (3.7%)") +
theme_minimal()
ggsave("pca.pdf", p, width = 5, height = 4) # a vector file; PDF worked on the test machine
ggplot2 builds a figure in layers on a "grammar": data (df), aesthetic mappings (aes: which column is x, y, colour), geometries (geom_point), scales (scale_colour_manual), labels and a theme. This is why a change such as "colour by sex instead of diet" is one word. Facets (facet_wrap(~ Batch)) split a figure by a variable, and geom_boxplot() with geom_jitter() shows summary and points together. Put the axis percentages in the labels, as above, because they tell the reader how much the picture explains.
This is how the same table work is written in the tidyverse, which many groups prefer for readability. Section 10 runs it and shows the results; the code here is the reshaping in one chain:
library(dplyr); library(tidyr)
long <- counts |>
tibble::rownames_to_column("feature") |>
pivot_longer(-feature, names_to = "sample", values_to = "count") |>
left_join(tibble::rownames_to_column(meta, "sample"), by = "sample") |>
filter(Sample_type == "Sample")
long |> group_by(Diet) |> summarise(reads = sum(count), features = n_distinct(feature[count > 0]))
The pipe |> passes the result of one step to the next, so the code reads as a sequence of verbs: reshape, join, filter, group, summarise. For omics data, Bioconductor adds objects that keep the count matrix, the feature table and the sample table together (SummarizedExperiment, phyloseq) and packages for models of counts (, , ) (Huber et al., 2015). Base R, tidyverse and Bioconductor combine well; use the one that makes the step clearest.
The analysis above used R as a calculator with packages. To read other people's scripts and to write your own functions you need the language itself. Gentleman's R Programming for Bioinformatics is the reference for the details (Read: ch. 2, language fundamentals; ch. 3, object-oriented programming; ch. 9, debugging and profiling). Every output below was produced by running the code.
x <- c(4.2, 3.8, NA, 5.1) # a numeric vector with one missing value
typeof(x) # "double"
mean(x) # NA: a missing value makes the mean missing
mean(x, na.rm = TRUE) # 4.366667: you must ask to drop it
is.na(x) # FALSE FALSE TRUE FALSE
1:10 %% 3 == 0 # a test on every element: FALSE FALSE TRUE FALSE FALSE TRUE ...
seq(0, 1, by = 0.25) # 0.00 0.25 0.50 0.75 1.00
rep(c("a", "b"), times = c(2, 3)) # "a" "a" "b" "b" "b"
v <- c(a = 1, b = 2, c = 3)
v["b"]; v[-1]; v[v > 1] # by name; all but the first; by a condition
In R there is no single number: everything is a vector, and operations apply to every element at once (this is R's "vectorisation", the counterpart of NumPy's). A vector has one type; mixing types coerces to the most general one: c(1, "a", TRUE) gives "1" "a" "TRUE", all text. NA is contagious, deliberately, so that a missing value cannot disappear silently. Indexing starts at 1, a negative index drops elements, and a logical vector selects the TRUE ones.
Two numeric facts that catch everyone: integer division is 5 %/% 2 (2) and the remainder 5 %% 2 (1); and decimals are stored approximately, so 0.1 + 0.2 == 0.3 is FALSE. Compare with isTRUE(all.equal(0.1 + 0.2, 0.3)), which is TRUE.
d <- factor(c("Fibre", "Control", "Antibiotic", "Control"))
levels(d) # "Antibiotic" "Control" "Fibre": alphabetical by default
as.integer(d) # 3 2 1 2: the codes underneath
levels(relevel(d, ref = "Control")) # "Control" "Antibiotic" "Fibre": Control is now the reference
l <- list(name = "run1", reads = c(10, 20), ok = TRUE)
l$reads[2] # 20: a list holds objects of any type; $ takes one out by name
str(l) # a compact description of any object: use it constantly
A factor is a categorical variable stored as integer codes with labels. It is what models and plots use for groups, and the first level is the reference in every model. A list holds anything. A data frame is a list of equal-length columns, which is why meta$Diet and meta[["Diet"]] both work.
meta[meta$Sample_type == "Sample" & meta$Age > 40, "Subject"][1:3] # rows by condition, one column: "P02" "P02" "P02"
sapply(meta[, c("Age", "BMI", "Butyrate_mM")], function(z) round(mean(z, na.rm = TRUE), 2))
# Age BMI Butyrate_mM
# 44.50 24.65 21.57
tapply(meta$Butyrate_mM, meta$Diet, median, na.rm = TRUE) # a summary per group
# Antibiotic Blank Control Fibre
# 8.735 NA 20.265 33.375
aggregate(Age ~ Diet, data = meta, FUN = mean) # the same with a formula, returns a data frame
vapply(split(meta$BMI, meta$Diet), median, numeric(1), na.rm = TRUE) # split into pieces, apply, and check the result type
The apply family replaces loops: sapply applies a function to each element and simplifies the result, tapply and aggregate do it by group, vapply is the safe form because you declare the type of the result. The NA printed for the blanks is correct: the four blanks have no butyrate measurement, and na.rm = TRUE removes missing values within a group but cannot invent a value for a group that has none.
shannon <- function(x) { p <- x[x > 0] / sum(x); -sum(p * log(p)) }
sh_all <- apply(counts[, bio], 2, shannon) # apply by column (2); rows would be 1
round(head(sh_all, 3), 3) # P01_Day0 4.291 P01_Day14 4.274 P01_Day28 4.313
safe_log <- function(x, pc = 0.5) { stopifnot(is.numeric(x)); log(x + pc) } # a check, and a default argument
tryCatch(safe_log("a"), error = function(e) conditionMessage(e)) # "is.numeric(x) is not TRUE"
rarefy_to <- function(depth) function(x) min(x, depth) # a function that returns a function (a closure)
rarefy_to(100)(250) # 100
A function returns the value of its last expression, takes arguments with defaults, and can refuse bad input with stopifnot or stop. tryCatch turns an error into a value you can handle, which is how a script processing a hundred samples survives one broken file and says which one. A function that returns a function is a closure; the inner function remembers depth. This is the idea behind the functions you pass to sapply and to the packages' options.
summary.runinfo <- function(object, ...) cat("Run", object$name, "with", length(object$reads), "samples\n")
r <- structure(list(name = "run1", reads = 1:3), class = "runinfo")
summary(r) # Run run1 with 3 samples
This is why summary(fit), plot(fit) and print(fit) do something different for every kind of object: the generic function looks at the object's class and calls the matching method (summary.lm, summary.runinfo). Bioconductor uses the stricter S4 system for objects such as SummarizedExperiment, but reading S3 is enough to understand most code.
This section's code was run with dplyr, tidyr, stringr, purrr and readr (R 4.4.3). The tidyverse is a family of packages that share one design: functions take a data frame first, return a data frame, and chain with the pipe |> (Wickham, 2014).
library(dplyr); library(tidyr); library(stringr); library(purrr); library(tibble)
tax <- read.delim("taxonomy.tsv", header = FALSE, row.names = 1)
lin <- str_split_fixed(tax$V2, ";", 7) # split the lineage into 7 columns
colnames(lin) <- c("kingdom","phylum","class","order","family","genus","species")
lin <- as_tibble(lin) |> mutate(across(everything(), ~ str_sub(.x, 4))) |> mutate(feature = rownames(tax), .before = 1)
lin |> count(phylum, sort = TRUE) |> head(4)
# phylum n
# 1 Firmicutes 196
# 2 Bacteroidota 92
# 3 Actinobacteriota 31
# 4 Proteobacteria 29
The same parsing as in the Python chapter, written as a sequence of verbs. mutate adds or changes columns, across(everything(), ...) applies a function to all columns, count tabulates. Now the table most analyses need, the share of each phylum by diet:
cnt <- counts[, bio]
long <- cnt |> rownames_to_column("feature") |>
pivot_longer(-feature, names_to = "sample", values_to = "count") |> # wide to long: 27,144 rows
left_join(rownames_to_column(meta, "sample")[, c("sample", "Diet")], by = "sample") |>
left_join(lin[, c("feature", "phylum")], by = "feature")
res <- long |> group_by(Diet, phylum) |> summarise(reads = sum(count), .groups = "drop") |>
group_by(Diet) |> mutate(share = round(100 * reads / sum(reads), 1)) |>
filter(phylum %in% c("Firmicutes", "Bacteroidota"))
res |> select(Diet, phylum, share) |> pivot_wider(names_from = phylum, values_from = share) |>
mutate(FB = round(Firmicutes / Bacteroidota, 2))
# Diet Bacteroidota Firmicutes FB
# 1 Antibiotic 59.5 21.6 0.36
# 2 Control 41.4 44.1 1.07
# 3 Fibre 31.1 54.2 1.74
Read the result as biology: the Firmicutes-to-Bacteroidota ratio is 1.07 in the controls, rises to 1.74 on the fibre diet and collapses to 0.36 on antibiotics, where Bacteroidota make up 60% of the reads. The code made one thing explicit that is easy to miss: pivot_longer turns the 377 × 72 table into 27,144 rows, one per feature and sample, which is the shape that group_by and ggplot2 want. The Python equivalent is melt and groupby (Python chapter, section 10). purrr::map_dbl(split(x, g), mean) applies a function to each group and returns numbers; stringr has str_extract("P01_Day14", "Day[0-9]+") (gives "Day14"), str_detect and str_replace for patterns.
readr::read_csv(file) # guesses each column's type and says so; a column of "1", "NA", "3" becomes numeric sapply(read.csv(file), class) # base R on the same file: id "character", val "integer"
Always check the types after reading: a stray text value turns a numeric column into text, a sample called 1E5 can be read as a number, and Excel has famously turned gene names into dates. stringsAsFactors, check.names, na.strings and colClasses are the arguments that control this.
m$Diet <- factor(m$Diet, levels = c("Control", "Fibre", "Antibiotic"))
a <- sh[m$Diet == "Control"]; b <- sh[m$Diet == "Antibiotic"]
t.test(a, b) # Welch: t = 5.0138, df = 27.069, p = 2.915e-05; mean difference 0.478, 95% CI 0.282 to 0.674
t.test(sh[m$Diet == "Fibre"], a) # Fibre against Control alone: p = 0.0061, difference -0.112, 95% CI -0.190 to -0.034
wilcox.test(a, b, exact = FALSE) # W = 491, p = 2.974e-05
summary(aov(sh ~ Diet, data = m)) # F = 19.02 on 2 and 69 df, p = 2.63e-07
TukeyHSD(aov(sh ~ Diet, data = m))
# diff lwr upr p adj
# Antibiotic-Control -0.4781 -0.6723 -0.2839 0.0000004
# Fibre-Control -0.1118 -0.3061 0.0824 0.3572431
# Fibre-Antibiotic 0.3662 0.1720 0.5605 0.0000743
var.test(a, b)$p.value # 1.9e-07: the variances differ (R's t.test does Welch by default for this reason)
power.t.test(delta = 0.5, sd = 1, power = 0.8) # n = 63.8 per group
chisq.test(table(m$Sex, m$Diet)) # X-squared = 3.2, p = 0.20: no imbalance of sex between diets
cor.test(m$Butyrate_mM, m$Fibre_g_day, method = "spearman", exact = FALSE) # rho = 0.507, p = 5.5e-06
The results are identical to those of the Python chapter (t = 5.01, p = 2.9e-05; U/W = 491; power n = 63.8; rho = 0.507), which is the best test of a data scientist's own work. The Tukey table is the right follow-up to a one-way analysis of variance: it compares all pairs and controls the error over the family. Read alone, it says that Antibiotic differs from both other diets and that Fibre does not differ from Control (adjusted p = 0.36). That last statement does not survive a second look: the pairwise Wilcoxon test of section 3 gave an adjusted p of 0.0045 for Fibre against Control, and a Welch test on that pair alone gives p = 0.006 (difference −0.11, 95% CI −0.19 to −0.03). The cause is variance. Tukey's method uses one pooled error from all three groups, and the antibiotic group, whose standard deviation is three times as large as the others, inflates it, hiding a small and consistent difference between two tight groups. Tests that assume equal variance are not neutral when the variances differ. Choose the test for the data, and when methods disagree, say so and show the plot. The chi-squared test checks the design: sex is balanced across the diets, so it cannot explain a diet effect. The Spearman correlation says that butyrate rises with fibre intake across samples, which is the biology the study was built around.
set.seed(1)
obs <- mean(a) - mean(b); pool <- c(a, b)
perm <- replicate(10000, { s <- sample(pool); mean(s[seq_along(a)]) - mean(s[-seq_along(a)]) })
(sum(abs(perm) >= abs(obs)) + 1) / (10000 + 1) # 9.999e-05
bs <- replicate(5000, mean(sample(a, replace = TRUE)) - mean(sample(b, replace = TRUE)))
round(quantile(bs, c(.025, .975)), 3) # 0.297 0.673
set.seed(1) makes the random draws repeat; without it nobody, including you next week, can reproduce the p value. replicate(n, expr) runs an expression n times and collects the results, the compact way to write a simulation. The permutation p value cannot be smaller than 1/(n+1) (here 0.0001); report it as p < 0.0001, never as 0. The bootstrap interval, 0.30 to 0.67 Shannon units, agrees with the t-test interval (0.28 to 0.67): the assumptions of the t-test were good enough.
fit <- lm(sh ~ Diet + Age + Sex, data = m) round(coef(fit), 3) # (Intercept) 4.052 DietFibre -0.104 DietAntibiotic -0.466 Age 0.004 SexM 0.007 round(summary(fit)$r.squared, 3) # 0.376 anova(fit) # sequential tests: Diet F = 19.09, p = 2.8e-07; Age p = 0.14; Sex p = 0.93 shapiro.test(residuals(fit))$p.value # 0.0033: the residuals are not normal library(emmeans) emmeans(fit, pairwise ~ Diet, adjust = "tukey")$contrasts # contrast estimate SE df t.ratio p.value # Control - Fibre 0.104 0.0837 67 1.246 0.4307 # Control - Antibiotic 0.466 0.0823 67 5.667 <.0001 # Fibre - Antibiotic 0.362 0.0814 67 4.446 0.0001
The coefficient for the antibiotic diet, −0.466, is the Python result (statsmodels gave −0.466 for the same formula). emmeans turns a model into the comparisons a reader wants, adjusted for the other variables and for the number of comparisons, with the groups averaged over the levels of the covariates (these tests share the pooled-variance weakness described above, so the Control–Fibre line, p = 0.43, should not be read as "no difference"); every result in a paper that says "adjusted for age and sex" should come from something like this. The normality of the residuals fails (p = 0.003), driven by the very variable antibiotic group; with 24 samples per group the estimates of the means are still sound, the standard errors are the weak point, which is why the permutation and rank-based results above matter. The sequential anova adds the terms in order, so the answer for Age depends on Diet having been entered first; with orthogonal groups, as here, it hardly matters, but in unbalanced data use the marginal tests (car::Anova(fit, type = 2) or drop1).
library(MASS) dd <- data.frame(y = unlist(cnt["ASV0001", ]), Diet = m$Diet, depth = colSums(cnt)) pg <- glm(y ~ Diet + offset(log(depth)), family = poisson, data = dd) sum(residuals(pg, "pearson")^2) / pg$df.residual # 1290: far above 1; Poisson is wrong nb <- glm.nb(y ~ Diet + offset(log(depth)), data = dd) round(exp(coef(nb)), 3) # (Intercept) 0.046 DietFibre 0.377 DietAntibiotic 2.730 (rate ratios against Control) round(nb$theta, 2) # 0.94 round(summary(nb)$coefficients[, 4], 5) # p values: Fibre 0.00106, Antibiotic 0.00074
The negative binomial model, with sequencing depth as an offset, gives exactly the rate ratios and p values that the Python chapter got with statsmodels (2.730 and 0.377; p = 0.0007 and 0.001). When two languages agree to three decimals you can be confident that you understood the model and not only the function. glm(..., family = binomial) is the logistic regression: glm(ab ~ shannon, family = binomial, data = dl) gives the intercept 25.07, the slope −6.158 and the odds ratio exp(−6.158) = 0.002 per Shannon unit.
library(nlme) mm <- lme(sh ~ Diet * Day, random = ~ 1 | Subject, data = m) # a random intercept for every person round(summary(mm)$tTable, 3) # Value Std.Error DF t-value p-value # (Intercept) 4.230 0.087 45 48.601 0.000 # DietFibre -0.002 0.123 21 -0.017 0.987 # DietAntibiotic -0.255 0.123 21 -2.074 0.051 # Day 0.000 0.004 45 0.061 0.952 # DietFibre:Day -0.008 0.006 45 -1.388 0.172 # DietAntibiotic:Day -0.016 0.006 45 -2.818 0.007 anova(mm) # Diet F = 14.06 (p = 0.0001); Day p = 0.0017; Diet:Day F = 3.97 (p = 0.026) intervals(mm, which = "fixed")$fixed["DietAntibiotic:Day", ] # -0.027 -0.016 -0.005 VarCorr(mm) # between subjects 0.0189 (sd 0.137), residual 0.0500 (sd 0.224)
Each person gets their own baseline, and the degrees of freedom show what the model understands: 21 for a diet effect (24 subjects minus the three group means) and 45 for the effects of time, not 69 as a model that treated the samples as independent would assume. The Diet:Day interaction (F = 3.97, p = 0.026) says that the groups change differently over time, and the interval for the antibiotic slope, −0.027 to −0.005 per day, quantifies it. nlme ships with R; the lme4 package (lmer(sh ~ Diet * Day + (1 | Subject))) is the usual alternative and the one in most tutorials (Bates et al., 2015). Faraway's Extending the Linear Model with R (Read: ch. 10 random effects, ch. 11 repeated measures) explains the models and their assumptions.
The PERMANOVA of section 6 is the starting point. Four questions decide whether it is valid and what it means, and they are the questions a reviewer asks (Anderson, 2001).
d <- vegdist(t(rel), "bray") adonis2(d ~ Diet, data = m, permutations = 999) # R2 = 0.149, F = 6.06, p = 0.001 h <- how(plots = Plots(strata = m$Subject, type = "free"), within = Within(type = "none"), nperm = 999) a2 <- adonis2(d ~ Diet, data = m, permutations = h) # R2 = 0.1494, p = 0.001 adonis2(d ~ Diet, data = m, permutations = how(blocks = m$Subject, nperm = 999))$`Pr(>F)`[1] # 1
The three lines answer the question of the repeated measures. The first treats the 72 samples as exchangeable, which they are not (three come from one person). The second permutes whole subjects as units, keeping each person's three samples together, which is the valid test for a diet that was given to a person and not to a sample; it gives the same result, R2 = 0.149 and p = 0.001 (999 permutations cannot give less). The third is a trap: blocks = Subject permutes samples within each person, but diet does not vary within a person, so every permutation reproduces the observed data and the p value is exactly 1. A p value of 1 here does not mean "no effect"; it means the test was set up with the wrong structure. When a result is surprising, check that the permutation scheme matches the design.
adonis2(d ~ Diet + Sex + Age, data = m, by = "margin", permutations = 999) # Df SumOfSqs R2 F Pr(>F) # Diet 2 2.545 0.147 6.183 0.001 # Sex 1 0.414 0.024 2.011 0.010 # Age 1 0.465 0.027 2.261 0.009 table(m$Batch, m$Diet) # Control Fibre Antibiotic # Run1 9 12 15 # Run2 15 12 9
by = "margin" tests each term after all the others, so the diet effect (R2 = 0.147) is adjusted for sex and age, and it hardly changes. Sex and age have small but nominally significant effects (about 2% of the variation each); these p values come from free permutations and are therefore too optimistic for the same reason as above. The table is the more important result: the two sequencing runs are not balanced across the diets (15 antibiotic samples in Run 1 but only 9 in Run 2, the reverse for controls). A batch effect would therefore be partly confounded with diet. A good analysis tests the batch (adonis2(d ~ Batch + Diet, ...)), plots the samples coloured by batch, and, if the batch matters, includes it in the model as a covariate or a stratum. Confounding between a technical variable and the question cannot be repaired afterwards by statistics alone, only reduced; it is prevented by randomising samples to runs.
pc <- rda(t(log(as.matrix(cnt) + 0.5)))
ef <- envfit(pc, m[, c("Butyrate_mM", "Fibre_g_day", "BMI")], permutations = 999)
round(ef$vectors$r, 3) # Butyrate_mM 0.792 Fibre_g_day 0.400 BMI 0.147
envfit regresses each variable on the ordination axes. Butyrate explains 79% of the variation along its best direction in the plot, much more than fibre intake (40%) or BMI (15%): the butyrate gradient is a strong organising axis of this community (the p values, 0.001, 0.001 and 0.002, again assume exchangeable samples). Use it to describe a plot, and confirm anything important with a model.
set.seed(1) rc <- rrarefy(t(cnt), 900) # subsample every sample to 900 reads (the smallest sample has 949) tapply(rowSums(rc > 0), m$Diet, median) # Control 119.5 Fibre 114.0 Antibiotic 95.0 tapply(rarefy(t(cnt), 900), m$Diet, median) # the expected value, without random sampling: 122.0 112.2 93.5
Rarefying cuts every sample down to the same number of reads so that richness can be compared. It throws data away (here all but 900 reads of samples with up to 80,000), which is why it is controversial for testing differences between groups; for diversity indices and richness, the expected value computed by rarefy() is better than one random draw, as it has no sampling noise. The conclusion does not change: richness is lowest in the antibiotic group. Asking rarefy(t(cnt), 2000) fails with a warning, because the smallest sample (949 reads) cannot supply 2,000: the function does not silently do something else, and you should read warnings.
hc <- hclust(d, "average"); table(cutree(hc, 3), m$Diet) # Control Fibre Antibiotic # 1 24 18 7 # 2 0 6 0 # 3 0 0 17
Hierarchical clustering into three groups isolates 17 of the 24 antibiotic samples in one cluster; all 24 controls fall in the large cluster, together with 18 fibre samples and 7 antibiotic samples. It agrees with the PERMANOVA: the antibiotic effect is the strongest, and the fibre effect is subtler.
Testing 300 features one by one with a Wilcoxon test is a start; testing them with a model that shares information across features is better, and it is what the dedicated packages do. with converts counts to log counts per million, estimates how the variance depends on the mean and gives every observation a weight, then fits a linear model per feature with moderated statistics (Law et al., 2014; Ritchie et al., 2015).
library() design <- model.matrix(~ Diet, data = m) # Control is the reference v <- (cnt, design) # counts to weighted log-CPM fit <- eBayes(lmFit(v, design)) sum(topTable(fit, coef = "DietAntibiotic", number = Inf)$adj.P.Val < 0.05) # 87 features at FDR 5% sum(topTable(fit, coef = "DietFibre", number = Inf)$adj.P.Val < 0.05) # 14 dup <- duplicateCorrelation(v, design, block = m$Subject) # 0.195: samples of one person are correlated fit2 <- eBayes(lmFit(v, design, block = m$Subject, correlation = dup$consensus.correlation)) sum(topTable(fit2, coef = "DietAntibiotic", number = Inf)$adj.P.Val < 0.05) # 52 sum(topTable(fit2, coef = "DietFibre", number = Inf)$adj.P.Val < 0.05) # 1
This is the most important result of the chapter. The first model treats the 72 samples as independent: 87 features differ with antibiotics and 14 with fibre. Once the model knows that three samples come from one person (the within-person correlation is 0.195), the numbers fall to 52 and 1. The count for antibiotics fell by 40% and the count for fibre from 14 to 1: a large part of the first answer came from counting the same 24 people three times. This is not a technicality: an analysis that ignores repeated measures reports too many findings, and here would have overstated the effect of fibre.
Two further remarks. The Wilcoxon test on CLR values (section 4) found 74 features and -; they share only 39 features. Differential-abundance methods disagree strongly in published benchmarks as well (Nearing et al., 2022), so a careful report uses the method that matches the design, states it, and trusts the features that several methods agree on (this is the idea behind the consensus in the MicroRes tool). Second, and are the standard negative binomial tools for count data (Love et al., 2014); they were not installed on the machine used for this chapter, so they were not run, and the design formula ~ Diet with a subject block, or a model that includes the subject as a term for paired designs, is how they take the same structure into account.
f <- function(x) { if (x < 0) stop("x must not be negative: ", x); sqrt(x) }
tryCatch(f(-1), error = function(e) paste("caught:", conditionMessage(e))) # "caught: x must not be negative: -1"
withCallingHandlers(as.integer("a"), warning = function(w) { message("warning seen: ", conditionMessage(w)); invokeRestart("muffleWarning") })
g <- function() { on.exit(cat("cleaned up\n")); stop("failed") }; try(g(), silent = TRUE) # cleaned up
Reading an error is the first skill: the message names the function and often the cause, and traceback() after a failure shows the chain of calls. stop raises an error, warning a warning, message information; tryCatch handles them, and on.exit guarantees cleanup (closing a file, removing a temporary folder) even when the function fails. To look inside a function while it runs, put browser() in it, or call debugonce(f), and step through line by line.
x <- runif(2e6); y <- numeric(length(x)) system.time(for (i in seq_along(x)) y[i] <- x[i] * 2) # 0.119 s: a loop of 2 million steps system.time(y2 <- x * 2) # 0.007 s: the same result, 17 times faster v <- c(); system.time(for (i in 1:50000) v <- c(v, i)) # 1.8 s: growing a vector copies it every time v2 <- numeric(50000); system.time(for (i in 1:50000) v2[i] <- i) # 0.003 s: allocate once, fill in Rprof(tf <- tempfile()); invisible(sapply(1:200, function(i) sort(runif(1e4)))); Rprof(NULL); summaryRprof(tf)$by.total
The two rules of fast R are to vectorise and to allocate once: growing a result inside a loop with c() is the classic mistake, 600 times slower here at 50,000 elements, and the cost grows with the square of the length. A modern R compiles simple loops, so the plain loop is only 17 times slower than the vectorised call and not hundreds. Measure before optimising (system.time, and Rprof for a profile of where the time goes), because the slow part is rarely where you expect.
#!/usr/bin/env Rscript
# summarise.R: Shannon index of every sample in a count table (features x samples)
args <- commandArgs(trailingOnly = TRUE)
if (length(args) < 1) stop("usage: summarise.R counts.tsv [out.tsv]")
out <- if (length(args) >= 2) args[2] else "shannon.tsv"
shannon <- function(x) { p <- x[x > 0] / sum(x); -sum(p * log(p)) }
t <- read.delim(args[1], row.names = 1, check.names = FALSE)
res <- data.frame(sample = colnames(t), shannon = apply(t, 2, shannon))
write.table(res, out, sep = "\t", quote = FALSE, row.names = FALSE)
cat(ncol(t), "samples written to", out, "\n")
$ Rscript summarise.R counts.tsv shannon.tsv 76 samples written to shannon.tsv $ Rscript summarise.R Error: usage: summarise.R counts.tsv [out.tsv]
A script that takes its inputs from the command line and stops with a usage message when they are missing can be run in a loop, on a cluster, or from a workflow manager. The output file is identical to that of the Python script in the Python chapter (the same 76 Shannon values, 4.2913 for the first sample).
sessionInfo() (R, platform and the version of every loaded package) and save it with the results. The renv package gives a project its own library and a lock file, so that the same versions can be restored later; a container does the same for the whole system. Package behaviour changes between versions, and several results in this chapter would not be reproducible without knowing them.set.seed() at the top of any script that permutes, samples or starts an algorithm at random. set.seed(42); sample(1:10, 3) returns 1 5 10 every time.usethis::create_package, a DESCRIPTION file, roxygen comments and testthat tests). Gentleman's book (ch. 7) explains the structure, and it is the way Bioconductor software is built (Huber et al., 2015).renv or a container, and to note which R was used.Bioconductor is the collection of R packages for genomic data. Its strength is a small number of shared objects, so that a tool written by one group works on the output of another (Huber et al., 2015). Three of them cover most uses. Everything here was run with Biostrings 2.72 and the GenomicRanges and SummarizedExperiment packages on the demonstration data.
library(Biostrings)
seqs <- readDNAStringSet("sequences.fasta")
length(seqs); summary(width(seqs)) # 377 sequences, every one 253 bases (the demonstration data are simulated)
subseq(seqs[[1]], 1, 30) # AGTGACTTAAGCACTATAGCCTAGTGCCAG
summary(as.numeric(letterFrequency(seqs, "GC", as.prob = TRUE))) # GC fraction: 0.482 to 0.561, median 0.518
alphabetFrequency(seqs[1:3])[, 1:4] # counts of A, C, G, T per sequence
reverseComplement(subseq(seqs[[1]], 1, 12)) # GCTTAAGTCACT
sum(vcountPattern("GTGCCAGC", seqs) > 0) # 1 sequence contains the 8-base pattern exactly
start(matchPattern("GTGCCAGC", seqs[[1]], max.mismatch = 1)) # 24: found at position 24 when one mismatch is allowed
round(consensusMatrix(seqs[1:20], as.prob = TRUE)[c("A","C","G","T"), 1:5], 2) # base frequencies at the first 5 positions
# [,1] [,2] [,3] [,4] [,5]
# A 0.15 0.00 0.00 0.05 1
# G 0.00 0.90 0.00 0.95 0
# T 0.80 0.05 0.95 0.00 0
A DNAStringSet is a set of sequences that behaves like a vector: you subset it, measure widths, count letters and search for patterns in all of them at once, quickly, because the work is done in compiled code. consensusMatrix shows the first columns of an alignment-like set: position 2 is G in 90% of the first 20 sequences, position 3 is T in 95%, and position 5 is A in all of them (a conserved start, which is what primer sites look like). Pairwise and multiple alignment in Bioconductor lives in the pwalign and msa packages, which were not installed here, so no alignment is shown; the Biopython chapter showed alignment in Python. Gentleman's book (Read: ch. 5, working with character data) and the Bioinformatics with R Cookbook (Read: ch. 3, sequence analysis with R) cover these tasks.
library(GenomicRanges)
gr <- GRanges("chr1", IRanges(c(100, 400, 800), width = c(200, 300, 150)), strand = c("+", "-", "+"), gene = c("g1", "g2", "g3"))
reads <- GRanges("chr1", IRanges(c(150, 350, 420, 900), width = 50))
countOverlaps(gr, reads) # 1 1 1: reads falling in each gene
findOverlaps(reads, gr) # read 1 with gene 1, read 3 with gene 2, read 4 with gene 3 (read 2 hits none)
start(flank(gr, 50, start = TRUE)) # 50 700 750: the 50 bases upstream of each gene, on the correct side of each strand
(A toy example with invented coordinates; no real genome is involved.) The last line is the reason this object exists: for a gene on the minus strand, "upstream" is at the higher coordinate, and flank knows that, whereas hand-written interval arithmetic often does not. Counting reads in genes, finding which variants fall in an exon, and extracting promoters are overlap problems, and findOverlaps solves them efficiently for millions of ranges.
library(SummarizedExperiment)
se <- SummarizedExperiment(assays = list(counts = counts), # features x samples
colData = DataFrame(meta), # one row per sample
rowData = DataFrame(lineage = tax[rownames(counts), "V2"])) # one row per feature
dim(se) # 377 76
dim(se[, se$Diet == "Fibre"]) # 377 24: choosing samples by a metadata column subsets the counts and the metadata together
assay(se, "rel") <- sweep(assay(se, "counts"), 2, colSums(assay(se, "counts")), "/") # a second assay
assayNames(se) # "counts" "rel"
keep <- rowSums(assay(se, "counts") > 0) / ncol(se) >= 0.10
dim(se[keep, ]) # 300 76: the filter of section 2, applied to every part of the object at once
The count table, the sample table and the feature table sit in one object whose dimensions are tied together. Subsetting the samples subsets the counts and the metadata at once, so the mismatch of "sample order in the counts is not the order in the metadata" cannot happen. That error has produced wrong published results; avoiding it by construction is the main reason to use these objects. , and accept a SummarizedExperiment, as do many single-cell tools, and phyloseq is the equivalent for microbiome data.
The same simulations as in the Python chapter (section 19), written with replicate. R's random numbers differ from NumPy's, so the values differ slightly in the third decimal; the conclusions are the same.
set.seed(7); n <- 24; B <- 10000 tp <- function(delta) replicate(B, t.test(rnorm(n), rnorm(n, delta))$p.value) mean(tp(0) < 0.05) # 0.0513: with no difference, about 5% of experiments are "significant" mean(tp(.5) < 0.05) # 0.3945: power of 24 per group for a medium effect, about 39% p <- sapply(1:300, function(i) t.test(rnorm(n), rnorm(n, if (i <= 60) 1 else 0))$p.value) # 60 real differences, 240 null c(sum(p < 0.05), sum(p[61:300] < 0.05)) # 73 hits, 18 of them false q <- p.adjust(p, "BH"); c(sum(q < 0.05), sum(q[61:300] < 0.05)) # 50 hits, 3 false
Without correction, 18 of 73 hits are false (25% in this run; the Python run gave 9 of 61, and the figure varies from run to run); with the Benjamini–Hochberg correction, 3 of 50 (6%). The point is the same as in Python: power is limited at this sample size, and without multiplicity control a screen of many features reports many false discoveries. The design in section 11 and the model in section 14 are the tools to keep that under control.
On this analysis they give the same answers where the methods are the same. Choose by the question and the team: R has the larger collection of statistical methods for ecology and omics, and its model formulas (y ~ a + b) and graphics are hard to beat; Python is better for software that others will run, for machine learning pipelines and for gluing tools together. A good bioinformatician reads both, and a good analysis can be rerun by someone who uses only one: give the data, the script, the versions and the seed.
Use the demonstration files. The answers are below each list; do the exercise first.
Fibre_g_day) in each diet group, and what does it tell you about the design?richness(x) that counts the features present in a sample, apply it to every sample of the unfiltered table and give the median per diet.how(blocks = m$Subject) gave a PERMANOVA p value of 1 in section 13, and what you would use instead.1. tapply(m$Fibre_g_day, m$Diet, mean): Antibiotic 18.53, Control 18.54, Fibre 29.42 g per day. The fibre group eats about 11 g more, so the intervention did what it was meant to do, while antibiotic and control groups have the same intake: fibre intake cannot explain the antibiotic effect.
2. richness <- function(x) sum(x > 0); tapply(apply(counts[, bio], 2, richness), m$Diet, median) gives Antibiotic 141, Control 185.5, Fibre 172.5.
3. Group by diet and phylum, sum the reads, divide by the diet total, then divide the two phyla: 0.36 for antibiotics, 1.07 for controls and 1.74 for fibre (section 10).
4. t.test(sh[m$Diet == "Fibre"], sh[m$Diet == "Control"]): p = 0.0061, difference −0.112 (95% CI −0.190 to −0.034). Tukey's 0.36 uses a pooled error dominated by the very variable antibiotic group, so it is too conservative for this pair. The rank-based test of section 3 (adjusted p 0.0045) agrees with the Welch test.
5. Means: Antibiotic 11.67, Control 21.50, Fibre 31.53 mM; medians: 8.74, 20.26, 33.38. The antibiotic mean is above its median because a few samples have a high butyrate value (the distribution is skewed to the right); the median describes the typical sample better there. Always look at the distribution before choosing the summary.
6. blocks permutes the labels within each person. Diet is the same for all three samples of a person, so every permutation reproduces the original table and gives the same F; the observed F is never exceeded by a larger one, but is always matched, so p = 1. For a variable that differs between people, permute whole subjects: how(plots = Plots(strata = m$Subject, type = "free"), within = Within(type = "none")).
Diet explains about 15% of the variation in community composition, a moderate and significant effect. The groups do not differ detectably in spread, so the result reflects a shift in the centre of the groups.
No. The two programs treat tied values in the rank test differently, which moves a few borderline p values. Report the method and the effect sizes, not only the count.
diversity(t(cnt), "shannon") need the transpose?vegan expects samples in rows and features in columns; the count table has features in rows.
DietAntibiotic = -0.48. What is it relative to?The reference level, the first level of the factor: Control. Antibiotic samples have a Shannon index 0.48 lower than control samples, holding the other variables fixed.
Bioinformatics files are text, and the shell is the fastest way to look inside them. This chapter works through nine short labs, in the order a newcomer meets the formats: FASTA, FASTQ and Phred quality, motifs, on the command line, annotation files, alignments, variants, read counts, and finally a pipeline script that ties them together. Everything runs on one real genome, the 5,386-base phage phiX174, and every output shown is the real output of the command above it.
Chapter P1 introduced the shell, P3 the text tools (grep, awk, sed, join) and P4 the language of scripts. This chapter uses all three on the file formats of sequence analysis, and it is organised the way a practical course is: one lab per format, each ending with a result you can check. The labs follow the order of Hendrix, Applied Bioinformatics (Lab 1 FASTA files and GNU/Linux, Lab 2 FASTQ and quality scores, Lab 3 motifs, Lab 4 on the command line, Lab 6 annotation data, Lab 7 RNA-seq), and the habits behind them come from Buffalo, Bioinformatics Data Skills (Chapters 3, 7, 9, 10, 11 and 12). The examples, data and numbers are our own.
The tools beyond the standard shell are the ones every bioinformatician installs first: (sequence files), +, , , , , and for gene prediction. All of them are on Bioconda. awk examples that use and() need gawk; the plain awk on some systems (mawk) lacks it.
The reference is the Illumina PhiX control genome, phiX174 (accession NC_001422.1), 5,386 bases. It is a good teaching genome: tiny, completely known, and the first DNA genome ever sequenced (Sanger et al. 1977). It is also the control that is spiked into many Illumina runs, so you will meet its reads in real data. The copy used here is the one in the BBTools resources (phix174_ill.ref.fa.gz); the current record of the same accession differs from it at five positions, which is a reminder that a reference has versions and that you should record which one you used.
The reads are simulated by a small seeded script (shown at the end of this section) so that we know the truth. S1 has 6,000 reads of 100 bases from the unchanged genome. S2 has 6,000 reads from a copy in which we planted six single-base changes, one 3-base deletion and one 2-base insertion. Quality values follow a typical Illumina pattern (high at the start, falling along the read), and a base is changed with exactly the probability its quality states. queries.fa holds five 150-base pieces of phiX174 with 2 to 5 % differences and one random decoy. Because the truth is known, each lab can end with a check.
Set up a folder with data/ for inputs and results/ for everything we make. Raw data go in data/ and are never edited; every command writes its output to results/.
$ ls -l data | awk 'NR>1 {print $5, $9}'
1360723 S1.fq
1360750 S2.fq
1951 genes.gff
1825 phix174.fa.gz
5496 phix_mutated.fa
88 planted_variants.tsv
985 queries.fa
$ gunzip -kf data/phix174.fa.gz && head -c 200 data/phix174.fa
>gi|9626372|ref|NC_001422.1| Coliphage phiX174, complete genome
GAGTTTTATCGCTTCCATGACGCAGAAGTTAACACTTTCGGATATTTCTGATGAGTCGAAAAATTATCTT
GATAAAGCAGGAATTACTACTGCTTGTTTACGAATTAAATCGAAGTGGACTGCTGGCGGAAAATG
Save the script as simulate.py. It needs only Python 3. Run python3 simulate.py data/phix174.fa.gz data with the gzipped reference in data/; the random seed is fixed, so you get the same reads. The genes were predicted with prodigal -i data/phix174.fa -f gff -o data/genes.gff -p meta.
"""Simulate the lab data for the bash chapter: single-end Illumina-like reads with position-dependent Phred qualities, from the
phiX174 reference (wild type, sample S1) and from a copy with planted variants (sample S2). Seeded, so the chapter can be rerun."""
import gzip
import random
import sys
random.seed(174)
ref_gz, out = sys.argv[1], sys.argv[2]
seq = "".join(l.strip() for l in gzip.open(ref_gz, "rt") if not l.startswith(">")).upper()
L = len(seq)
# planted variants (1-based positions on the reference): six SNPs, one 3-bp deletion, one 2-bp insertion
snps = {500: "A", 1200: "C", 2100: "T", 3000: "G", 4100: "A", 5000: "C"}
mut = list(seq)
planted = []
for pos, alt in snps.items():
if mut[pos - 1] == alt:
alt = "T" if alt != "T" else "G"
planted.append((pos, mut[pos - 1], alt))
mut[pos - 1] = alt
del_at = 2600 # delete 3 bases after position 2600
deleted = "".join(mut[del_at:del_at + 3])
mut[del_at:del_at + 3] = ["", "", ""]
ins_at = 3500 # insert 'GG' after position 3500
mut[ins_at - 1] = mut[ins_at - 1] + "GG"
mutated = "".join(mut)
open(f"{out}/planted_variants.tsv", "w").write("pos\tref\talt\n" + "".join(f"{p}\t{r}\t{a}\n" for p, r, a in planted)
+ f"{del_at}\t{seq[del_at-1]+deleted}\t{seq[del_at-1]}\n{ins_at}\t{seq[ins_at-1]}\t{seq[ins_at-1]}GG\n")
open(f"{out}/phix_mutated.fa", "w").write(">phiX174_mutated planted variants\n" + "\n".join(mutated[i:i + 70] for i in range(0, len(mutated), 70)) + "\n")
comp = str.maketrans("ACGT", "TGCA")
def qual_profile(n):
q = []
for i in range(n):
base = 38 - 12 * (i / n) ** 2 # quality decays along the read
q.append(max(2, min(41, int(random.gauss(base, 2.5)))))
return q
def reads(genome, n, length, name):
G = len(genome)
with open(f"{out}/{name}.fq", "w") as fh:
for i in range(n):
start = random.randrange(0, G - length)
frag = genome[start:start + length]
if random.random() < 0.5: # the fragment is read from the other strand
frag = frag.translate(comp)[::-1]
r = list(frag)
q = qual_profile(length) # quality belongs to the sequencing cycle, not to the genome strand
for j, qq in enumerate(q):
if random.random() < 10 ** (-qq / 10): # an error with exactly the probability the quality states
r[j] = random.choice([b for b in "ACGT" if b != r[j]])
s = "".join(r)
fh.write(f"@{name}_read{i + 1:05d} sim:{start + 1}\n{s}\n+\n{''.join(chr(33 + x) for x in q)}\n")
reads(seq, 6000, 100, "S1")
reads(mutated, 6000, 100, "S2")
# queries: 5 pieces of phiX with 2-5% differences, and one decoy that is not phiX
with open(f"{out}/queries.fa", "w") as fh:
for k, st in enumerate((300, 1500, 2800, 3900, 4800), 1):
piece = list(seq[st:st + 150])
for _ in range(int(150 * random.choice((0.02, 0.03, 0.05)))):
j = random.randrange(150)
piece[j] = random.choice([b for b in "ACGT" if b != piece[j]])
fh.write(f">q{k}_from_{st + 1}\n{''.join(piece)}\n")
fh.write(">q6_decoy\n" + "".join(random.choice("ACGT") for _ in range(150)) + "\n")
print("planted:", planted, "del", del_at, deleted, "ins", ins_at)
A FASTA file is a header line that starts with >, followed by the sequence, usually wrapped at a fixed width. Two facts decide how you handle it in the shell: the sequence of one record spans many lines, and > is the only mark that separates records. Counting records is therefore a count of headers, and measuring a sequence means removing the line breaks first.
$ grep -c '>' data/phix174.fa 1 $ grep -v '>' data/phix174.fa | tr -d '\n' | wc -c 5386
The length, 5,386, agrees with the genome's published size. Base composition comes next: fold -w1 puts one base on each line, and sort | uniq -c counts them (P3). Then a single awk command gives the GC content, with gsub doing the counting (it returns the number of replacements).
$ grep -v '>' data/phix174.fa | tr -d '\n' | fold -w1 | sort | uniq -c
1292 A
1155 C
1253 G
1686 T
$ grep -v '>' data/phix174.fa | tr -d '\n' | awk '{n=length($0); gc=gsub(/[GC]/,""); printf "GC = %.2f%% of %d bases\n", 100*gc/n, n}'
GC = 44.71% of 5386 bases
GC content of 44.71 % is moderate and is one of the numbers that identify an organism, a contig or a contaminant. A single global value hides local differences, and the shell can slide a window along the genome: fold -w 1000 cuts the sequence into 1,000-base pieces, and awk prints the GC of each one. A window that departs from its neighbours points to a foreign segment, a repeat or a gene island.
$ grep -v '>' data/phix174.fa | tr -d '\n' | fold -w 1000 | awk '{n=length($0); gc=gsub(/[GC]/,""); printf "%d\t%.1f\n", (NR-1)*1000+1, 100*gc/n}'
1 44.7
1001 45.0
2001 43.2
3001 44.7
4001 45.8
5001 45.1
The reverse complement matters in every analysis, because DNA has two strands and a gene may lie on either. For a short sequence the shell does it in one line: rev reverses the characters and tr ACGT TGCA swaps each base for its partner. (For IUPAC codes and for whole files use seq -r -p.) Compare the first thirty bases with their reverse complement:
$ sed -n 2p data/phix174.fa | cut -c1-30; sed -n 2p data/phix174.fa | cut -c1-30 | rev | tr ACGT TGCA GAGTTTTATCGCTTCCATGACGCAGAAGTT AACTTCTGCGTCATGGAAGCGATAAAACTC
The two most common problems with FASTA files are invisible. Windows line endings: a file edited on Windows ends every line with a carriage return (\r), which becomes part of the sequence, so lengths are off by one per line and grep patterns at the end of a line never match. file names the problem, grep -c $'\r' counts the lines it affects, and tr -d '\r' removes it. The second command below is the fix; zero lines are left with a carriage return.
$ sed 's/$/\r/' data/phix174.fa > results/windows.fa; file results/windows.fa | cut -d: -f2; grep -c $'\r' results/windows.fa; tr -d '\r' < results/windows.fa | grep -c $'\r' || true ASCII text, with CRLF line terminators 78 0
The other is awkward headers. The header of this record, gi|9626372|ref|NC_001422.1| Coliphage phiX174, complete genome, contains vertical bars and spaces, which many tools treat as separators. sed replaces the bars and cuts everything after the first space. When you need one record out of many, awk can switch a flag on at the header you want and print while it is on; grep -p NAME does the same for lists of names. Finally, check your own arithmetic against a tool: stats reports the same 5,386 bases.
$ grep '>' data/queries.fa | head -3; grep '>' data/phix174.fa | sed 's/|/_/g; s/ .*//'
>q1_from_301
>q2_from_1501
>q3_from_2801
>gi_9626372_ref_NC_001422.1_
$ awk '/^>/ {keep = ($1 == ">q3_from_2801")} keep' data/queries.fa
>q3_from_2801
CGAGCAATAATGTTTATGCTGGTTTCATGGTTTGGTCTAACTTTACCGCTACTAAATGCCGCGGATTGGTTTCGCTGAATCAGGTTATTAAAGAGATTGTTTGTCTCCAGCCACTTAAGTGAGGTGATTTATGTTTGGTGCTATTGCTGG
$ stats data/phix174.fa
file format type num_seqs sum_len min_len avg_len max_len
data/phix174.fa FASTA DNA 1 5,386 5,386 5,386 5,386
A FASTQ record has four lines: an identifier that starts with @, the sequence, a separator line that starts with +, and one quality character per base. The quality of a base is a Phred score, Q = −10 log10 P, where P is the probability that the base is wrong. Q10 is a 1-in-10 chance of error, Q20 one in 100, Q30 one in 1,000. In the modern (Sanger or Illumina 1.8) encoding the character is the score plus 33 turned into an ASCII character: I is Q40, 5 is Q20, + is Q10 (Cock et al. 2010). Older Illumina files used an offset of 64, so a file whose first character is lower case needs checking before you trust it.
$ head -4 data/S1.fq
@S1_read00001 sim:39
TCTCATTTTCCGCCAGCAGTCCACTTCGATTTAATTCGTAAACAAGCAGTAGTAATTCCTGCTTTATCAAGATAATTTTTCGACTCATCAGAAATATCCG
+
IGGHHEEDCEECECJGIHIGFHFECCGFGHGHJFCFGDABDHGDGBIDG@ICIG@@GAB@D@?CBD>J?=DC@@@A@:;==@@B@<A>???>><=9=<<:
$ echo $(( $(wc -l < data/S1.fq) / 4 )) reads
6000 reads
$ awk 'NR%4==2 {print length($0)}' data/S1.fq | sort | uniq -c
6000 100
$ awk 'BEGIN {for (q=10; q<=40; q+=10) printf "Q%d\terror probability %g\n", q, 10^(-q/10)}'
Q10 error probability 0.1
Q20 error probability 0.01
Q30 error probability 0.001
Q40 error probability 0.0001
Counting reads is a line count divided by four, and the lengths are tallied with awk on every fourth line (here all 100). To use the qualities you have to decode them. awk has no built-in function that turns a character into its number, so the program builds a lookup table once in BEGIN, using sprintf("%c", i), and then adds up the table values character by character. The command below writes the mean quality of every read to a file and shows the first four:
$ awk 'BEGIN {for (i=33; i<127; i++) ord[sprintf("%c", i)] = i-33}
> NR%4==1 {id=substr($1,2)}
> NR%4==0 {n=length($0); s=0; for (i=1; i<=n; i++) s+=ord[substr($0,i,1)]; printf "%s\t%.2f\n", id, s/n}' data/S1.fq > results/S1.meanq.tsv; head -4 results/S1.meanq.tsv
S1_read00001 33.96
S1_read00002 33.56
S1_read00003 33.56
S1_read00004 33.96
One more awk turns that file into a summary. All 6,000 reads have a mean quality of at least 30, which looks excellent, and the mean of the means is 33.55. A mean hides the shape: quality falls along a read, and the per-position view shows where. Positions 1, 34, 67 and 100 give a sketch of the usual curve; a real run is read with a full per-position plot ( and draw one, P1).
$ awk '{s+=$2; if ($2>=30) k++} END {printf "mean of the read means %.2f; %d of %d reads (%.1f%%) have a mean quality of at least 30\n", s/NR, k, NR, 100*k/NR}' results/S1.meanq.tsv
mean of the read means 33.55; 6000 of 6000 reads (100.0%) have a mean quality of at least 30
$ awk 'BEGIN {for (i=33; i<127; i++) ord[sprintf("%c", i)] = i-33}
> NR%4==0 {for (i=1; i<=length($0); i++) sum[i]+=ord[substr($0,i,1)]; n++}
> END {for (i=1; i<=100; i+=33) printf "position %d\tmean quality %.1f\n", i, sum[i]/n; printf "position 100\tmean quality %.1f\n", sum[100]/n}' data/S1.fq
position 1 mean quality 37.4
position 34 mean quality 36.2
position 67 mean quality 32.3
position 100 mean quality 25.7
position 100 mean quality 25.7
A filter is a program that keeps a record only when a test passes. Because a FASTQ record is four lines, the program collects four lines before it decides. Here every read passes, since the simulated reads are good; on real data you would keep a read only if its mean quality or its worst window meets a threshold. A tool such as stats -a gives the same kind of summary in one line (Q20 and Q30 shares, average quality, N50), and is the right way to check a hand-made calculation:
$ awk 'BEGIN {for (i=33; i<127; i++) ord[sprintf("%c", i)] = i-33}
> {rec[NR%4]=$0}
> NR%4==0 {n=length($0); s=0; for (i=1; i<=n; i++) s+=ord[substr($0,i,1)]; if (s/n >= 30) print rec[1] "\n" rec[2] "\n" rec[3] "\n" rec[0]}' data/S1.fq > results/S1.q30.fq; echo $(( $(wc -l < results/S1.q30.fq) / 4 )) reads kept
6000 reads kept
$ stats -a data/S1.fq | cut -f1,4,6,8,9,13-17
file format type num_seqs sum_len min_len avg_len max_len Q1 Q2 Q3 sum_gap N50 N50_num Q20(%) Q30(%) AvgQual GC(%) sum_n
data/S1.fq FASTQ DNA 6,000 600,000 100 100 100 100 100 100 0 100 3,000 99.98 80.87 31.24 44.8 0
Notice the difference: AvgQual in the table (31.24) is lower than our 33.55. It is not an error. A mean of quality scores and a mean of error probabilities converted back to a score are different quantities, and reports the second, which is the more honest summary because a few bad bases count for more (chapter P9 reproduces both numbers in Python). Always know which one a tool reports.
A motif is a short pattern that means something: a restriction site, a ribosome-binding site, a transcription-factor binding site. In the shell, the simplest motif search is grep -o (print only the match) with -b (print the byte offset). Because the sequence has been joined onto one line, the offset is the position, counted from zero.
$ grep -v '>' data/phix174.fa | tr -d '\n' | grep -ob 'AGGAG' 232:AGGAG 835:AGGAG 2382:AGGAG 3540:AGGAG 5062:AGGAG
The pattern AGGAG occurs five times. DNA has two strands, so a motif that is not its own reverse complement has to be searched on both: the other strand reads the reverse complement. GGAGG (the core of the Shine–Dalgarno sequence that helps a ribosome find a start codon) occurs four times as given and six times as CCTCC, which is the same motif on the other strand. A search of one strand would have missed more than half.
$ seq=$(grep -v '>' data/phix174.fa | tr -d '\n') $ fwd=$(echo "$seq" | grep -o 'GGAGG' | wc -l) $ rev=$(echo "$seq" | grep -o 'CCTCC' | wc -l) $ echo "GGAGG on the given strand: $fwd; its reverse complement CCTCC (the motif on the other strand): $rev" GGAGG on the given strand: 4; its reverse complement CCTCC (the motif on the other strand): 6
Real motifs are rarely exact. The IUPAC code writes R for A or G, Y for C or T, W for A or T, S for C or G and N for any base, and a pattern is a regular expression once these are replaced. A small shell function does the translation. A ribosome-binding site followed by a start codon after a gap can then be written AGGAG[ACGT]{4,10}ATG, and grep -E finds it. The three hits are real: positions 835, 2,382 and 5,062 lie just before genes predicted by , whose own output records an AGGAG motif for the same genes.
$ iupac2re() { echo "$1" | sed 's/R/[AG]/g; s/Y/[CT]/g; s/W/[AT]/g; s/S/[CG]/g; s/N/[ACGT]/g'; }
$ iupac2re AGGAGRNNNNATG
AGGAG[AG][ACGT][ACGT][ACGT][ACGT]ATG
$ grep -v '>' data/phix174.fa | tr -d '\n' | grep -obE 'AGGAG[ACGT]{4,10}ATG'
835:AGGAGTGATGTAATG
2382:AGGAGTTTAATCATG
5062:AGGAGCTAAAGAATG
The most important question about a motif count is: more or fewer than chance? In a random sequence of length L, a pattern of k bases is expected about (L − k + 1) / 4k times. For a six-base site in 5,386 bases that is 1.31, and for the four-base site GATC it is 21.03. The loop below counts five famous sites and prints the expectation next to each:
$ L=$(grep -v '>' data/phix174.fa | tr -d '\n' | wc -c)
$ seq=$(grep -v '>' data/phix174.fa | tr -d '\n')
$ printf 'site\tobserved\texpected\n'
$ for m in GAATTC GGATCC AAGCTT CTGCAG GATC; do
> obs=$(echo "$seq" | grep -o "$m" | wc -l)
> awk -v m=$m -v o=$obs -v L=$L 'BEGIN {k=length(m); printf "%s\t%d\t%.2f\n", m, o, (L-k+1)/4^k}'
> done
site observed expected
GAATTC 0 1.31
GGATCC 0 1.31
AAGCTT 0 1.31
CTGCAG 0 1.31
GATC 0 21.03
Each of the six-base sites (EcoRI, BamHI, HindIII, PstI) is absent, and absence is unremarkable when about one copy was expected: the chance of none is e−1.31, about 27 %. GATC is different. About 21 copies are expected and there are none. Using the real base composition instead of equal frequencies (the next command) the expectation is 20.2 and the chance of none is about 2 in a billion. A gap like that is selection, not luck. GATC is the site that the E. coli Dam methylase marks, and a plausible reading is that a phage which depends on its host's repair and restriction systems gains by avoiding the sites they recognise. That is a hypothesis the count suggests, not a result it proves. The method is the lesson: an observed count means nothing until you know the expected one.
$ grep -v '>' data/phix174.fa | tr -d '\n' | fold -w1 | sort | uniq -c | awk -v L=5386 '{f[$2]=$1/L} END {p=f["G"]*f["A"]*f["T"]*f["C"]; e=(L-3)*p; printf "P(GATC at a position) = %.5f; expected copies %.1f; chance of none at all = %.1e\n", p, e, exp(-e)}'
P(GATC at a position) = 0.00375; expected copies 20.2; chance of none at all = 1.7e-09
Finally, a motif can be found instead of searched. Take the 12 bases in front of each predicted start codon, align them by the start, and count the bases in each column. faidx and getfasta cut the windows out (the coordinate details come in Lab 5); an awk counter builds the count matrix; and the most common base in each column is the consensus.
$ faidx data/phix174.fa
$ cut -f1,2 data/phix174.fa.fai > results/genome.sizes
$ grep -v '^#' data/genes.gff | awk 'BEGIN{OFS="\t"} $4>13 {split($9,a,";"); sub("ID=","",a[1]); print $1, $4-13, $4-1, a[1], 0, $7}' > results/rbs_windows.bed
$ cat results/rbs_windows.bed | cut -f2-4
38 50 1_1
377 389 1_2
835 847 1_3
988 1000 1_4
2382 2394 1_5
2918 2930 1_6
3968 3980 1_7
$ getfasta -fi data/phix174.fa -bed results/rbs_windows.bed -nameOnly -tab | tee results/rbs_windows.tsv
1_1 CGGATATTTCTG
1_2 AGGTAAGAAATC
1_3 AGGAGTGATGTA
1_4 TGAGGATAAATT
1_5 AGGAGTTTAATC
1_6 GTGAGGTGATTT
1_7 GGAGGCTTTTTT
$ cut -f2 results/rbs_windows.tsv | awk '{for (i=1; i<=length($0); i++) c[i, substr($0,i,1)]++; n=NR} END {split("A C G T", b, " "); for (j=1; j<=4; j++) {printf "%s", b[j]; for (i=1; i<=12; i++) printf "\t%d", c[i, b[j]]; printf "\n"}}'
A 3 0 2 4 1 3 0 3 4 3 0 1
C 1 0 0 0 0 1 0 0 0 1 0 2
G 2 6 5 2 5 1 2 1 0 1 0 1
T 1 1 0 1 1 2 5 3 3 2 7 3
The columns are the 12 positions and the rows are A, C, G and T. The first five columns lean to A and G, which is the AGGAG signal, and the eleventh column is T in all seven windows. The consensus read from the largest count per column is AGGAGATAAATT. Treat it as a sketch only: seven short windows are too few, and because the gap between the motif and the start codon varies, aligning at the start smears the motif. Real motif finders, such as MEME, search over the offset (Hendrix, Ch. 2, de novo motif finding). The shell is for the first look.
+ is a pair of steps: makeblastdb turns a FASTA file into a searchable database, and (or , ) searches it with queries (Camacho et al. 2009). The tabular output, -outfmt 6, is the one to use in a pipeline: one hit per line, twelve tab-separated columns, no header.
$ makeblastdb -in data/phix174.fa -dbtype nucl -out results/phix -title phix174 > results/makeblastdb.log $ blastdbcmd -db results/phix -info | head -3 Database: phix174 1 sequences; 5,386 total bases $ -query data/queries.fa -db results/phix -evalue 1e-5 -outfmt 6 | tee results/hits.tsv q1_from_301 gi|9626372|ref|NC_001422.1| 97.333 150 4 0 1 150 301 450 6.59e-72 255 q2_from_1501 gi|9626372|ref|NC_001422.1| 97.333 150 4 0 1 150 1501 1650 6.59e-72 255 q3_from_2801 gi|9626372|ref|NC_001422.1| 98.000 150 3 0 1 150 2801 2950 1.42e-73 261 q4_from_3901 gi|9626372|ref|NC_001422.1| 97.333 150 4 0 1 150 3901 4050 6.59e-72 255 q5_from_4801 gi|9626372|ref|NC_001422.1| 98.000 150 3 0 1 150 4801 4950 1.42e-73 261
The twelve columns of the default format are the query and subject names, percent identity, alignment length, mismatches, gap openings, the start and end on the query, the start and end on the subject, the E-value and the bit score. All five phiX pieces are found at their true places (query q1_from_301 hits subject positions 301 to 450), with 97–98 % identity, as designed. You can ask for other columns, including the query coverage that the default omits, with a quoted list after the number: -outfmt '6 qseqid sseqid pident length qcovs ...'. The command below adds a header with awk:
$ -query data/queries.fa -db results/phix -evalue 1e-5 -outfmt '6 qseqid sseqid pident length qcovs sstart send evalue bitscore' | sed 's/gi|9626372|ref|//; s/|\t/\t/' | awk 'BEGIN {OFS="\t"; print "query", "subject", "identity", "length", "qcov", "sstart", "send", "evalue", "bits"} {print}'
query subject identity length qcov sstart send evalue bits
q1_from_301 NC_001422.1 97.333 150 100 301 450 6.59e-72 255
q2_from_1501 NC_001422.1 97.333 150 100 1501 1650 6.59e-72 255
q3_from_2801 NC_001422.1 98.000 150 100 2801 2950 1.42e-73 261
q4_from_3901 NC_001422.1 97.333 150 100 3901 4050 6.59e-72 255
q5_from_4801 NC_001422.1 98.000 150 100 4801 4950 1.42e-73 261
Filtering is the usual next step, and it is plain awk on the columns: keep a hit if identity is at least 95 and the aligned length at least 140. A best hit per query table sorts by query and by bit score (descending) and then keeps the first line of each query, using the idiom awk '!seen[$1]++', which prints a line only the first time its first field is seen. Last, comm lists the queries that had no hit at all, which is often the question that matters; the random decoy is the only one.
$ awk -F'\t' '$3 >= 95 && $4 >= 140' results/hits.tsv | cut -f1,3,4,7-9 | head q1_from_301 97.333 150 1 150 301 q2_from_1501 97.333 150 1 150 1501 q3_from_2801 98.000 150 1 150 2801 q4_from_3901 97.333 150 1 150 3901 q5_from_4801 98.000 150 1 150 4801 $ sort -k1,1 -k12,12gr results/hits.tsv | awk '!seen[$1]++' | cut -f1,3,4,9,10,12 q1_from_301 97.333 150 301 450 255 q2_from_1501 97.333 150 1501 1650 255 q3_from_2801 98.000 150 2801 2950 261 q4_from_3901 97.333 150 3901 4050 255 q5_from_4801 98.000 150 4801 4950 261 $ comm -23 <(grep '>' data/queries.fa | tr -d '>' | sort) <(cut -f1 results/hits.tsv | sort -u) q6_decoy
An annotation file says where things are on a genome. GFF3 has nine tab-separated columns: sequence name, source, feature type, start, end, score, strand, phase, and a column of key=value attributes separated by semicolons. Its coordinates are 1-based and inclusive: a feature from 51 to 221 covers bases 51, 52, …, 221, and its length is end − start + 1. BED has at least three columns (sequence, start, end), with optional name, score and strand, and its coordinates are 0-based and half-open: the same feature is written 50 and 221, and its length is end − start. Confusing the two shifts every feature by one base, and nothing warns you.
$ grep -v '^#' data/genes.gff | head -3 | cut -f1-8 | sed 's/gi|9626372|ref|NC_001422.1|/NC_001422.1/'
NC_001422.1 _v2.6.3 CDS 51 221 3.3 + 0
NC_001422.1 _v2.6.3 CDS 390 848 54.5 + 0
NC_001422.1 _v2.6.3 CDS 848 964 15.7 + 0
$ grep -v '^#' data/genes.gff | cut -f3,7 | sort | uniq -c
7 CDS +
predicted seven genes, all on the forward strand. The attribute column holds the identifier; awk splits it on the semicolon and strips ID=. A table of start, end, length and codons gives a first quality check: the length of a protein-coding gene must be a multiple of three. All seven pass.
$ grep -v '^#' data/genes.gff | awk -F'\t' 'BEGIN {OFS="\t"; print "gene", "start", "end", "strand", "length", "codons", "multiple_of_3"} {split($9,a,";"); sub("ID=","",a[1]); L=$5-$4+1; print a[1], $4, $5, $7, L, L/3, (L%3==0 ? "yes" : "no")}'
gene start end strand length codons multiple_of_3
1_1 51 221 + 171 57 yes
1_2 390 848 + 459 153 yes
1_3 848 964 + 117 39 yes
1_4 1001 2284 + 1284 428 yes
1_5 2395 2922 + 528 176 yes
1_6 2931 3917 + 987 329 yes
1_7 3981 5384 + 1404 468 yes
To convert to BED, subtract one from the start and keep the end. Then getfasta cuts each region out of the genome (-s takes the strand into account and reverse-complements minus-strand features, -nameOnly uses just the name). A good test of a gene table is to look at the first and last codon of every gene:
$ grep -v '^#' data/genes.gff | awk 'BEGIN {OFS="\t"} {split($9,a,";"); sub("ID=","",a[1]); print $1, $4-1, $5, a[1], $6, $7}' > results/genes.bed
$ cut -f2-4,6 results/genes.bed
50 221 1_1 +
389 848 1_2 +
847 964 1_3 +
1000 2284 1_4 +
2394 2922 1_5 +
2930 3917 1_6 +
3980 5384 1_7 +
$ getfasta -fi data/phix174.fa -bed results/genes.bed -s -nameOnly -fo results/genes.fa
$ fx2tab -n -l results/genes.fa | head -3
$ echo 'first codons:'; seq -s results/genes.fa | cut -c1-3 | sort | uniq -c
$ echo 'last codons:'; seq -s results/genes.fa | rev | cut -c1-3 | rev | sort | uniq -c
1_1(+) 171
1_2(+) 459
1_3(+) 117
first codons:
7 ATG
last codons:
1 CTG
3 TAA
3 TGA
All seven genes start with ATG, as the predictor requires. At the other end three finish with TAA and three with TGA, which are stop codons, and one finishes with CTG, which is not. The last gene, 3,981 to 5,384, runs to the end of the sequence: marked it partial=01, an incomplete gene: it found no stop codon before the sequence ends. The record (chapter P9 downloads it) shows why: this is gene A, which starts at base 3,981 and, because the genome is circular, continues across the origin, so its stop codon lies at the start of the sequence. Checking the last codon found what a glance at the table would not.
Now the trap itself. If the GFF start is used as the BED start without subtracting one, every sequence begins one base late. The first codons of the same seven genes now read TGA, TGG or TGT instead of ATG, which would make any later translation wrong:
$ grep -v '^#' data/genes.gff | awk 'BEGIN {OFS="\t"} {split($9,a,";"); sub("ID=","",a[1]); print $1, $4, $5, a[1], $6, $7}' > results/genes_wrong.bed
$ getfasta -fi data/phix174.fa -bed results/genes_wrong.bed -s -nameOnly -tab | cut -f2 | cut -c1-3 | sort | uniq -c
2 TGA
1 TGG
4 TGT
Two more questions that a table of genes answers directly. complement lists the regions between the genes, given the genome size file made from the FASTA index, and a short awk adds up the coding bases. The predicted genes cover 91.9 % of the genome, which is typical of a compact viral genome. Be careful with the number all the same: phiX174 is known to carry about eleven genes, several of them overlapping, which was one of the surprises of its sequence (Sanger et al. 1977). A gene finder reports one gene per region and misses genes that lie inside others, so the seven predicted genes are a floor, not the gene count. The record lists eleven (chapter P9).
$ complement -i results/genes.bed -g results/genome.sizes | awk '{print $2+1 "-" $3 "\t" $3-$2 " bp"}'
$ awk '{c+=$3-$2} END {printf "predicted coding: %d of 5386 bases (%.1f%%)\n", c, 100*c/5386}' results/genes.bed
1-50 50 bp
222-389 168 bp
965-1000 36 bp
2285-2394 110 bp
2923-2930 8 bp
3918-3980 63 bp
5385-5386 2 bp
predicted coding: 4950 of 5386 bases (91.9%)
Mapping places each read on the reference. -ax sr maps short reads and writes SAM text, which sort turns into a sorted BAM; index makes it searchable. flagstat is the first sanity check of any mapping: how many reads, how many mapped (Li 2018; Danecek et al. 2021).
$ -ax sr data/phix174.fa data/S1.fq 2> results/S1..log | sort -o results/S1.bam $ index results/S1.bam $ flagstat results/S1.bam 6000 + 0 in total (QC-passed reads + QC-failed reads) 6000 + 0 primary 0 + 0 secondary 0 + 0 supplementary 0 + 0 duplicates 0 + 0 primary duplicates 6000 + 0 mapped (100.00% : N/A) 6000 + 0 primary mapped (100.00% : N/A) …
All 6,000 reads map, as they should: they came from this genome. A SAM record is a line of eleven mandatory columns: read name, FLAG, reference name, position (1-based), mapping quality, CIGAR, then the mate fields, sequence and quality. The CIGAR 100M means 100 aligned bases.
$ view results/S1.bam | head -2 | cut -f1-9 | cut -c1-110 S1_read02138 0 gi|9626372|ref|NC_001422.1| 1 60 100M * 0 0 S1_read00710 16 gi|9626372|ref|NC_001422.1| 1 60 100M * 0 0
The FLAG is a set of bits added into one number: 1 paired, 2 proper pair, 4 read unmapped, 8 mate unmapped, 16 read on the reverse strand, 32 mate on the reverse strand, 64 first in pair, 128 second in pair, 256 secondary alignment, 512 failed quality control, 1,024 duplicate, 2,048 supplementary. Single-end reads on the forward strand have flag 0 and on the reverse strand 16, so counting flags gives the strand balance. In gawk, and($2, 16) tests a bit; view -f 16 and -F 4 do the same filtering without you decoding anything.
$ view results/S1.bam | cut -f2 | sort | uniq -c | sort -k1,1nr
3052 16
2948 0
$ view results/S1.bam | gawk '{rev += and($2, 16) ? 1 : 0; fwd += and($2, 16) ? 0 : 1} END {print fwd, "reads on the forward strand;", rev, "on the reverse strand"}'
2948 reads on the forward strand; 3052 on the reverse strand
Mapping quality is the second thing to look at. 60 is the best value gives; here 5,995 of 6,000 reads have it, and five reads have lower values (they come from places where the genome is locally repetitive or where errors cluster). Depth is the third. depth -a prints one line per base, -a including bases with no reads. The average is 111.4, and 11 of the 5,386 bases are below 10 reads: the first four and the last seven, at the two ends of the linear sequence, where few reads can start or end.
$ view results/S1.bam | cut -f5 | sort -n | uniq -c | sort -k1,1nr | head -4
5995 60
2 50
1 34
1 37
$ depth -a results/S1.bam | awk '{s+=$3; if ($3<10) low++; if ($3>m) m=$3} END {printf "mean depth %.1f; maximum %d; positions below 10x: %d of %d\n", s/NR, m, low, NR}'
mean depth 111.4; maximum 152; positions below 10x: 11 of 5386
Finally, depth per gene, using the BED file from Lab 5. bedcov adds up the depth over each interval, and dividing by the length gives the mean depth of the gene. An even depth across genes is what a random library gives; a gene with a much lower depth would suggest a deletion, a repeat that cannot be mapped, or a gene that is absent from this sample. Here all seven genes are between 96 and 120.
$ bedcov results/genes.bed results/S1.bam | awk 'BEGIN {OFS="\t"; print "gene", "length", "mean_depth"} {len=$3-$2; print $4, len, sprintf("%.1f", $7/len)}'
gene length mean_depth
1_1 171 106.3
1_2 459 119.8
1_3 117 95.7
1_4 1284 115.7
1_5 528 112.0
1_6 987 111.9
1_7 1404 110.1
A variant is a difference between a sample and the reference. mpileup collects, for every position, the bases that the reads show, and call decides which positions differ; the result is a VCF file (Danecek et al. 2011). For a haploid organism such as a phage or a bacterium use --ploidy 1; -m is the multiallelic caller and -v keeps only variant sites. We map both samples and then call the variants in S2, the one that carries the planted changes.
$ for s in S1 S2; do -ax sr data/phix174.fa data/$s.fq 2> results/$s..log | sort -o results/$s.bam; index results/$s.bam; done $ ls results/*.bam results/S1.bam results/S2.bam $ mpileup -f data/phix174.fa results/S2.bam 2> results/S2.mpileup.log | call --ploidy 1 -mv -Ov -o results/S2.vcf 2> results/S2.call.log $ grep -c '^##' results/S2.vcf; grep -v '^##' results/S2.vcf | cut -f1-8 | cut -c1-150 | sed 's/gi|9626372|ref|NC_001422.1|/NC_001422.1/' 28 #CHROM POS ID REF ALT QUAL FILTER INFO NC_001422.1 500 . G A 225.417 . DP=101;VDB=0.343023;SGB=-0.693147;MQSBZ=-0.951662;MQ0F=0;AC=1;AN=1;DP4=0,0,48,53;MQ=59 NC_001422.1 1200 . T C 225.417 . DP=101;VDB=0.976817;SGB=-0.693147;MQSBZ=-0.623579;MQ0F=0;AC=1;AN=1;DP4=0,0,51,50;MQ=59 NC_001422.1 2100 . A T 225.417 . DP=120;VDB=0.137336;SGB=-0.693147;MQSBZ=-0.0118345;MQ0F=0;AC=1;AN=1;DP4=0,0,60,60;MQ=59 NC_001422.1 2600 . CTAAT CT 228.377 . INDEL;IDV=87;IMF=0.887755;DP=98;VDB=0.216449;SGB=-0.693147;RPBZ=-5.38662;MQBZ=0;MQSBZ=0;BQBZ=-1. NC_001422.1 3000 . G T 225.422 . DP=118;VDB=0.861561;SGB=-0.693147;MQSBZ=0.592246;MQ0F=0;AC=1;AN=1;DP4=0,0,58,60;MQ=59 NC_001422.1 3500 . C CGG 228.396 . INDEL;IDV=109;IMF=0.931624;DP=117;VDB=0.0161494;SGB=-0.693147;RPBZ=-3.53227;MQBZ=-0.473288;MQSBZ=0. NC_001422.1 4100 . G A 225.417 . DP=120;VDB=0.45931;SGB=-0.693147;MQSBZ=0.258413;MQ0F=0;AC=1;AN=1;DP4=0,0,54,66;MQ=59 NC_001422.1 5000 . T C 225.417 . DP=128;VDB=0.70269;SGB=-0.693147;MQSBZ=-1.66202;MQ0F=0;AC=1;AN=1;DP4=0,0,74,54;MQ=59
A VCF file starts with ## meta-information lines (28 here: the format, the contigs, the definitions of every INFO field, the commands that made it), then one header line starting with #CHROM, then one line per variant with eight fixed columns: chromosome, position, ID, reference allele, alternate allele, quality, filter and INFO. In INFO, DP is the depth and DP4 the number of reference-forward, reference-reverse, alternate-forward and alternate-reverse reads that support the call. The alternate allele here is supported by every read (DP4=0,0,48,53: none for the reference, 48 and 53 for the alternate), as expected for a haploid sample.
The two rows with INDEL need a word. An insertion or deletion is written with a padding base in front, taken from the reference, because a VCF line has to name a position that exists in the reference. We planted the deletion as CTAA to C; the caller wrote it as CTAAT to CT. Both remove the same three bases, TAA. The same event can be written in more than one way, which is why you compare variant sets after normalising them with norm, or, as here, by position.
$ view -H results/S2.vcf | awk '{print (length($4)==length($5)) ? "SNP" : "indel"}' | sort | uniq -c
6 SNP
2 indel
$ cat data/planted_variants.tsv
pos ref alt
500 G A
1200 T C
2100 A T
3000 G T
4100 G A
5000 T C
2600 CTAA C
3500 C CGG
The test of any variant caller is whether it finds what is there and nothing else. comm compares two sorted lists. All eight planted positions are found, nothing is missed and nothing extra is called. The unchanged sample, S1, gives no variants at all (the count below is 0; note that grep -vc returns exit status 1 when its count is zero, which would stop a script with set -e, a good reason to read the status of a command before you rely on it).
$ awk 'NR>1 {print $1}' data/planted_variants.tsv | sort -n > results/planted.pos
$ query -f '%POS\n' results/S2.vcf | sort -n > results/called.pos
$ echo 'planted and found:'; comm -12 results/planted.pos results/called.pos | tr '\n' ' '; echo
$ echo 'planted, missed:'; comm -23 results/planted.pos results/called.pos | tr '\n' ' '; echo
$ echo 'called, not planted:'; comm -13 results/planted.pos results/called.pos | tr '\n' ' '; echo
planted and found:
500 1200 2100 2600 3000 3500 4100 5000
planted, missed:
called, not planted:
$ mpileup -f data/phix174.fa results/S1.bam 2>/dev/null | call --ploidy 1 -mv -Ov 2>/dev/null | grep -vc '^#'
0
This is a clean case: haploid, 111-fold depth, no repeats and a perfectly simulated error model. Real data are harder, so a call is filtered on its quality and depth before it is trusted. filter -i 'QUAL>=30 && DP>=10' keeps the variants that pass both tests, and query -f prints just the fields you ask for. Choose thresholds from the distribution you see, not from a recipe: here every variant has a quality above 225 and a depth above 98, so any reasonable threshold keeps all eight.
$ query -f '%POS\t%REF\t%ALT\t%QUAL\t%DP\n' results/S2.vcf $ echo '--- QUAL>=30 and DP>=10:' $ filter -i 'QUAL>=30 && DP>=10' results/S2.vcf | query -f '%POS\t%REF\t%ALT\n' | wc -l 500 G A 225.417 101 1200 T C 225.417 101 2100 A T 225.417 120 2600 CTAAT CT 228.377 98 3000 G T 225.422 118 3500 C CGG 228.396 117 4100 G A 225.417 120 5000 T C 225.417 128 --- QUAL>=30 and DP>=10: 8
RNA-seq turns the same machinery into a counting problem: how many reads fall on each gene? coverage -counts counts the reads of a BAM file that overlap each BED interval. We use it on S1 and S2 as a stand-in (these are DNA reads, not RNA, so the numbers are depths, but the commands are the ones used for RNA-seq counting with a gene BED file). Putting two samples side by side is a paste.
$ for s in S1 S2; do coverage -counts -a results/genes.bed -b results/$s.bam | cut -f4,7 | sort -k1,1 > results/$s.counts; done $ paste results/S1.counts results/S2.counts | cut -f1,2,4 1_1 250 257 1_2 657 620 1_3 215 231 1_4 1599 1542 1_5 701 684 1_6 1206 1223 1_7 1599 1691
Raw counts cannot be compared between genes, because a long gene collects more reads than a short one, nor between samples, because one library may be larger. TPM (transcripts per million) corrects for both in two steps: divide each count by the gene length in kilobases to get reads per kilobase (RPK), then divide each RPK by the sum of all RPK values and multiply by one million. TPM values of a sample sum to one million, so a gene's share can be compared between samples. The awk program reads the gene lengths from the BED file in its first pass (NR==FNR, as in P3) and the counts in its second:
$ awk 'BEGIN {OFS="\t"} NR==FNR {len[$4]=$3-$2; next} {r=$2/(len[$1]/1000); id[++n]=$1; rpk[n]=r; tot+=r} END {for (i=1; i<=n; i++) printf "%s\t%.0f\n", id[i], 1e6*rpk[i]/tot}' results/genes.bed results/S1.counts | sort -k2,2nr
1_3 190136
1_1 151271
1_2 148103
1_5 137371
1_4 128853
1_6 126427
1_7 117840
Gene 1_3 is the shortest (117 bases) and has the highest TPM although its raw count is among the lowest: it is dense in reads per base. That is exactly what the correction is for, and why a table of raw counts is never ranked directly. (In real RNA-seq, differential expression uses the raw counts with a count model, not TPM; see chapters 27 and S4.)
The labs so far were interactive. A result you will need again belongs in a script, with the habits of P4: strict mode, checked inputs, a clear failure and a log. The script below is the mapping and variant-calling labs in one file. Three features deserve a name. Checks first: the script tests that the reference exists and that the tools are installed before it starts. Skip what is done: the function step runs a command only when its output file is missing, so a re-run after a crash continues where it stopped, and the cost of fixing the last sample is the last sample. Record the versions: the first line of the log holds the date and the versions of the three programs, which is the cheapest reproducibility measure there is.
#!/usr/bin/env bash
# pipeline.sh: reads -> sorted BAM -> variants -> one summary file per sample (results/SAMPLE.summary.tsv).
# usage: ./pipeline.sh REF.fa SAMPLE [SAMPLE ...] (expects data/SAMPLE.fq; writes into results/)
set -euo pipefail
ref=${1:?usage: $0 REF.fa SAMPLE [SAMPLE ...]}; shift
[[ -s $ref ]] || { echo "reference not found: $ref" >&2; exit 1; }
for tool in ; do
command -v "$tool" >/dev/null || { echo "missing tool: $tool" >&2; exit 1; }
done
mkdir -p results
log=results/pipeline.log
{ echo "# $(date +%F) $( --version) $( --version | head -1 | cut -d' ' -f2) $( --version | head -1 | cut -d' ' -f2)"; } > "$log"
# a step runs only if its output is missing: a re-run continues where the last one stopped
step() { local out=$1; shift; if [[ -s $out ]]; then echo "skip $out" >&2; else echo "make $out" >&2; "$@"; fi; }
map() { -ax sr "$ref" "data/$1.fq" 2>>"$log" | sort -o "results/$1.bam" -; index "results/$1.bam"; }
call() { mpileup -f "$ref" "results/$1.bam" 2>>"$log" | call --ploidy 1 -mv -Ov -o "results/$1.vcf" 2>>"$log"; }
for s in "$@"; do
[[ -s data/$s.fq ]] || { echo "no reads for $s" >&2; exit 1; }
step "results/$s.bam" map "$s"
step "results/$s.vcf" call "$s"
reads=$( view -c results/$s.bam)
mapped=$( view -c -F 4 results/$s.bam)
depth=$( depth -a results/$s.bam | awk '{t+=$3} END {printf "%.1f", t/NR}')
snps=$( view -H -v snps "results/$s.vcf" | wc -l)
indels=$( view -H -v indels "results/$s.vcf" | wc -l)
printf '%s\t%d\t%.2f\t%s\t%d\t%d\n' "$s" "$reads" "$(echo "100*$mapped/$reads" | bc -l)" "$depth" "$snps" "$indels" > "results/$s.summary.tsv"
done
echo "done: $*" >&2
Run on both samples, the script prints what it makes. The per-sample summary files are put together into one table:
$ chmod +x pipeline.sh
$ ./pipeline.sh data/phix174.fa S1 S2 2>&1
make results/S1.bam
make results/S1.vcf
make results/S2.bam
make results/S2.vcf
done: S1 S2
$ { printf 'sample\treads\tmapped_pct\tmean_depth\tsnps\tindels\n'; cat results/S1.summary.tsv results/S2.summary.tsv; } | column -t -s $'\t'
sample reads mapped_pct mean_depth snps indels
S1 6000 100.00 111.4 0 0
S2 6000 100.00 111.4 6 2
Run again, every step is skipped, because the outputs exist. The first line of the log records the versions. And the script fails loudly, with a non-zero exit status that a scheduler or a workflow engine can read, when the reads or the reference are missing:
$ ./pipeline.sh data/phix174.fa S1 S2 2>&1 skip results/S1.bam skip results/S1.vcf skip results/S2.bam skip results/S2.vcf done: S1 S2 $ head -1 results/pipeline.log | cut -c1-110 # 2026-11-22 2.28-r1209 $ ./pipeline.sh data/phix174.fa S9; echo "exit=$?" exit=1 $ ./pipeline.sh missing.fa S1; echo "exit=$?" exit=1
To use several cores, hand the sample names to xargs -P 2, which runs two copies of the script at once. The result is the same, and each job wrote only its own files. That is a rule worth stating: parallel jobs must never write to the same file. The first version of this script wrote one shared summary.tsv; run in parallel, the jobs overwrote each other and only the last sample survived. Writing one file per sample and merging them afterwards in a single serial step fixed it. The shared log file has the same weakness, and a production script gives each job its own log.
$ rm -f results/S[12].bam* results/S[12].vcf results/*.summary.tsv
$ nproc | awk '{print $1 " cores available"}'
$ printf 'S1\nS2\n' | xargs -P 2 -I{} bash -c './pipeline.sh data/phix174.fa {} >/dev/null 2>&1 && echo "finished {}"'
$ cat results/S1.summary.tsv results/S2.summary.tsv
12 cores available
finished S1
finished S2
S1 6000 100.00 111.4 0 0
S2 6000 100.00 111.4 6 2
When a pipeline grows past a script, move to a workflow manager ( or ; chapter 22): they keep the idea of skipping finished work, add parallelism and cluster submission, and make the whole run restartable. The step to take first is the one you have just seen, written by hand.
chmod a-w data/*), write results somewhere else, and store a checksum (md5sum data/* > data/MD5.txt, checked later with md5sum -c) so a silent corruption is caught.join, comm and need sorted input, and the sort order depends on the locale; set LC_ALL=C for a fixed order, and sort -k1,1 -k2,2n for a table keyed on a name and a number.gzip and bgzip files can be streamed with zcat; a BGZF file indexed with can be read at any region without unpacking it (Buffalo, Ch. 13).head trap above), and read the exit status of every command that matters.grep 'ATG$' finds nothing. What do you suspect, and how do you fix it?Windows line endings: every line ends with a carriage return that is part of the text. file confirms it, and tr -d '\r' removes it.
The character is the ASCII value 30 + 33 = 63, ?. The error probability is 10−30/10 = 0.001.
The motif can also lie on the other strand, where it appears as its reverse complement, CCTCC. Searching only the given strand misses those (here six more).
BED is 0-based and half-open: 50 and 221. The length is 221 − 50 = 171 (or 221 − 51 + 1 in GFF terms).
It is mapped and lies on the reverse strand (bit 16). A flag of 4 would mean unmapped.
Not necessarily: overlapping genes, which phiX174 has, are reported once per region. The prediction is a floor; compare with a curated annotation when one exists.
summary.tsv. What can go wrong, and what is the fix?They overwrite or interleave each other's lines, so results are lost or garbled. Let each job write its own file and merge them afterwards in one serial step.
Long genes collect more reads, and libraries differ in size. TPM divides by gene length and by the total of the length-corrected counts, so values are comparable between genes and between samples.
An amplicon study turns millions of reads into a table of counts: rows are sequence features, columns are samples. How those features are defined decides the resolution of everything that follows.
The 16S rRNA gene has nine hypervariable regions (V1 to V9) separated by conserved stretches where universal primers bind. Short-read studies sequence one or two regions, commonly V4 or V3–V4. No region separates all taxa equally well: V4 is robust and widely used but resolves few species, and primers differ in which groups they miss. Full-length 16S from long reads resolves many more taxa to species and sometimes strain level (Johnson et al., 2019). For fungi the ITS1 or ITS2 region is standard; its length varies a lot between taxa, which affects both PCR and read merging.
Early pipelines clustered reads into operational taxonomic units (OTUs) at 97 percent identity to absorb sequencing errors. Clustering merges real biological variants and gives results that depend on the dataset. Denoising methods instead learn the error profile of a run and infer the exact biological sequences present, called amplicon sequence variants (ASVs). DADA2 fits a parametric error model to each run and tests whether each unique sequence can be explained as an error from a more abundant one (Callahan et al., 2016). ASVs are reproducible across studies because they are sequences, not cluster labels, and are now the standard unit (Callahan, McMurdie & Holmes, 2017).
Xia & Sun devote three chapters to the shift from clustering to denoising: how OTU clustering grew out of numerical taxonomy, why a fixed 97 % threshold merges real variants and splits error clouds unevenly, and how DADA2 and Deblur instead model the error process so that every exact sequence variant is either real or an error (Xia & Sun, Bioinformatic and Statistical Analysis of Microbiome Data, ch. 6–8). Mitra's volume compares the main pipelines side by side (Mitra (ed.), Metagenomic Data Analysis, ch. 4).
Each ASV is assigned a lineage by comparison with a reference database, typically SILVA ( et al., 2013) or -derived 16S sets for bacteria and archaea, and for fungi. The naïve Bayesian classifier counts shared k-mers between the query and reference sequences and reports a bootstrap confidence at each rank (Wang et al., 2007). Classifiers trained on the exact amplified region perform better than those trained on full-length genes (Bokulich et al., 2018). Species-level calls from short amplicons should be treated with caution; genus is usually the finest reliable rank.
A phylogenetic tree of the ASVs is needed for phylogeny-aware measures such as Faith's phylogenetic diversity and UniFrac. Trees are usually built by aligning ASVs with and inferring a tree with , or by inserting ASVs into a reference tree.
One OTU at 97 % identity, but two ASVs if the denoiser judges the difference to be real rather than a sequencing error.
A 250 bp region of a conserved gene often has identical sequences across close species; genus is usually the reliable level.
Shotgun sequencing reads every genome in the sample, not one gene. Profilers turn those reads into species abundances in two different ways, and the difference matters when you interpret the numbers.
2 assigns each read to the lowest common ancestor of all reference genomes that share its k-mers (Wood, Lu & Langmead, 2019). It is fast and classifies reads from any domain in the database, including host, fungi and viruses. Because many reads stop at genus or family when species share sequence, Bracken redistributes them to species using the expected read distribution of each genome (Lu et al., 2017). The output is a read share: the fraction of sequenced reads that came from each taxon. Larger genomes contribute more reads per cell, so read share is not the same as cell share.
Zhong Wang explains the algorithms behind both families, from k-mer matching and lowest common ancestor assignment to marker-gene and binning approaches (Zhong Wang, Introduction to Computational Metagenomics, ch. 6).
MetaPhlAn maps reads only to a curated set of clade-specific marker genes, about a million of them in MetaPhlAn 4, and estimates abundances from marker coverage (Blanco-Míguez et al., 2023). Because coverage is normalised by marker length, the result approximates the relative abundance of genomes, closer to a cell share. MetaPhlAn 4 defines species-level genome bins (SGBs) from large collections of metagenome-assembled genomes, so it can profile species that have never been cultured (Pasolli et al., 2019). Reads from organisms without markers, and from host or unknown sources, are simply not counted, so profiles sum to 100 percent of the known fraction.
| Property | + Bracken | MetaPhlAn 4 |
|---|---|---|
| What is counted | Reads, assigned by k-mers | Reads on clade-specific marker genes |
| Units | Read fraction | Genome (cell-like) fraction |
| Unknown organisms | Unclassified or misassigned to relatives | Covered when an SGB exists, otherwise ignored |
| Main risk | False positives from database gaps; needs confidence filtering | Misses taxa without markers; low-abundance detection threshold |
Both approaches see only what is in their database. A taxon missing from the reference set is either invisible or attributed to its nearest relative. When comparing studies, use the same tool and database version; when reporting, give both. For environments far from the human gut, such as soils or fermented foods, database completeness is often the main limit, and assembly-based approaches (binning metagenome-assembled genomes) recover organisms that profilers cannot see (Quince et al., 2017).
Running a mock community through the same pipeline shows whether expected species are recovered, whether unexpected ones appear (false positives) and how far abundances drift from the truth. Recovering all expected species with few false positives is a better test than matching the exact proportions, which extraction bias distorts anyway.
Possibly neither: counts reads (biased to large genomes and database content), MetaPhlAn counts marker-gene abundance. Check the database and use a mock community.
Its reads are assigned to the closest relative that is in the database, which then appears present.
A raw feature table contains real taxa, off-target sequences, contaminants, artefacts and samples that failed. Filtering removes what cannot be biology, but every filter is a decision that must be reported.
16S primers also amplify chloroplast and mitochondrial rRNA, which are bacterial in origin. In plant, food or host-tissue samples these can dominate. Features assigned to Chloroplast (order) or Mitochondria (family), and features unassigned at kingdom or phylum level, are normally removed before analysis.
Most features in a microbiome table are rare: seen in one or two samples at a few reads. They add noise, inflate the number of tests in differential abundance and destabilise network inference. Common filters keep features present in at least a minimum number or fraction of samples (for example 5 to 10 percent) and above a minimum mean relative abundance. These filters should be applied without looking at the group labels, otherwise they bias downstream tests. Alpha diversity is the exception: richness estimators need the rare features, so compute diversity on the unfiltered table.
Filtering is a modelling decision, not housekeeping. Xia, Sun & Chen show how prevalence filters reduce the number of tests and the share of zeros, which improves power, but also how a filter applied separately to each group creates differences that are not there (Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R, ch. 7). Apply one filter to all samples, decide it before testing, and report it.
Samples with very low read counts after processing are usually failed libraries. Plot depth per sample and per group: if one group systematically has lower depth, depth becomes a confounder. Ordinations of all samples, including controls, also reveal outliers and mislabelled samples. Controls should cluster away from real samples.
Contaminant DNA comes from reagents, kits and the lab, and it behaves in two recognisable ways (Davis et al., 2018):
The decontam method tests both patterns per feature and flags features with a score below a threshold. Flagged features should be inspected, not deleted blindly: common gut taxa can appear in blanks through well-to-well cross-contamination, and removing them from real samples would be wrong. Known reagent contaminant genera, such as Ralstonia, Bradyrhizobium and Sphingomonas, deserve suspicion in low-biomass samples (Salter et al., 2014).
Choosing the filter that gives the most significant result is a hidden form of multiple testing and inflates false discoveries.
Contaminant DNA is roughly constant per extraction, so its share is largest in samples with little real DNA.
Sequencing counts carry information only about ratios between taxa. Normalisation corrects for unequal sequencing depth; transformation puts the data on a scale where standard statistics behave. They solve different problems and are best chosen separately.
The total number of reads per sample is set by the sequencing run, not by the biology. Counts are therefore a random sample of a composition, and only ratios between parts are meaningful (Gloor et al., 2017). Two consequences follow. First, correlations between relative abundances are biased: when one taxon increases, the others must decrease, creating negative correlations that do not exist in the community. Second, a taxon can appear to increase only because others decreased. Aitchison's log-ratio framework (Aitchison, 1986) is the principled way around this.
The full theory, from Aitchison's simplex to CLR, ILR and ALR transformations and their use in tests, is in Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R, ch. 10, and Xia & Sun, Bioinformatic and Statistical Analysis of Microbiome Data, ch. 14.
Counts are skewed and their variance grows with the mean, so most methods based on Euclidean distance or linear models need a transformation first.
It falls, although B did not change. This is the compositional effect: one change moves every proportion.
The log of zero is undefined; a pseudocount or a model-based replacement is required, and the choice affects rare taxa most.
Alpha diversity summarises a single community in one number: how many kinds of organisms it holds and how evenly they are spread. It is the most reported microbiome statistic and one of the easiest to get wrong.
Richness counts the taxa present. Shannon entropy and the Simpson index also weight taxa by their abundance, so they reflect evenness as well. These indices sit on different scales, which makes them hard to compare. Hill numbers put them on one scale, the effective number of taxa (Hill, 1973):
qD = ( Σ piq )1/(1−q)
With q = 0 this is richness, q = 1 (as a limit) is the exponential of Shannon entropy, and q = 2 is the inverse Simpson index. Plotting D across q gives a diversity profile: a community with many rare taxa has high D at q = 0 that falls steeply as q rises. Faith's phylogenetic diversity sums the branch lengths of the tree spanned by the taxa present and rewards communities that cover distant lineages (Faith, 1992).
Observed richness grows with sequencing depth, so samples sequenced more deeply look more diverse. There are three common ways to compare at equal effort. Rarefying once to the smallest depth is simple but adds noise and discards data. Repeated rarefaction averages over many subsamples and reduces that noise (Schloss, 2024). Coverage-based standardisation compares samples at equal sample completeness rather than equal read count, which is fairer when communities differ in how many rare taxa they hold (Chao & Jost, 2012). Richness is the most sensitive to effort; Shannon and Simpson much less (Willis, 2019).
The missing-species problem has a famous history. Fisher was asked how many new butterfly species a collector would find in a second year of trapping in Malaya; his answer, refined by Good and Toulmin, is the alternating series above. Efron & Hastie use it to open their discussion of empirical Bayes (Efron & Hastie, Computer Age Statistical Inference, section 6.2). MicroRes reports these quantities for every sample in step 06 ("How many taxa are still unseen?"), together with the sample size a follow-up study needs (chapter 2).
Diversity values are one number per sample and can be compared with ordinary tests: Wilcoxon or Kruskal–Wallis for groups, linear models when there are covariates, and mixed models when subjects are measured repeatedly. Report effect sizes, not only p-values: a statistically significant difference of two taxa in a richness of 300 is rarely meaningful.
It depends on the order q: richness (q = 0) favours A, Shannon or Simpson (q = 1, 2) likely favour B. Report more than one.
Many species seen once means many more are unseen. Chao1 and Good–Turing reasoning use singletons and doubletons to estimate the missing species.
Beta diversity asks how different two communities are. A distance between every pair of samples, an ordination to look at it, and a permutation test to judge it form the backbone of most microbiome papers.
The distance encodes what you consider "different":
Different distances can lead to different conclusions, and that is information. A treatment that shifts unweighted UniFrac but not Bray–Curtis changes which rare lineages are present without changing the dominant taxa.
Ordination is unsupervised learning: PCA, PCoA and NMDS all try to show high-dimensional distances in two dimensions, and the share of variance explained by each axis tells you how much they hide (James et al., An Introduction to Statistical Learning, ch. 12; Xia & Sun, Bioinformatic and Statistical Analysis of Microbiome Data, ch. 10).
Principal coordinates analysis (PCoA) places samples in a few dimensions that preserve the chosen distance as well as possible. The share of variation on each axis tells you how much of the structure the plot shows. PCA is PCoA with Euclidean distance and works directly on CLR or Hellinger data. Non-metric multidimensional scaling (NMDS) preserves only the rank order of distances and reports a stress value; stress above about 0.2 means the picture is unreliable. Constrained ordinations (RDA, CCA, distance-based RDA) show only the variation explained by chosen metadata and are useful for asking how much a set of variables explains.
PERMANOVA partitions the distance matrix among groups or covariates and assesses significance by permuting labels (Anderson, 2001). Its R² is the fraction of variation explained. In microbiome studies, single factors often explain 1 to 10 percent, which is still meaningful given how variable communities are. Two cautions apply. First, PERMANOVA is sensitive to differences in spread (dispersion) as well as location, so test dispersion with PERMDISP and report both. Second, repeated measures must be handled by restricting permutations within subject (strata); otherwise the test treats correlated samples as independent. Covariates are added sequentially, so their order matters unless marginal tests are requested.
Groups differ, but possibly only in spread, not in centre. Look at the ordination before claiming a shift in composition.
Jaccard uses presence and absence only, so a rare taxon counts as much as a dominant one; Bray–Curtis weights by abundance.
Which taxa differ between groups is the question most microbiome studies end on, and the one where methods disagree most. Using more than one method and reading the agreement is the honest answer.
Microbiome counts are sparse (most entries are zero), over-dispersed, compositional and measured at unequal depth. A method must decide what "no change" means. Because only ratios are observed, a test on relative data can only say that a taxon changed relative to something: the rest of the community, a reference taxon or an assumed unchanged majority. When many taxa change in the same direction, methods that assume most taxa are unchanged break down.
Microbiome counts are also over-dispersed (the variance far exceeds the mean, unlike the Poisson model) and full of zeros. Faraway's chapter on count regression is the clearest introduction to Poisson, quasi-Poisson and negative binomial models (Faraway, Extending the Linear Model with R, ch. 5); Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R, ch. 11–12, apply them to microbiome data, including zero-inflated and hurdle models. Benchmarks agree that no single method wins everywhere, which is why MicroRes runs several and reports the taxa they agree on.
A comparison of 14 methods on 38 datasets found that they identified very different sets of significant taxa, and that ALDEx2 and ANCOM-II gave the most consistent results across datasets (Nearing et al., 2022). The practical lesson is to run several methods with different assumptions and give most weight to taxa they agree on.
With hundreds of taxa tested, control the false discovery rate with the Benjamini–Hochberg procedure (Benjamini & Hochberg, 1995). Report the fold change and its uncertainty, the prevalence of the taxon in each group, and the method. A taxon present in 3 of 40 samples can be "significant" in some methods and is rarely worth a biological claim.
Poisson forces variance = mean; microbiome counts vary far more between samples (over-dispersion). The negative binomial adds a dispersion parameter.
Not necessarily: they test different hypotheses (relative versus log-ratio change) under different assumptions. Taxa found by several methods are the robust core.
Correlating taxa with each other or with host variables is easy to do and easy to over-interpret. Compositional methods remove the worst artefacts; careful reading removes most of the rest.
Associations between taxa and continuous metadata, such as a metabolite, a clinical score or age, are usually tested with Spearman correlation on a transformed table, or with per-feature linear models that adjust for covariates. Partial correlations remove the shared influence of a third variable. Because every taxon is tested against every variable, multiple-testing correction is essential.
Plain correlations between relative abundances are biased by compositionality (chapter 7). Two methods address this. SparCC estimates correlations between the underlying absolute abundances from log-ratio variances, assuming the network is sparse (Friedman & Alm, 2012). SPIEC-EASI goes further and estimates conditional dependence, through neighbourhood selection or the graphical lasso on CLR data, so that two taxa linked only through a third are not connected directly (Kurtz et al., 2015). SPIEC-EASI networks are sparser and more interpretable, at the cost of more computation and a need for many samples. Several hundred samples are a reasonable target, and networks built from 20 samples are mostly noise.
The problem is old: Karl Pearson warned in 1897 of "spurious correlation" between ratios that share a denominator. Proportions all share the sample total, so two unrelated taxa become negatively correlated when a third one blooms. SparCC and SPIEC-EASI estimate associations between log-ratios and add sparsity, assuming most taxa do not interact directly (Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R, ch. 10). Graph concepts such as degree, hubs, modules and betweenness are explained for biological networks in Baxevanis, Bader & Wishart, Bioinformatics, ch. 13, and the ecology of microbial interactions in Zhong Wang, Introduction to Computational Metagenomics, ch. 10.
Common summaries are node degree, betweenness centrality, modules (clusters of densely connected taxa, for example from the Louvain algorithm (Blondel et al., 2008)) and the balance of positive and negative edges. Highly connected or central taxa are sometimes called keystone taxa, but a statistical hub is not the same as an ecological keystone (Berry & Widder, 2014). An edge means co-variation across samples: it can come from a shared niche, a shared response to diet or a direct interaction, and the network cannot tell these apart (Röttjers & Faust, 2018).
Comparing summary numbers of two networks, such as density or modularity, says little about where they differ. Differential network analysis asks, for every pair of features, whether their association changes between the groups. With correlations this is a test of the difference between two Fisher-transformed coefficients (Fukushima, 2013; McKenzie et al., 2016). The pairs whose association changes form a differential network. Clustering that network gives modules of features that rewire together, for example a group of taxa that co-vary tightly in controls but fall apart after antibiotics. For each module, the change in connectivity can be tested by permuting group labels. That turns a picture into a statistic and highlights the modules that carry the difference. NetCoMi implements related comparisons for microbiome data (Peschel et al., 2021).
Networks can also be built between samples: each sample is linked to its most similar samples, and community detection splits the graph into clusters of samples with similar whole profiles. Cross-tabulating these clusters against a study variable shows whether groups form their own clusters or mix. This is a graph-based complement to ordination and PERMANOVA (chapter 9).
Differential associations need more samples than a single network: each group must support its own correlation estimates. With fewer than about 20 to 30 samples per group, few changes pass a false discovery threshold. Results are then best read as exploratory.
Not necessarily: they may share a habitat preference, respond to the same host factor, or be correlated by the compositional constraint. A network edge is a hypothesis.
With few samples, chance correlations are large; sparsity penalties and stability selection need more samples to separate signal from noise.
Predicting a phenotype from the microbiome tests whether the community carries information about it. Done carelessly, the same exercise produces impressive accuracies that vanish on new data.
Microbiome datasets have many more features than samples. Regularised linear models handle this well: LASSO selects a small set of features, ridge shrinks all of them, and the elastic net mixes the two and copes better with correlated taxa (Zou & Hastie, 2005). Random forests capture non-linear effects and interactions and need little tuning (Breiman, 2001). In a systematic comparison, L2-regularised logistic regression performed about as well as more complex models on microbiome classification while being far easier to interpret (Topçuoğlu et al., 2020).
The model must be evaluated on samples it never saw during training, including during feature selection and tuning. Three rules prevent most leakage:
Report the area under the ROC curve (or balanced accuracy) with its spread across folds, and compare with a model trained on shuffled labels. An external validation cohort is the strongest evidence.
James et al., An Introduction to Statistical Learning explains cross-validation and the bootstrap (ch. 5), regularised regression such as the lasso (ch. 6) and random forests (ch. 8) with the clearest diagrams in the literature; it is freely available from the authors.
Model coefficients and importances show what the model used, not what causes the phenotype. Features that are selected in most folds are more trustworthy than those selected once. Linear discriminant analysis effect size (LEfSe) is a popular biomarker ranking (Segata et al., 2011), but it is a univariate screen, not a predictive model, and should not be reported as one.
Feature selection saw the test samples: the reported accuracy is optimistic. Selection must happen inside each training fold.
With 90 % controls, predicting 'control' always gives 90 % accuracy; AUC measures ranking across all thresholds.
Taxonomy tells you who is there; function tells you what the community can do. Functional profiles add a layer of interpretation, and a layer of assumptions that should be stated.
HUMAnN tables contain UNMAPPED, UNINTEGRATED and UNGROUPED rows that are bookkeeping, not functions; they are usually removed or reported separately. Gene families can be regrouped to KEGG orthologs or enzyme commission numbers for interpretation. Functional tables are compositional just like taxonomic ones, so the same normalisation logic applies (chapter 7). Copies per million (CPM) or relative abundance followed by a log transformation are common choices.
Per-function models with covariates and random effects, in the style of MaAsLin (Mallick et al., 2021), are standard. MaAsLin 3 separates abundance (how much, when present) from prevalence (whether present) and adds a correction for compositionality (Nickols et al., 2024). With thousands of functions, false discovery rate control is essential, and pathway-level results are easier to interpret than gene-family results.
Stratified output links each function to its contributing taxa, so a change in a pathway can be traced to the species behind it. A function carried by many taxa is functionally redundant: losing one carrier changes little. Tian and colleagues define functional redundancy from the gap between taxonomic and functional diversity (Tian et al., 2020). Contributional diversity, the diversity of carriers for one function, distinguishes functions that depend on a single keystone carrier from those spread across the community. Bipartite species–function networks make these patterns visible; their nestedness (NODF) shows whether specialist carriers hold subsets of the functions of generalists (Almeida-Neto et al., 2008).
A functional profile from DNA describes potential; the community may not express it. The step from genes to activity is covered by metatranscriptomics and by function-based screens, in which DNA from the community is cloned and tested for an activity directly, as described by Charles, Liles & Sessitsch in Functional Metagenomics: Tools and Applications. For the computational side, Zhong Wang, Introduction to Computational Metagenomics, ch. 7, covers gene prediction and annotation in metagenomes and ch. 8 the recovery of genomes (binning), which is what makes "who does what" possible beyond reference databases.
Not necessarily: DNA shows potential. Activity needs RNA (metatranscriptomics), proteins or metabolites.
They show whether a change in a pathway comes from one taxon gaining abundance or from a shift across many carriers.
Most of the methods in this handbook were built for bacteria. Viruses break several of their assumptions: there is no gene every virus shares, most sequences have no relative in any database, and a sample can be full of viral reads from sources that are not in it. This chapter explains why, so that the next two chapters make sense.
Viruses are the most numerous biological entities on Earth, and the viruses of bacteria, the bacteriophages or phages, shape bacterial communities by killing cells, moving genes between them and changing their metabolism (Suttle, 2007). In the human gut most of the virome is phage, and most of it has never been cultured (Shkoporov & Hill, 2019). That is why viromes are studied from sequence: there is nothing else to study them from. For a company, phages matter in three ways: as a risk (a phage can carry a toxin or a resistance gene into a production strain), as a tool (phage therapy, biocontrol of pathogens in food) and as a marker of a community's state.
A bacterial genome is double-stranded DNA with a few well-known features. A virus genome can be DNA or RNA, single- or double-stranded, linear or circular, in one piece or several. Baltimore's classification sorts viruses by how their genome becomes messenger RNA, and the practical consequences for a sequencing project are direct:
A virulent (lytic) phage infects a cell, makes new virions and bursts the cell. A temperate phage can do the same, or it can integrate into the host chromosome as a prophage and be copied with the host for generations, until a signal makes it excise and turn lytic. Lysogeny has two consequences for anyone using or monitoring phages. A prophage can carry genes that change the host, such as the Shiga toxin gene carried by phage 933W, which makes its Escherichia coli host a Shiga-toxin-producing strain. And a temperate phage can move genes between hosts, which is the reason regulators of phage therapy look for strictly lytic phages (Philipson et al., 2018). The genes that give a temperate phage away, an integrase or excisionase, are among the first things a safety screen looks for (chapter V3).
Virus taxonomy is set by the International Committee on Taxonomy of Viruses (ICTV) and it changed fundamentally in 2022. For decades the tailed phages were sorted by the shape of the tail into the families Myoviridae, Siphoviridae and Podoviridae, in the order Caudovirales. That sorting was found not to reflect descent, and the ICTV abolished those names and the order, placing the tailed phages in the class Caudoviricetes with new families based on genome relationships (Turner et al., 2023). A lineage now runs from realm (for tailed phages, Duplodnaviria) through kingdom, phylum, class, order and family to genus and species, and a great many viruses can be placed only part of the way down. Old literature uses names that no longer exist: when you read "Siphoviridae", read "a tailed phage".
A virus-like-particle (VLP) virome is prepared to enrich virions: the sample is filtered, treated with nucleases to destroy free DNA, and the DNA that remains inside protein coats is extracted (Thurber et al., 2009; Conceição-Neto et al., 2015). A bulk metagenome is not enriched, so it contains viral sequence only as a small share of the reads, together with the prophages integrated in the bacterial genomes. VLP viromes show free virions, bulk metagenomes show the prophages that the bacteria carry, and the two do not give the same picture of the same sample. estimates how well a virome was enriched, by counting the reads that come from bacterial ribosomal RNA and single-copy genes, which a clean VLP preparation should lack (Zolfo et al., 2019).
The gut virome is dominated by tailed phages that infect the gut bacteria, and it is individual: each person carries a largely different set that changes slowly over time (Shkoporov & Hill, 2019). One virus stands out. crAssphage was found in 2014 by assembling the sequences in human faecal metagenomes that matched nothing known, and it is among the most abundant viruses of the human gut (Dutilh et al., 2014). Large catalogues now collect the genomes recovered from gut metagenomes, the Gut Virome Database and the Gut Phage Database among them, and they are the reference sets against which a new gut virome is compared (Gregory et al., 2020; Camarillo-Guerrero et al., 2021).
Brock, Biology of Microorganisms, ch. 8 on virus replication and ch. 10 on viral genomics, diversity and ecology, is the best general introduction to the biology (Madigan et al.). For the human virome in a metagenomics setting, read Izard and Rivera, ch. 9 (Metagenomics for Microbiology). The bioinformatics of viruses, from comparative genomics to host prediction, is the subject of Frishman and Marz (eds.), ch. 1, 2 and 10 (Virus Bioinformatics).
16S rRNA is present in every bacterium and archaeon. No gene is shared by all viruses, so there is no amplicon to target, and viruses must be found in shotgun data.
It can integrate into a host genome and move genes between hosts, including toxin or resistance genes. Products usually use strictly lytic phages.
A tailed phage. The family was abolished in 2022 because tail shape does not reflect descent; the tailed phages are now in the class Caudoviricetes.
It shows the 40 % that can be named. The unclassified share is the largest part and must be shown with it.
A virome table is the end of a pipeline with five decisions in it: which sequences are viral, how good each genome is, which genomes are one virus, what each one is and infects, and how much of each is in each sample. Each decision leaves a trace in the table, and a reader who knows the pipeline can see what the numbers mean.
A virome starts as reads from many viruses and, usually, from many bacteria as well. The reads are assembled into contigs with a metagenome assembler such as or , or with , which is built for the repeats and uneven coverage of viral genomes (Antipov et al., 2020). Then each contig has to be classified, because an assembly of a virome still contains bacterial chromosome, plasmids and other mobile elements.
Tools identify viruses by combining several weak signals. classifies a sequence as chromosome, plasmid or virus with a neural network on its sequence, supported by a set of marker genes, assigns a taxonomy that follows the ICTV, and finds proviruses inside larger sequences (Camargo et al., 2024). uses several classifiers and expert-curated rules (Guo et al., 2021), uses annotation of protein families (Kieft et al., 2020), and uses a neural network on the sequence alone, without homology (Ren et al., 2020). They disagree on the borderline, so a study should state which was used and at what score. Every tool trades missed viruses against false calls, and a high score cut-off removes the doubtful sequences and the real but unusual ones together.
A contig that is called viral may be a complete genome, a fragment of one, or a chimera of a virus and its host. estimates completeness by comparing each contig with a database of complete viral genomes, using amino-acid identity to place it and the length of its closest relatives to judge how much is missing. It estimates contamination by counting host genes at the ends of the contig, and it trims host sequence from proviruses (Nayfach et al., 2021). Its output follows the MIUViG standard, which sorts genomes into tiers (Roux et al., 2019):
| Tier | Estimated completeness | Use it for |
|---|---|---|
| Complete | 100 %, with evidence such as terminal repeats | genome-level conclusions |
| High-quality | 90 % or more | abundance and most gene-content questions |
| Medium-quality | 50 to 90 % | abundance with care; gene absence cannot be asserted |
| Low-quality | below 50 % | presence of a fragment only |
| Not-determined | no close relative to judge against | often novel viruses; treat as unknown, not as bad |
The same virus is assembled again and again, from different samples and with different ends, and counting each copy would inflate the virome. Genomes are therefore clustered into vOTUs. The convention is at least 95 % average nucleotide identity over at least 85 % of the shorter genome (Roux et al., 2019), which for tailed phages is close to species level. includes scripts that calculate the identity between genomes and cluster them on these thresholds. One representative, usually the longest or most complete, stands for each vOTU, and all later steps use only the representatives. A vOTU table is therefore a table of decisions: change the thresholds and the number of vOTUs changes.
Lineage. 's taxonomy places a sequence by its marker genes in the ICTV system, and gene-sharing networks, as in , cluster viruses by the proteins they share (Bin Jang et al., 2019). Both stop at family or genus for most sequences, as chapter V1 showed.
Host. A phage's host is not in its sequence. It is inferred from what it leaves behind in bacterial genomes: CRISPR spacers that match the phage, integrated prophages, shared tRNAs and genes, and similarity of k-mer composition to the host. combines these methods and gives each prediction a confidence score (Roux et al., 2023). A predicted host is a hypothesis, reported at genus level with a confidence score, and many viruses get no prediction at all, especially those of poorly known hosts. A phage may also infect more than one host.
Lifestyle. predicts whether a phage is virulent or temperate from the conserved protein domains in its genome (Hockenberry & Wilke, 2021). A simpler screen looks for an integrase or excisionase. The absence of an integrase does not make a phage virulent: an incomplete genome may lack the gene.
Abundance is measured by mapping the reads of each sample back to the vOTU genomes, with a mapper such as Bowtie 2 (Langmead & Salzberg, 2012) or (Li, 2018). Two numbers per vOTU and sample matter: the reads that map, and the breadth, the share of the genome that those reads cover. The reads alone are misleading, for two reasons.
A widely used rule therefore counts a virus as present only if reads cover at least 75 % of its genome, and works with genomes of at least 10 kb for abundance (Roux et al., 2017). It is a convention, not a law: it removes false presences and some true low-abundance viruses, and the threshold should be stated with every result. Reads should also be filtered for mapping quality, so that a read which maps equally well to two related viruses is not credited to either.
Mitra (ed.), ch. 17, describes the assembly and annotation of viral metagenomes from short reads step by step (Metagenomic Data Analysis), and Frishman and Marz (eds.), ch. 2, reviews the techniques for exploring phage diversity in metagenomes (Virus Bioinformatics). The wet-laboratory side, from isolating phages to preparing them for sequencing, is in Pantaleo and Chiumenti (eds.), ch. 1 (Viral Metagenomics).
Not by the usual rule: reads cover about 13 % of the genome, far below 75 %. The reads probably come from a relative or a shared gene.
About 9 % and 91 %. Reads are proportional to length times copy number, so abundances must be divided by length before comparison.
No. It means the database held no close relative to judge completeness against. Many novel viruses fall in this tier.
A vOTU is defined by the threshold (95 % identity over 85 % of the shorter genome), so the number of vOTUs and the diversity computed from them change if the threshold changes.
A virome result is only as good as the checks behind it. This chapter follows the order of the MicroRes viral module: decide which genomes and which presences to trust, look for contamination, compare viromes with statistics that respect composition, link viruses to hosts, screen phages for use in a product, and report all of it so that a reviewer can check it.
The first question about a virome is whether the viruses in it came from the sample. Extraction kits and reagents carry traces of nucleic acid, and with the small amounts of DNA in a virome those traces can make up much of the library (Salter et al., 2014). The defence is a blank, a tube taken through the whole procedure without a sample, sequenced beside the samples. A virus that is more prevalent in the blanks than in the samples is a reagent contaminant. The prevalence method of the decontam package tests this with Fisher's exact test per virus (Davis et al., 2018), and the module applies it in V2.
Some contaminants are known in advance. phiX174 is added to Illumina runs as a sequencing control, usually at 1 % and at higher levels for libraries of low diversity, so it appears in a virome as a real virus genome that was never in the sample. It is a sequencing control, not a finding, and a study should name it and remove it. The module reports it in V2 when a genome named phiX174 is present. A second control is the enrichment itself: if a virome contains many reads from bacterial ribosomal RNA genes, the virus-like-particle preparation did not work, and quantifies this (Zolfo et al., 2019).
Chapter V2 explained why reads on a genome are not presence. The practical rule is a breadth threshold, set before the analysis and reported with it. Everything downstream is affected by it: the number of vOTUs seen in a sample (richness), the prevalence of each virus, and the core virome. A study that changes the threshold changes its conclusions, so the module records it with the result.
Once the table is made, the statistical methods of Part IX apply, with the cautions of a compositional table (Gloor et al., 2017).
If bacterial abundances are measured on the same samples, a virus can be compared with its predicted host: does it rise and fall with it? The comparison needs care. Both tables are compositional, so they are put on a centred log-ratio scale before correlation, and the p values are corrected for the number of pairs tested. A positive link supports a host prediction. It does not prove infection, because a virus and a bacterium can both respond to the same condition, and a lytic phage and its host can show the opposite pattern when the phage is killing the host. A link is a reason to look, not a conclusion.
A phage intended for therapy, for biocontrol in food or as a feed additive must be shown not to carry genes that would harm. Philipson and colleagues list what a genome characterisation should look for: that the phage is strictly lytic, that it has no integrase or other lysogeny genes, and that it carries no toxin, virulence or antibiotic-resistance gene (Philipson et al., 2018). The module's screen follows this and returns for each vOTU one of three verdicts: Concern (a strong hit to a toxin, virulence or resistance gene), Review (a weak hit, a temperate lifestyle, an incomplete genome or sequence from the host), and No flag found. Each verdict carries its reasons.
Three limits belong in every report. The screen finds only genes that the databases know. A genome that is 85 % complete can hide a gene in the missing 15 %, which is why the screen sends incomplete genomes to review. And a clean screen is not a clearance: the evidence that a regulator requires for a product is wider than a genome screen.
A virome report has to answer the questions a reviewer asks before they read a result. What did the controls show? Which genomes were included, and which thresholds were set? How many samples failed? What was fixed before the analysis, and what was decided after it? The MIUViG standard defines the information that should accompany a viral genome (Roux et al., 2019), and the MIxS checklists define the metadata of the samples (Yilmaz et al., 2011).
The module's report opens with a quality gate: a short list of checks (was presence tested, are the blanks clean, did the samples pass, do the reads fall on good genomes, are there enough samples, are the genomes annotated, was the safety screen run), each passed, flagged for review or failed under a rule printed beside it. The overall verdict follows from the checks and is not chosen by the analyst. The rules are Bioinformaniac's own acceptance criteria and not a regulatory standard, and the report says so; a reviewer can apply a stricter rule to the same printed numbers. After the gate come the study, the data fingerprint (a hash of the analysed table, so that two copies of a report can be shown to describe the same data), each analysis with its methods and settings, the limitations and a glossary.
For the statistics, see chapters S1 to S9, especially S4 (counts and compositional data) and S5 (diversity and PERMANOVA). For the human virome as a clinical and ecological subject, read Izard and Rivera, ch. 9 (Metagenomics for Microbiology), and for the computational side of host prediction, Frishman and Marz (eds.), ch. 10 (Virus Bioinformatics).
A reagent contaminant: it is more prevalent in the blanks than in the samples, which is the signature the prevalence method tests for.
Relative abundances sum to a constant, so they are not independent and raw correlations are distorted. The centred log-ratio removes this, to a first approximation.
Only that no flag was found in the part that was sequenced. A toxin or resistance gene could lie in the missing 30 %, so the screen sends such genomes to review.
It follows mechanically from stated rules applied to printed numbers, so a reviewer can check it, apply a stricter rule or see that a check failed. It does not depend on the analyst's judgement.
An isolate genome is only as good as the reads behind it. A few minutes spent on read statistics saves hours of puzzling over a broken assembly.
For every sample, check the number of reads, total bases, mean and N50 read length, and mean quality. Divide total bases by the expected genome size to get coverage. Short-read assemblers need roughly 50 to 100-fold coverage; long-read assemblers do well at 30 to 50-fold of good reads, and more is not always better because very deep data can be subsampled to the longest, most accurate reads.
The same checks as in chapter 3 apply, with one addition: an isolate should contain one genome. A k-mer profile with one coverage peak and a read classifier that assigns nearly all reads to one species confirm it. Ismail, Bioinformatics: A Practical Guide to NGS Data Analysis, ch. 1, walks through and trimming, and Buffalo, Bioinformatics Data Skills, ch. 10, explains the FASTQ format and quality encodings at the level needed to write your own checks.
Nanopore runs often write many small FASTQ files per barcode, which are concatenated into one file per sample. Unclassified reads and barcodes with very few reads should be inspected: a near-empty barcode is a failed sample, not a small genome. Long reads are filtered by minimum length (for example 1 kb) and mean quality, keeping the longest reads up to a target coverage. Short reads are adapter- and quality-trimmed with or (Chen et al., 2018; Bolger et al., 2014).
A quick taxonomic screen of a subset of reads (for example with ) reveals mixed cultures or contamination before assembly. Two species in one culture produce an assembly roughly twice the expected size and high contamination scores later (chapter 17).
Usually not: beyond about 100× short reads add little and can slow assemblers or add error k-mers. Subsample to a target coverage.
Two strains or species in the culture. Check with a read classifier before assembling.
Assembly reconstructs a genome from overlapping reads. Repeats are what make it difficult, and read length is what solves them.
Short-read assemblers build a de Bruijn graph: reads are broken into k-mers, and k-mers that overlap by k−1 bases are linked. A unique region becomes a simple path; a repeat becomes a node where paths cross. uses several k-mer sizes and paired-end information to resolve as much as possible (Bankevich et al., 2012). Long-read assemblers mostly use overlap or repeat graphs. builds a repeat graph from approximate overlaps and resolves repeats using reads that span them (Kolmogorov et al., 2019). When a read is longer than every repeat, the graph collapses to one circle per replicon.
A worked de Bruijn graph, with the repeat that breaks it, is in chapter 31 (fungal assembly); the logic is the same for bacteria, whose main repeats are rRNA operons (about 5 kb, often 5–10 copies) and insertion sequences (1–2.5 kb). Reads longer than the longest repeat resolve the whole chromosome, which is why a single Nanopore or HiFi run now routinely gives complete bacterial genomes. Ismail, Bioinformatics: A Practical Guide to NGS Data Analysis, ch. 3, compares greedy, overlap–layout–consensus and de Bruijn assemblers; Low & Tammi, Practical Bioinformatics for Beginners, ch. 6, compares short-read, long-read and hybrid assemblies of one genome.
| Data | Common choice | Typical result |
|---|---|---|
| Illumina only | | Tens to hundreds of contigs; repeats and rRNA operons collapsed |
| Nanopore only | , then polishing | Complete circular chromosome and plasmids at good coverage |
| PacBio HiFi | (HiFi mode) or | Complete genomes, very accurate without polishing |
| Short + long reads | (hybrid) | Complete genome with short-read base accuracy |
builds a short-read assembly graph and uses long reads to bridge its repeats, which gives complete genomes with high base accuracy when both data types are good (Wick et al., 2017). Benchmarks of long-read assemblers show that each has characteristic failure modes, such as missing small plasmids or duplicating the start and end of a circular contig, and that combining several assemblies into a consensus (Trycycler) gives the most reliable finished genomes (Wick & Holt, 2019; Wick et al., 2021).
Small plasmids (a few kilobases) are often lost by long-read assemblers because size selection removes their short molecules, or because they are filtered out as low-length reads. If plasmids matter, check the short-read assembly or rerun with relaxed read-length filters.
Reads shorter than the repeat cannot tell which copy they come from, so the graph collapses the copies and contigs end at their borders.
Small plasmids are under-represented in size-selected long-read libraries and may be filtered as short reads; short-read or hybrid assembly recovers them.
A long-read assembly gets the structure right. Polishing fixes the remaining base-level errors, and circularisation gives the genome a clean, conventional start.
Residual errors in long-read assemblies are mostly small insertions and deletions in homopolymers. Each indel in a coding region shifts the reading frame and splits a gene into fragments, which inflates gene counts and creates false pseudogenes. Polishing aligns reads back to the draft and corrects positions where the consensus disagrees with the draft.
A circular replicon assembled as a linear contig may have overlapping ends. Circularisation trims the overlap, and rotation sets the start position, by convention at the dnaA gene for chromosomes and at repA for plasmids where possible. A consistent start makes genomes easier to compare and annotate.
An indel inside a gene shifts the reading frame, creating a premature stop; the gene is predicted as two or more fragments.
It gives a standard coordinate system, so genomes can be compared and annotations do not split a gene across the sequence ends.
Two different questions decide whether an assembly is fit for use: is it contiguous, and is it complete and clean?
reports the number of contigs, total length, largest contig, N50, L50 and GC content (Gurevich et al., 2013). N50 is the length such that contigs of that size or longer contain half the assembly. A finished bacterial genome has one contig per replicon and an N50 close to the chromosome size; short-read assemblies typically have N50 values of tens to hundreds of kilobases. Total length and GC content should match what is known for the species. With a reference genome, also counts misassemblies and mismatches.
uses sets of marker genes that are present in a single copy in nearly all genomes of a lineage (Parks et al., 2015). Missing markers lower completeness; duplicated markers raise contamination. Lineage-specific marker sets, chosen by placing the genome in a reference tree, are more precise than universal sets. replaces fixed marker sets with a machine-learning model and works better for novel or reduced genomes (Chklovski et al., 2023).
| Category (MIMAG) | Completeness | Contamination |
|---|---|---|
| High quality | > 90 % | < 5 %, plus rRNA genes and ≥ 18 tRNAs |
| Medium quality | ≥ 50 % | < 10 % |
| Low quality | < 50 % | < 10 % |
These thresholds were defined for genomes from metagenomes (Bowers et al., 2017); an isolate genome should comfortably exceed the high-quality bar. High contamination in an isolate usually means a mixed culture.
Total 300 kb; half is 150 kb, reached at the second contig. N50 = 80 kb, L50 = 2.
Completeness is excellent, but 8 % contamination exceeds the usual 5 % limit for high quality: check for a second organism before publishing.
With a whole genome in hand, species identity is decided by genome-wide similarity, not by a single marker gene.
Average nucleotide identity (ANI) compares two genomes over their shared regions. Across many thousands of genome pairs, ANI values show a clear gap around 95 percent, which matches traditional species boundaries well (Jain et al., 2018). Two genomes above about 95 percent ANI are generally the same species. Fast estimators such as use k-mer sketches to screen large collections before an exact calculation (Ondov et al., 2016).
Christensen (ed.), Introduction to Bioinformatics in Microbiology, ch. 7, explains the move from 16S identity and DNA–DNA hybridisation to genome-based species definitions; ch. 6 covers the phylogenetic methods behind .
is a genome-based taxonomy built from a phylogeny of conserved single-copy proteins and normalised so that ranks correspond to comparable evolutionary depth (Parks et al., 2022). It reorganised many groups whose traditional names did not reflect their phylogeny, so names sometimes differ from those in older literature and from . places a genome in the reference tree and assigns species by ANI to the closest reference (Chaumeil et al., 2022). A genome with no species-level match is reported to genus, which may indicate a new species.
Very likely yes: the boundary is 95–96 %. Check that the aligned fraction is high enough.
It defines taxa from a genome phylogeny with consistent rank depths, splitting groups that were lumped together by phenotype.
Annotation finds the genes and other features in a genome and gives them names. The quality of every downstream biological claim depends on it.
Structural annotation locates features: protein-coding sequences (CDS), predicted by in most bacterial pipelines, plus rRNAs, tRNAs, tmRNA, non-coding RNAs, CRISPR arrays and origins of replication. Functional annotation assigns a product name and identifiers to each CDS by comparison with protein databases. made this fast and standard for a decade (Seemann, 2014). uses a large precomputed database of protein clusters, identifies known proteins by exact hashing, and returns database cross-references for a high share of genes (Schwengers et al., 2021).
Bacterial gene finders such as score open reading frames with the codon and start-site statistics of the genome itself, a simple form of the hidden Markov models that eukaryotic gene finders need (chapter 33). Baxevanis, Bader & Wishart, Bioinformatics, ch. 5, and Zvelebil & Baum, Understanding Bioinformatics, ch. 9–10, explain both, including how accuracy is measured. Christensen (ed.), Introduction to Bioinformatics in Microbiology, ch. 2–3, describe the databases used for functional annotation and how far a function can be transferred from a distant homolog.
A typical bacterial genome has about one gene per kilobase. Counts far above that often mean frameshift errors fragmenting genes (chapter 16); counts far below suggest a fragmented assembly. Missing rRNA operons in a short-read assembly are usually an assembly artefact, because identical operon copies collapse. A large fraction of "hypothetical protein" annotations is normal, especially for less studied genera.
Bacterial genes are mostly continuous open reading frames without introns and the genome is densely coding, so long ORFs with good codon statistics are usually genes.
Not necessarily: it is typical for bacteria. It means no characterised homolog was found, not that the gene is fake.
Once a genome is annotated, the practical questions are usually about lineage, mobile elements and genes of concern.
assigns an allele number to each of about seven housekeeping genes; the combination defines a sequence type (ST). STs are stable, comparable across laboratories and widely used in epidemiology. Schemes and allele definitions are curated in (Jolley, Bray & Maiden, 2018). An "untyped" result usually means no scheme exists for the species or a new allele is present.
assigns an allele number to each of seven housekeeping gene sequences and a sequence type (ST) to each combination of alleles: two isolates share an ST only if all seven alleles are identical. Core-genome does the same with hundreds to thousands of genes and resolves outbreaks. Christensen (ed.), Introduction to Bioinformatics in Microbiology, ch. 11, traces typing from to whole-genome methods.
Plasmids and prophages carry many of the genes that make strains different, including resistance and virulence genes. In a complete assembly, small circular contigs are plasmid candidates. classifies sequences as chromosome, plasmid or virus using marker genes and a neural network, and annotates them (Camargo et al., 2024). It works on whole genomes and on metagenome assemblies.
Resistance genes are found by comparison with curated databases such as 's database or (Feldgarden et al., 2021). A gene's presence predicts potential resistance but not the phenotype: expression, point mutations and the genetic context matter. Report the database version and identity thresholds, and confirm important findings with phenotypic testing.
Not necessarily: uses 7 genes and is coarse. Use cgMLST or SNP distances to judge recent transmission.
Usually, but not always: the gene may be truncated, not expressed, or need a partner. Phenotypic testing remains the reference.
Comparing a strain with its parent or with close relatives reveals the mutations behind an evolved trait, an outbreak link or a lab adaptation.
Reads from the strain of interest are aligned to a reference genome, ideally a complete genome of the parent strain, with for short reads or for long reads (Li & Durbin, 2009; Li, 2018). A variant caller then reports single-nucleotide polymorphisms (SNPs) and short indels with their read support. is a widely used haplotype-based caller (Garrison & Marth, 2012). For haploid bacteria, true variants should be supported by nearly all reads at a position; variants supported by a fraction of reads point to mixed populations, mapping errors in repeats or contamination.
Variant calling is two problems: mapping each read to the right place, then deciding whether a difference is real. Ismail, Bioinformatics: A Practical Guide to NGS Data Analysis, ch. 2 and 4, cover both, including base and mapping qualities, duplicate marking and the statistics of genotype calling; Buffalo, Bioinformatics Data Skills, ch. 11, explains the SAM/BAM format you will meet at every step.
Insertions, deletions, inversions and duplications larger than about 50 bases are detected from split and discordant alignments. Long reads make this much more reliable because a single read can span the whole event (Sedlazeck et al., 2018). Comparing two complete assemblies directly is an alternative that avoids reference bias.
Across several genomes, gene presence and absence (the pangenome) shows which genes are shared and which are strain-specific. A heatmap of gene families across genomes, clustered by similarity, quickly shows groups of strains with similar accessory content.
Reads from different copies map to the same place; differences between copies look like variants. Mask repeats or require unique mapping.
The core is genes present in (nearly) all strains; the accessory genome is genes present in some, often on mobile elements.
An analysis is reproducible when someone else, or you in a year, can rerun it and get the same result. That depends less on good intentions than on how the work is set up.
Results change with tool versions and, even more, with reference database versions. Record both for every step. Package managers such as with Bioconda channels make exact versions installable (Grüning et al., 2018); containers (, ) freeze the whole environment. A one-line note such as "SILVA 138.1, DADA2 1.30, MetaPhlAn 4.1 with database vJun23" is worth more than a paragraph of prose.
Workflow managers describe each step, its inputs and outputs, and its software environment, then run the steps in the right order, in parallel where possible, and resume after a failure without redoing finished work. is widely used in genomics, including by the nf-core community, and runs the same workflow on a laptop, a cluster or the cloud (Di Tommaso et al., 2017). follows a similar idea with Python-like rules.
Buffalo, Bioinformatics Data Skills is the best single book on this side of the work: project layout and documentation (ch. 2), version control with Git (ch. 5), Unix data tools (ch. 7) and writing robust pipelines (ch. 12). Its rule of thumb applies everywhere: anything you did by hand you will need to redo, so script it.
Raw data should be read-only and stored once. Code lives in version control. Results are generated and can always be deleted and remade. Random seeds should be fixed for anything stochastic: rarefaction, permutation tests, cross-validation splits and random forests.
Deposit raw reads in an archive of the International Nucleotide Sequence Database Collaboration (ENA, SRA or DDBJ), with sample metadata that follows the MIxS checklists. Genomes go to the same archives with their annotation. The FAIR principles (findable, accessible, interoperable and reusable) summarise what reviewers and funders now expect (Wilkinson et al., 2016).
Recording exact versions (a environment file or container) for the tool and its database.
Every result can then be regenerated from the same starting point; accidental edits to raw data cannot be undone.
A good methods section lets a reader judge every choice that could have changed the result. A good results section separates what the data show from what they suggest.
Give effect sizes with uncertainty: PERMANOVA R², fold changes with confidence intervals, AUC with its spread. State how many methods agreed on a differentially abundant taxon. Distinguish association from causation, and genetic potential from activity. Negative results reported with their power are useful to the field.
Report what was estimated, not only what was significant: effect sizes with intervals, the number of tests and the false discovery rate. When thousands of features are tested, a q-value of 0.05 means that about 5 % of the reported list is expected to be false; Efron & Hastie explain this and the related local false discovery rate in plain terms (Efron & Hastie, Computer Age Statistical Inference, ch. 15).
Names and assignments change between releases; without the version the result cannot be reproduced or compared.
No: add the effect size (for example log fold change with a confidence interval), the method and the number of tests.
RNA-seq turns the messenger RNA in a sample into millions of short sequences and counts them. Every later step, from normalisation to pathways, rests on two things decided before the first read exists: what RNA you captured, and how the samples were arranged.
A gene is transcribed into RNA; only a small part of a cell's RNA is protein-coding messenger RNA (mRNA, about 2–4 %), while most is ribosomal RNA. In eukaryotes the primary transcript contains exons and introns; introns are spliced out (most begin with GT and end with AG), and one gene can produce several isoforms by alternative splicing. Bacterial genes have no introns and are often organised in operons, several genes transcribed together from one promoter. These differences shape everything that follows: eukaryotic reads must be aligned across splice junctions, and bacterial libraries cannot be built by catching poly(A) tails (Ismail, Bioinformatics: A Practical Guide to Next Generation Sequencing Data Analysis, ch. 5.1).
The library is made by enriching mRNA, reverse-transcribing it into complementary DNA (cDNA), fragmenting it, adding adapters and sequencing. Each read is a random fragment of a random transcript. A gene's count is therefore proportional to how many copies of its transcript were present multiplied by its length: a long transcript breaks into more fragments than a short one at the same expression.
A biological replicate is an independent organism, culture or patient; a technical replicate is the same RNA prepared or sequenced twice. Differential expression asks whether a difference between groups is larger than the difference between independent individuals, so only biological replicates answer it. Three per group is a bare minimum; six or more is safer for modest effects, and beyond about 20 million reads per sample, more replicates add far more power than more depth (Christensen, ch. 10; Schurch et al. 2016).
Baxevanis and colleagues recommend a small pilot experiment: it gives a realistic estimate of variability, which makes a proper power calculation possible before the full study is funded (Baxevanis, Bader & Wishart, Bioinformatics, ch. 10).
A batch effect is any systematic difference introduced by handling: extraction day, kit lot, technician, sequencing run or lane. Batch effects are harmless when every batch contains every group, because the model can estimate and remove them. They are fatal when a batch coincides with a group (all controls on Monday, all treated on Tuesday): then no method can separate treatment from Monday (Baxevanis et al., Bioinformatics, ch. 10).
The 5-kb gene, about five times as many, because it breaks into five times as many fragments. Comparing genes within a sample therefore needs length correction (TPM).
Usually 6 + 6. Power to detect differences comes mainly from the number of independent biological replicates; 10 million reads per sample already quantify most expressed genes.
Bacterial mRNA lacks a poly(A) tail, so oligo-dT beads cannot catch it; instead, the abundant ribosomal RNA is removed.
The goal of the first half of an RNA-seq analysis is one table: genes in rows, samples in columns, a number in each cell. There are three ways to get there, and choosing between them is a question of what you need to know.
Raw reads are first inspected with (one report per file) and summarised across samples with . The usual problems are adapter sequence at read ends (the fragment was shorter than the read), falling base quality towards the 3′ end, over-represented sequences (often ribosomal RNA) and very high duplication. Tools such as , or remove adapters and low-quality tails; is several times faster on large datasets (Low & Tammi, Practical Bioinformatics for Beginners, ch. 8). Duplication is normal in RNA-seq: a highly expressed gene produces many identical fragments, so duplicates are not removed as they are in genome sequencing.
and align reads to the genome and can split a read across an intron. first finds the longest exact matches of each read ("seeds"), then stitches seeds that lie near each other into a spliced alignment (Ismail, ch. 5.3.2). Alignment gives positions, so it supports splicing analysis, variant calling and the discovery of unannotated genes; it needs a genome and an annotation (GTF) and, for , a lot of memory.
and skip base-by-base alignment. They determine which transcripts are compatible with each read and then use an expectation-maximisation algorithm to share reads that fit several isoforms. They are fast, model multi-mapping reads and correct for fragment-level biases (GC content, sequence context). Transcript estimates are summed to genes for gene-level testing (). For gene-level differential expression they are as accurate as alignment-based counting.
Without a reference, or assemble reads into transcripts. This is the route for non-model organisms; it is slower and produces redundant transcripts that need clustering (Low & Tammi, ch. 8).
After alignment, or assign each read to a gene using the annotation. A read overlapping two genes is ambiguous and, in the default "union" mode, not counted; a read aligning to several places is multi-mapping and usually skipped (Ismail, ch. 5.3.4). With a stranded library, telling the counter the right strand setting matters: the wrong setting can silently lose most reads. Pseudo-aligners make this choice for you ('s -l A detects the library type).
Log.final.out).When you need positions: splice-junction or isoform discovery, variant calling in transcripts, or genes missing from the annotation. For gene-level differential expression with a good transcriptome, is faster and equally accurate.
Because a highly expressed gene genuinely produces many identical fragments; removing them would compress exactly the differences you want to measure.
The strand setting: with a reversed strand parameter, reads land on the wrong strand and overlap antisense or neighbouring genes.
Raw counts mix three things: how much a gene is expressed, how deeply the sample was sequenced, and what else was in the sample. Normalisation removes the last two so that the first can be compared. Then, before any test, you look at the samples.
With k reads on a gene of length l in a library of N reads (Ismail, ch. 5.3.5):
FPKM is RPKM counted in fragments (read pairs). TPM divides by length first and then rescales, so TPM always sums to one million in every sample; RPKM does not, which makes RPKM values from different samples subtly incomparable.
Scaling by library size assumes the total amount of RNA is shared out the same way in every sample. When a few genes become very highly expressed in one condition, they take reads from every other gene, and simple scaling makes unchanged genes look down-regulated. Two robust methods assume instead that most genes do not change:
Genes with almost no reads carry no information and add to the multiple-testing burden. 's keeps genes with a minimum count-per-million in at least as many samples as the smallest group (Ismail, ch. 5.3.7.4).
Count variance grows with the mean, so on the raw scale a few highly expressed genes dominate every distance, and on a naive log scale the noisy low counts dominate. The variance-stabilising transformation (VST) or regularised log of , and log-CPM with a prior count in , give roughly constant variance across the range. Use them for PCA, clustering and heatmaps, never for the test itself.
PCA on the 500 most variable genes is the first plot to make. Colour it by condition, then by every technical variable: if samples separate by extraction date or sequencing run, a batch effect is present (Baxevanis et al., Bioinformatics, ch. 10). Hierarchical clustering joins the two closest samples, recomputes distances, and repeats until one tree remains; single, complete and average linkage differ only in how the distance between groups is defined (Baxevanis et al., ch. 10; Zvelebil & Baum, Understanding Bioinformatics, ch. 16). A heatmap of row-scaled expression with its dendrograms shows which genes drive the groups.
Sanity checks catch errors no statistic can: expression of Y-chromosome genes (or XIST) must match the recorded sex. Baxevanis and colleagues describe a dataset in which samples clustered by sex rather than by tissue, revealing mislabelled samples.
Sample swaps happen in every large study. A cheap test uses genes whose expression depends on sex: XIST is expressed from the inactive X in females, and genes on the Y chromosome (RPS4Y1, DDX3Y, KDM5D, UTY, EIF1AY) only in males. Plotting one against the other separates the samples into two clouds; a sample in the wrong cloud for its recorded sex was probably swapped or mislabelled (Baxevanis, Bader & Wishart, Bioinformatics, ch. 10). RNA-realm draws this plot in step 02 for human and mouse data and flags disagreements with a sex column in the metadata. The demo study contains one deliberately swapped label.
TPM normalises by length first and then divides by the sum of the length-normalised values, so the total is fixed at 10⁶; RPKM divides by the read total before length correction, so its sum depends on the length distribution of the sample's reads.
The median would itself move, and the size factors would be biased. Both TMM and median of ratios assume that most genes do not change; global shifts (for example, overall transcription doubling) need spike-in controls.
Check whether dates are balanced across conditions. If they are, include date as a covariate in the model. If one condition was extracted on one date only, the effect cannot be separated; report it as a limitation.
Differential expression asks, for every gene at once, whether the difference between groups is larger than the difference between replicates. The statistics look complicated, but they rest on four ideas: counts are over-dispersed, genes can borrow strength from each other, the design must be written down, and thousands of tests need a different kind of error control.
Each read either comes from gene g or not, so the count of a gene in one library is binomial; because any single gene takes a tiny share of millions of reads, this becomes a Poisson distribution, whose variance equals its mean (Ismail, ch. 5.3.6). The Poisson describes technical sampling only. Independent biological replicates vary more, so the variance is modelled as
where φ is the dispersion. Its square root is the biological coefficient of variation (BCV): φ = 0.1 means replicates differ by about 32 % (Ismail, ch. 5.3.7; Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R, ch. 11).
With three replicates per group, a gene's own variance estimate is very noisy. The solution is empirical Bayes shrinkage: fit a trend of dispersion against mean expression across all genes, then pull each gene's estimate part-way towards the trend; genes with less information are pulled further. computes a common, a trended and a gene-wise ("tagwise") dispersion; and use the same principle (Ismail, ch. 5.3.7.6).
Efron and Hastie explain why this works with a classic example: predicting baseball players' season batting averages from their first 90 at-bats. Shrinking every player's early average towards the overall mean (the James–Stein estimator) reduces the total error substantially, because each player borrows indirect evidence from the others. They also show the cost: a few genuinely extreme players are pulled too far (Efron & Hastie, Computer Age Statistical Inference, ch. 7). The same is true for genes, which is why shrunken fold changes ( in ) are best for ranking, while a gene with a large, well-supported effect keeps most of it.
Each gene is fitted with a generalised linear model on the log scale, with the library size as an offset (Ismail, ch. 5.3.6; Faraway, Extending the Linear Model with R, ch. 5):
) adds a second, gene-specific variability and is recommended for most designs (Ismail, ch. 5.3.7.8).All three control false discoveries well in designed experiments with few replicates. In large human cohorts (dozens of samples per group) their distributional assumptions can fail; the Wilcoxon rank-sum test then controls the false discovery rate better (Li et al. 2022).
At p < 0.05, testing 20,000 unchanged genes yields about 1,000 false positives. Controlling the family-wise error rate (Bonferroni) prevents even one, but is too strict to find anything. The false discovery rate (FDR) controls instead the expected proportion of false discoveries among the genes you report (Efron & Hastie, ch. 15).
The Benjamini–Hochberg recipe: order the N p-values, find the largest i with p(i) ≤ i·q/N, and reject hypotheses 1 to i.
The Benjamini–Hochberg q-value controls the false discovery rate of the whole list above a threshold. Efron's local false discovery rate answers a sharper question for each gene: what is the probability that this particular gene is null? It needs two densities: f(z), the distribution of all genes' z-values, and f0(z), the distribution of null genes. Then lfdr(z) = π0 f0(z) / f(z), where π0 is the share of null genes (Efron & Hastie, Computer Age Statistical Inference, ch. 15).
Efron's key observation is that the null need not be the theoretical N(0, 1). Correlation between genes, hidden batches or unmodelled covariates widen or shift the bulk of the z-values; estimating the null from that bulk ("empirical null", by fitting a parabola to the log density in the middle, called central matching) gives more honest calls.
RNA-realm estimates the empirical null in step 05, draws the histogram with f(z) and π0f0(z), and adds an lfdr column to the results table.
It assumes variance equals the mean, which only describes sampling of reads. Real individuals differ biologically, adding variance proportional to the mean squared; the negative binomial's dispersion term captures this.
Yes. The thresholds double (0.05 for i = 5), and 0.030 ≤ 0.050; in fact gene 6 (0.040 ≤ 0.060) would be reported too, giving six genes.
The large estimate rested on very little information (low counts or high variance); the data cannot support a big change for this gene. Rank it by the shrunken value.
A list of 400 differentially expressed genes is not yet a result. Gene-set methods ask which processes those genes belong to; co-expression methods ask which genes rise and fall together. Both turn individual genes into biology.
ORA asks whether a gene set (for example "interferon response") contains more of your significant genes than chance would give. The probability comes from the hypergeometric distribution, as when drawing coloured balls from an urn without replacement.
The background must be the genes that could have been called significant (those tested), not the whole genome: lowly expressed tissue-specific genes would otherwise make every expressed pathway look enriched (Timmons et al. 2015).
ORA throws away everything below the significance cut-off. uses the whole list ranked by the test statistic. Walking down the list, a running score rises at each gene in the set and falls otherwise; the maximum deviation is the enrichment score. A set whose genes cluster at the top (or bottom) of the ranking is enriched even if no single gene is significant, which is how coordinated small shifts in, say, mitochondrial genes are found. Significance comes from permuting gene sets of the same size.
Start with the 50 Hallmark sets: curated, largely non-redundant summaries of well-defined processes. Then use GO Biological Process or Reactome for detail. Gene Ontology terms overlap heavily, so dozens of related "significant" terms often describe one signal. KEGG pathways are free for academic use only (Baxevanis et al., Bioinformatics, ch. 13 Biological Networks and Pathways).
Genes that respond together are often regulated together. Compeau and Pevzner tell this story through yeast: when Saccharomyces cerevisiae runs out of glucose it inverts its metabolism and starts consuming the ethanol it produced, the diauxic shift, and hundreds of genes change expression in coordinated waves. Clustering their expression profiles over time recovers groups of genes with shared functions (Compeau & Pevzner, Bioinformatics Algorithms: An Active Learning Approach, ch. 8 How Did Yeast Become a Wine Maker?).
builds a network where every pair of genes is connected with a weight from their correlation, raised to a soft-threshold power β that keeps strong correlations and fades weak ones. Topological overlap then measures how many neighbours two genes share, and hierarchical clustering of 1 − overlap gives modules. Each module is summarised by its eigengene (its first principal component), which is correlated with sample traits; genes most correlated with the eigengene are the hubs.
Modules need many samples: with fewer than 15, correlations between genes are too noisy, and modules are hypotheses at best.
Because it tests whether the set's genes are concentrated at one end of the ranked list; many small shifts in the same direction add up even when each is too small to pass a per-gene threshold.
The expected count drops to 100 × 500 / 20,000 = 2.5, doubling the apparent enrichment. Unexpressed genes could never have been significant, so this inflates every result.
Two genes that both double after treatment have the same response even if one is 100 times more abundant; correlation compares shapes, Euclidean distance compares levels.
A walk through a complete bulk RNA-seq analysis in RNA-realm, from FASTQ files or a count table to a methods section, with the decision that matters at each step and the chapter that explains it.
Upload paired (_R1/_R2) or single-end FASTQ files; they go up in pieces, so tens of gigabytes are fine. Choose the human or mouse transcriptome, or upload your organism's transcript FASTA. RNA-realm trims with , quantifies with (selective alignment, GC and sequence bias correction) and sums transcripts to genes. Open the report: mapping rate, adapter content and duplication tell you whether each library is usable (chapter 25). Then load the results with a sample sheet.
Or start from counts: one table with all samples, or one file per sample from , , , , or . The first column of the sample sheet must contain the sample names exactly as in the count file; every other column is a variable. Human and mouse genes get symbols, gene types and chromosomes automatically.
Check library sizes, the gene-type composition, the mitochondrial share and the sample correlation heatmap. Flagged outliers are not removed: look at them in the PCA first (chapter 26).
Choose the grouping variable for ; keep TMM as a robust default, or median of ratios for size factors. Keep VST for plots.
Colour the PCA by condition and then by every technical variable. The "drivers" heatmap gives the R² of each variable on each component; a technical variable with high R² on PC1 or PC2 belongs in the model. "Remove a batch for display" changes only plots.
Choose the variable, the reference level and the test level; tick covariates. Run and - together; genes found by both form the consensus. With eight or more independent samples per group, add the Wilcoxon test. Read the p-value histogram before the gene list, and plot single genes (chapter 27).
Start with the Hallmark collection, then GO or Reactome. Click a term for its running enrichment score and leading-edge genes (chapter 28).
With fifteen or more samples, find modules, look at which module follows your condition and which follows batch, and read each module's hub genes and best Hallmark term.
The report writes a methods paragraph for exactly the steps you ran, with linked references. The Download menu exports every table.
Yes. Batch is balanced across conditions in the demo, so including it removes its variance and increases power without biasing the comparison.
Report the method you chose in advance with its FDR; the 220 consensus genes are the most robust and are a good starting point for interpretation, but the choice should be stated before looking.
Fungal genomes make sense only in the light of how fungi live: they digest food outside their cells, grow as networks of hyphae that may hold many nuclei, switch between haploid and diploid stages, and make chemical weapons. Each of these habits leaves a mark that a genome analyst has to recognise.
Fungi include moulds, yeasts and mushrooms. About 100,000 species have been described, and estimates of the real number reach 1.5 million or more; environmental sequencing suggests that over 90 % of fungal species remain undiscovered. Fungi are the closest relatives of animals among eukaryotes (Madigan et al., Brock Biology of Microorganisms, ch. 18).
Fungi are chemoorganotrophs that feed by secreting enzymes: they digest polymers such as cellulose, chitin and proteins outside the cell and absorb the monomers. Decomposition of wood is a hallmark: white-rot basidiomycetes degrade both cellulose and lignin, while brown-rot fungi take the cellulose and leave the lignin (Brock, ch. 18.8). In a genome this lifestyle appears as large families of carbohydrate-active enzymes (CAZymes) and secreted proteins.
(Brock, ch. 18.9–18.13)
Most fungi grow as hyphae, tubular cells that extend at the tip and form a mycelium. Hyphae can be divided by cross-walls (septate) or be one continuous cell with hundreds of nuclei (coenocytic). In many basidiomycetes, two compatible haploid mycelia fuse and the resulting dikaryon keeps two genetically different nuclei in every cell for most of its life (Brock, ch. 18.13). For a genome project this means:
In baker's yeast, the active MAT locus determines mating type a or α. Two silent copies of the mating-type genes sit elsewhere in the genome, and a haploid cell can switch type by copying a silent cassette into MAT (Brock, ch. 18.12). Three near-identical copies of the same sequence are exactly the kind of repeat that breaks short-read assemblies. Mating-type loci are also of interest in their own right: their structure (one locus or two, two alleles or many) defines whether a species is heterothallic or homothallic.
| Feature | Typical range | Consequence for analysis |
|---|---|---|
| Genome size | 2.9 Mb (Microsporidia) to > 1 Gb (some rusts); most 20–60 Mb | Long reads and 30–50× coverage are affordable |
| Genes | ~2,000 to ~20,000 | Gene prediction needs a species-appropriate model |
| Introns | Few in yeasts, several per gene in filamentous fungi | Ab initio predictors must find splice sites |
| Repeats | Under 5 % in yeasts to over 50 % in some pathogens | Mask before predicting genes |
| Chromosome ends | Telomeric repeat TTAGGG in most fungi; irregular TG-rich repeats in budding yeasts | Complete chromosomes have telomeres at both ends |
| Ribosomal DNA | Tandem arrays of 18S–ITS1–5.8S–ITS2–28S | Collapsed in assemblies; source of the ITS barcode |
Genome history matters too. Compeau and Pevzner recount how the ancestor of Saccharomyces underwent a whole-genome duplication; most duplicates were later lost, leaving about 13 % of genes in pairs, such as the alcohol dehydrogenases ADH1 and ADH2 that let yeast make ethanol and then consume it (Compeau & Pevzner, Bioinformatics Algorithms, ch. 8). Such ancient duplicates explain why even a perfect yeast assembly shows a few "duplicated" genes in (chapter 32).
Fungi produce antibiotics, toxins and pigments: penicillin from Penicillium, statins, aflatoxins from Aspergillus. The genes for one compound are usually clustered in the genome as a biosynthetic gene cluster, which is why tools such as search for clusters rather than single genes (chapter 34).
Fruiting bodies are dikaryotic: each cell carries two different haploid nuclei. Where the two haplotypes differ enough, the assembler builds both copies; purging duplicated haplotypes or assembling haplotypes separately solves this.
White-rot fungi degrade lignin as well as cellulose, which requires oxidative enzymes (CAZy "auxiliary activities" such as class II peroxidases); brown-rot fungi leave lignin largely intact.
As an obligate intracellular parasite it relies on its host for basic metabolism, and lost the genes for whole pathways such as the citric acid cycle.
Before assembling, count k-mers: they tell you the genome size, how heterozygous it is and how much of it repeats. Then choose an assembler that matches your reads, and judge the result with numbers you understand.
Cut every read into overlapping words of length k (k = 21 is typical) and count how often each distinct k-mer occurs. Plot the number of distinct k-mers against their count, and the genome speaks:
Ismail groups assembly algorithms into three families (Ismail, Bioinformatics: A Practical Guide to Next Generation Sequencing Data Analysis, ch. 3):
Long reads (Oxford Nanopore, PacBio HiFi) span most fungal repeats, including transposons and rDNA units, and routinely give chromosome-scale contigs; and are standard choices. Short reads alone () give hundreds of contigs but accurate bases. A hybrid strategy assembles long reads and then polishes the consensus with short reads (, ), correcting the small insertions and deletions that remain in long-read assemblies, especially in homopolymers (Low & Tammi, Practical Bioinformatics for Beginners, ch. 6). Polishing should converge: the number of changes should fall sharply from one round to the next.
Sort contigs from longest to shortest and add their lengths until half of the total is reached. The length of the contig that crosses the half-way mark is the N50; the number of contigs needed is the L50 (Ismail, ch. 3).
A heterozygous diploid (or dikaryon) with homozygous k-mers at 50 and haplotype-specific k-mers at 25. Expect heterozygosity to complicate the assembly.
With a larger k the repeated words (TC, CA) become longer, unique k-mers, so the branch disappears. Repeats shorter than k no longer create ambiguity.
It is optional: the changes are converging. Another round may fix a few more, but large numbers in later rounds would suggest the reads disagree with the assembly for other reasons (repeats, heterozygosity).
Three questions decide whether an assembly can be trusted: does it contain the whole genome, does it contain only this genome, and is the organism what you think it is?
searches the assembly for genes that are present in a single copy in almost every member of a lineage (for example 758 genes for all fungi, 2,137 for Saccharomycetes). The narrower the lineage, the more genes and the more sensitive the test (Low & Tammi, Practical Bioinformatics for Beginners, ch. 6). Each gene is reported as:
Run twice: on the genome (in genome mode, using the aligner) and on the predicted proteins. A gap between the two (complete in the genome, missing among proteins) points to gene-prediction problems, not a missing genome.
Most fungi end their chromosomes with the telomeric repeat TTAGGG (budding yeasts use irregular TG₁₋₃ repeats). A contig with telomeric repeats at both ends is a complete chromosome. Counting them is the simplest proof of a telomere-to-telomere assembly; a contig with a telomere at one end is a chromosome arm.
Plot every contig by its GC content against its read depth, with dot size proportional to length:
classifies each contig as nuclear, mitochondrial, plastid or prokaryotic with a neural network trained on k-mer composition, which makes these clouds easy to name.
The internal transcribed spacers (ITS1 and ITS2) between the ribosomal RNA genes are the official DNA barcode of fungi: variable enough to separate species, flanked by conserved genes that are easy to find. extracts the region; a search against or 's curated fungal ITS reference set confirms the species. Identification by ITS is as good as the reference database, and some genera (for example Fusarium, Aspergillus section-level) need secondary markers such as TEF1 or β-tubulin.
The assembly contains both haplotypes of a heterozygous genome (or two related genomes). Check the k-mer profile; purge duplicated haplotigs () or assemble haplotypes separately.
Almost certainly the mitochondrial genome: high copy number per cell, low GC.
18S is too conserved to separate close species; the spacers evolve faster, giving species-level resolution, while the flanking rRNA genes make the region easy to amplify and find.
Finding genes in a eukaryotic genome means recognising a grammar: a start codon, exons, splice sites, introns, a stop codon. Repeats must be hidden first, or the predictor will happily call every transposon a gene.
Fungal genomes carry transposable elements (retrotransposons with long terminal repeats, DNA transposons), simple sequence repeats, rDNA arrays and segmental duplications. Repeat content ranges from a few percent in yeasts to more than half the genome in some plant pathogens, where effector genes often sit in repeat-rich, fast-evolving compartments. builds a species-specific library of repeat families de novo; then marks every copy. Soft-masking writes repeats in lower case (ACGTacgtacgtACGT) instead of replacing them with N, so gene predictors can avoid starting genes in repeats yet still extend a real gene across a short repeat.
A eukaryotic protein-coding gene is a sequence of states: intergenic region, start codon (usually in a Kozak-like context), initial exon, donor splice site (GT), intron, acceptor splice site (AG), internal exons, terminal exon, stop codon (Baxevanis, Bader & Wishart, Bioinformatics, ch. 5). A hidden Markov model (HMM) represents each element as a state with its own sequence statistics (codon usage in exons, base composition in introns, position-specific signals at splice sites) and transition probabilities between states. Given a genome sequence, the Viterbi algorithm finds the most probable path through the states, which is the predicted gene structure (Zvelebil & Baum, Understanding Bioinformatics, ch. 10).
The parameters come from a training set of genes whose structure is known, which is why the species model matters: ships models for Saccharomyces, Aspergillus, Fusarium, Neurospora and others, and a model trained on a distant species misplaces introns. introduced this design for human genes; , -ES (self-training) and follow it (Baxevanis et al., ch. 5).
RNA-seq of the same strain is the single most valuable addition to a fungal annotation project; the RNA-realm chapters (24–29) describe how to produce it.
Accuracy is measured against a curated reference at the nucleotide, exon or gene level, counting true positives (TP), false positives (FP) and false negatives (FN) (Baxevanis et al., ch. 5).
In practice, without a curated reference, check in protein mode, the distribution of protein lengths (a spike of very short proteins suggests fragmented models) and the share of genes with domains.
tRNA genes are found with , which combines a fast first-pass search with covariance models of the tRNA secondary structure; ribosomal RNA genes with or RNAmmer.
The 61 sense codons encode 20 amino acids, so most amino acids have synonymous codons. Genomes do not use them equally: base composition (mutation bias) pushes towards GC- or AT-ending codons, and in highly expressed genes selection favours codons matching abundant tRNAs. Gene finders exploit this bias to tell coding from non-coding sequence (Zvelebil & Baum, Understanding Bioinformatics, ch. 10).
Wright's effective number of codons (ENC) summarises the bias of a gene in one number, from 20 (one codon per amino acid) to 61 (all synonymous codons equally used). Plotted against GC at synonymous third positions (GC3s), genes whose codon choice is explained by composition alone fall on the curve ENC = 2 + s + 29 / (s² + (1 − s)²); genes well below it are shaped by selection. Relative synonymous codon usage (RSCU) shows which codon is preferred for each amino acid.
FungFlow draws the ENC–GC3s plot and the RSCU table in the Genes results.
Introns have their own composition, length distribution and boundary signals; modelling them explicitly lets the model enforce the grammar (an intron must start with a donor and end with an acceptor, and must be followed by an exon).
It calls too much: almost all real coding sequence is found, but 40 % of what it calls coding is not. Typical causes are unmasked repeats or a model that over-predicts short exons.
In gene prediction: the genes are in the assembly but the models are incomplete or split. Better species model or RNA-seq evidence will help.
A list of 10,000 predicted proteins becomes biology when you know which ones digest plant walls, which make toxins, which are secreted to manipulate a host, and how this compares with relatives.
Most proteins are annotated by similarity to proteins whose function is known. Orthologs (genes separated by speciation) usually keep their function; paralogs (separated by duplication) often diverge. assigns each protein to a precomputed orthologous group and transfers its annotations: gene name, GO terms, EC numbers, KEGG pathways and COG functional category. Protein domains (, ) add evidence at the level of building blocks (Baxevanis et al., Bioinformatics, ch. 7 Predictive Methods Using Protein Sequences).
Because fungi digest food outside their cells (chapter 30), the enzymes they secrete are a direct readout of what they eat. The CAZy database groups them into glycoside hydrolases (GH), glycosyltransferases (GT), polysaccharide lyases (PL), carbohydrate esterases (CE), auxiliary activities (AA, mostly oxidative enzymes, including those that attack lignin) and carbohydrate-binding modules (CBM). annotates them with HMMs and sequence similarity.
Genes for a fungal natural product usually lie together: a core enzyme (polyketide synthase, non-ribosomal peptide synthetase, terpene synthase), tailoring enzymes, a transporter and often a regulator. finds these clusters, predicts their class and compares them with clusters of known products. Penicillin, made by Penicillium, is the classic example (Brock, ch. 18.9). A typical filamentous ascomycete has 30–70 clusters, most of whose products are unknown; this "cryptic" chemistry is why fungal genomes are mined for new drugs.
Secreted proteins carry an N-terminal signal peptide (predicted by or similar tools) and lack transmembrane domains. Plant- and animal-pathogenic fungi secrete effectors, often small, cysteine-rich proteins that manipulate the host; predicts them from sequence features. Predictions are a shortlist for experiments, not proof of function.
Annotation becomes meaningful by comparison: gene families expanded or lost relative to relatives (orthogroups from ), clusters unique to one strain, CAZyme profiles that set a pathogen apart from its saprotrophic sister species (Bergman, Comparative Genomics).
After a duplication, one copy is free to change function (chapter 30's ADH1/ADH2 example); orthologs in different species more often keep the ancestral function.
A wood decayer, probably a white-rot fungus able to degrade lignin.
Usually only a handful: most fungal clusters are cryptic, with no close match to a characterised cluster. Their products need expression and chemistry to identify.
How to go from sequencing reads of one fungal isolate to an assembled, checked and annotated genome in FungFlow, and how to read each part of the report, with the chapter that explains it.
In Reads, upload Illumina pairs, Nanopore or PacBio HiFi FASTQ files. If you already have an assembly, upload the FASTA in Assembly instead: FungFlow then starts at quality control. For a new isolate the best value today is Nanopore R10 (30–50×) plus Illumina (40×) for polishing, or HiFi alone (chapter 31).
In New run, mark each file (Illumina R1, R2, long reads, assembly). Give the expected genome size if you know it, the ploidy (haploid for most filamentous ascomycetes), the narrowest lineage that contains your fungus, the gene model of the closest species, and the telomere repeat (TTAGGG for most fungi). De novo repeat modelling is slower but gives much better masking. Runs queue on the server and continue when you close the page; follow them in Runs.
Remove contaminant contigs, check that duplication is low, and consider RNA-seq evidence for gene models ( or ) if genes are your focus. Submit the genome to or ENA with the run's methods text and the tool versions from the log.
Heterozygosity in the k-mer profile and duplication: both haplotypes are probably assembled. , or assemble the haplotypes separately.
The mitochondrial genome and small contigs from repeats such as the rDNA array or subtelomeric regions; the GC/depth plot and show which is which.
These docs are a map. This chapter points to the territory: the textbooks that teach each module properly, the order in which to read them, and the papers that define the current state of the art. Every entry is in the reference list at the end, with a link.
| Stage | Goal | Read |
|---|---|---|
| 1 · Foundations | What sequencing data are and how they are handled | Christensen, Introduction to Bioinformatics in Microbiology (ch. 1–3); Low & Tammi, Practical Bioinformatics for Beginners (ch. 3–4); Buffalo, Bioinformatics Data Skills (ch. 2–3, 10–11) |
| 2 · Core methods | Alignment, assembly, annotation, expression | Ismail, Bioinformatics: A Practical Guide to NGS Data Analysis (ch. 1–5); Baxevanis, Bader & Wishart, Bioinformatics (ch. 3, 5, 10) |
| 3 · Statistics | Models, testing and learning from many variables | James et al., An Introduction to Statistical Learning (ch. 2–6, 12–13); Faraway, Extending the Linear Model with R (ch. 5) |
| 4 · Your module | The specialist texts below | Xia, Sun & Chen and Xia & Sun for microbiomes; Christensen and Bergman for bacterial genomes; Ismail ch. 5 and Gentleman et al. for RNA-seq; Brock ch. 18 with Ismail ch. 3 for fungi |
| 5 · Research level | Why the methods work and where they fail | Efron & Hastie, Computer Age Statistical Inference (ch. 6–7, 15); Compeau & Pevzner, Bioinformatics Algorithms; Zvelebil & Baum, Understanding Bioinformatics; the papers below |
Books. Xia, Sun & Chen, Statistical Analysis of Microbiome Data with R (2018), is the statistician's text: power and sample size (ch. 5), diversity (ch. 6–9), compositional analysis (ch. 10) and models for over-dispersed and zero-inflated counts (ch. 11–12). Its successor, Xia & Sun, Bioinformatic and Statistical Analysis of Microbiome Data (2023), adds the path from raw reads (QIIME 2, DADA2, ASVs) to longitudinal mixed models. Zhong Wang's Introduction to Computational Metagenomics (2022) explains the algorithms of shotgun metagenomics; Mitra's Metagenomic Data Analysis (2023) gives protocols; Streit & Daniel's Metagenomics: Methods and Protocols and Izard & Rivera's Metagenomics for Microbiology cover the laboratory side; Charles, Liles & Sessitsch's Functional Metagenomics covers function-based screening; Paradis's Analysis of Phylogenetics and Evolution with R covers trees.
The state of the art. Microbiome data are compositional and must be analysed as such (Gloor et al. 2017). Exact sequence variants replaced OTUs (Callahan et al. 2016, 2017). For shotgun data, MetaPhlAn 4 extended marker-based profiling to unnamed species (Blanco-Míguez et al. 2023), and HUMAnN 3 profiles function (Beghini et al. 2021). Diversity should be compared at equal sample coverage (Chao & Jost 2012); whether to rarefy is still debated, with recent evidence in favour of repeated rarefaction (Schloss 2024). For differential abundance, a benchmark of 14 methods on 38 datasets found that they disagree widely, and recommended consensus across methods (Nearing et al. 2022); bias-corrected methods such as ANCOM-BC2 (Lin & Peddada 2024) and linear models such as MaAsLin 2 (Mallick et al. 2021) are current standards. Networks use compositionally aware estimators (Kurtz et al. 2015). The STORMS checklist sets the reporting standard (Mirzayi et al. 2021).
Books. Christensen's chapters on assembly and annotation (ch. 2), identification (ch. 7) and typing (ch. 11); Ismail on quality control, mapping, assembly and variants (ch. 1–4); Low & Tammi on comparing assemblies (ch. 6); Bergman's Comparative Genomics for whole-genome comparison.
The state of the art. With Nanopore R10.4 chemistry, long reads alone now give near-finished bacterial genomes (Sereika et al. 2022); consensus assembly from several assemblers improves reliability further (Wick et al. 2021), and short-read polishing that respects repeats remains useful (Wick & Holt 2022, ). Deep-learning variant callers outperform classical ones on bacterial Nanopore data (Hall et al. 2024). Annotation moved to fast, database-rich tools such as (Schwengers et al. 2021); completeness and contamination to machine learning (; Chklovski et al. 2023); taxonomy to the Genome Taxonomy Database and (Parks et al. 2022; Chaumeil et al. 2022) with fast ANI (Jain et al. 2018; Shaw & Yu 2023). Pangenomes are built with error-aware tools such as Panaroo (Tonkin-Hill et al. 2020), and resistance genes found with curated databases such as (Feldgarden et al. 2021).
Books. Ismail's RNA-seq chapter (ch. 5) covers quantification, normalisation and differential expression with worked commands; Christensen (ch. 10) and Low & Tammi (ch. 8) give shorter introductions; Baxevanis et al. (ch. 10) cover design, batch effects and exploration; Gentleman et al. explain and multiple testing; Efron & Hastie (ch. 15) the false discovery rate and empirical null; Compeau & Pevzner (vol. II, ch. 8) and James et al. (ch. 12) clustering.
The state of the art. A community survey still frames good practice (Conesa et al. 2016). Lightweight, bias-aware quantification (; Patro et al. 2017) imported at gene level (Soneson et al. 2015) is standard. (Love et al. 2014) with heavy-tailed fold-change shrinkage (Zhu et al. 2019) and - (Law et al. 2014) remain the reference methods; with large human cohorts, however, they can produce excess false positives and rank tests are safer (Li et al. 2022). At least six biological replicates per group are recommended (Schurch et al. 2016). Batch effects in counts can be adjusted with (Zhang et al. 2020). Gene-set analysis uses fast pre-ranked (Korotkevich et al. 2021) on curated sets such as the Hallmarks (Liberzon et al. 2015), and co-expression networks follow (Langfelder & Horvath 2008).
Books. Brock Biology of Microorganisms (ch. 18) for fungal biology; Compeau & Pevzner for yeast genome duplication; Ismail (ch. 3) and Low & Tammi (ch. 6) for assembly; Baxevanis et al. (ch. 5) and Zvelebil & Baum (ch. 9–10) for gene prediction.
The state of the art. Fungal genomics is organised around large sequencing programmes such as the 1000 Fungal Genomes project and the portal (Grigoriev et al. 2014); Stajich (2017) reviews what genomes revealed about the kingdom. Genome profiling uses k-mers (; Ranallo-Benavidez et al. 2020); assembly uses long reads with (Kolmogorov et al. 2019) or, for HiFi data, (Cheng et al. 2021). Completeness is measured with (Manni et al. 2021). Repeats are modelled with (Flynn et al. 2020); gene prediction now combines RNA-seq and protein evidence (; Gabriel et al. 2024). Function is annotated with for CAZymes (Zheng et al. 2023), 7 for secondary metabolism (Blin et al. 2023), 6 for secretion (Teufel et al. 2022) and 3 for effectors (Sperschneider & Dodds 2022). The evolution of wood decay was reconstructed from CAZyme repertoires across 31 genomes (Floudas et al. 2012).
Christensen (ch. 1–3 and 8) for foundations, then Xia & Sun (2023) for the path from reads to tables and tests, then Xia, Sun & Chen (2018) for the statistics in depth.
Method papers are written by the developers, on data and settings where the method works well. Independent benchmarks test many methods on the same, varied datasets.
This manual is the place to look when you are stuck. Part I–VII of the handbook explain the science; this part explains the website: where your data goes, how a run is started and followed, where the results are, how to read them and how to take them home. Each tool has its own chapter (M2–M8) with the same structure, so once you know one you can find your way in all of them.
The top bar is the same on every page. Tools opens the seven workbenches; each opens inside the site, so the top bar, your account and the status bar stay with you. Docs is this handbook, My data holds everything you uploaded and every result, and your picture opens Account. On a phone the same entries sit in the tab bar at the bottom.
| Your data | Your question | Tool | Manual |
|---|---|---|---|
| Feature or count tables (QIIME 2, DADA2, MetaPhlAn, /Bracken, HUMAnN, PICRUSt2) | Who is there, how communities differ, which taxa or functions change | MicroRes | M2 |
| Reads of one bacterial isolate | Assemble, check and annotate a bacterial genome | BactFlow | M3 |
| RNA-seq FASTQ files or a gene count table | Which genes and pathways change between conditions | RNA-realm | M4 |
| Reads of one fungal isolate (Nanopore, HiFi, Illumina) | A complete, annotated fungal genome; species; mitochondrion; food-safety genes | FungFlow | M5 |
| Human genome, exome or panel reads, BAM/CRAM or VCF | Variants, their interpretation, families, tumour–normal | Humaniac | M6 |
| Insect reads, or many individuals on one reference | A chromosome-level insect genome, or population structure | Insectra | M7 |
| Reads or an assembly of a bacterial strain used in food or feed | Is the strain safe? A dossier along EFSA's requirements | Bactomaniac | M8 |
Without an account you can look at every tool, open the example and demo results and read the handbook. Uploading data, starting runs, downloading reports and asking our AI need a free account: press Sign in or Create a free account in the top bar. Pricing lists the plans; a plan sets how many CPUs and how much memory your runs may use at once and how much storage you have. Your current plan and usage are under Account → Plan & usage.
In Account you can also change your name, affiliation and picture (they appear on your reports), your password, and whether you receive an e-mail when a run ends. Your data → Download my data gives you everything the site stores about you; Delete account removes the account and all its files and cannot be undone.
Every tool has a Data page (in the left menu of the tool) with a Choose files button. Files go up in 8 MB pieces, so files of many gigabytes are fine; a progress bar shows each file and a Cancel button stops it. You can leave the page open while it uploads; closing the tab stops the upload.
_R1/_R2 (or _1/_2) so the pairs are found..fa, .fasta, .fna), with GFF3 or files where a tool asks for them.Administrators see Use files already on the server: paste full paths, one per line (a folder or a pattern such as /data/run7/*.fastq.gz works too). The files are linked, not copied, and removing them from Data never deletes the original.
Heavy work runs on the server's queue (), not in your browser. When you press Start, every step of the run becomes its own job; you can close the page, and the run goes on. The status bar under the top bar shows what is running for you on any page, with its progress; click a run there to open its live page. When a run ends you get an e-mail (switch it off in Account).
Your plan's CPUs and memory are shared by your running steps: when they are all in use, further steps wait in the queue until one finishes. Steps that run out of memory are retried with more, up to your plan's limit.
FungFlow, Insectra, Humaniac and Bactomaniac are built the same way, and RNA-realm and MicroRes follow the same ideas:
Figures are interactive: hover for values, drag to zoom, double-click to reset, click a legend entry to hide a series, and use the camera icon in the figure's corner to save it as an image (SVG, which stays sharp at any size). Tables can be searched with the box above them (every column is searched), paged with ‹ and ›, and downloaded as CSV with the CSV button; numbers are right-aligned. When several runs are compared, the rows where the runs differ are highlighted. Very large result tables also come as files in Download ▾.
My data (top bar) lists, per tool, everything you uploaded and every result the tools wrote, with the storage you use. Download a single file, or a whole folder as a ZIP, or delete what you no longer need. MicroRes keeps your loaded datasets and saved analysis there too.
The Validation tab in the top menu (and the link at the foot of every page) shows how FungFlow and Bactomaniac perform on public genomes with a known biology: strains made, strains correct, each acceptance criterion with its measured value, and its status (met, missed or not run yet). Three buttons follow: Open the full report (a printable page with every strain and check), Download the evidence pack (accessions, expected results, measured results and software versions, as files) and Validate on your own strains (opens Contact with the request filled in). The address is /validation. Each tool's own manual has a section "Validation tests for the pipeline".
The ✦ Ask our AI button is on every page. It answers questions about using the site, about what is on your screen and your own runs, and about the bioinformatics behind them, from this handbook first and with references.
It reads only your own runs and cannot start, stop or delete anything. Answers can be wrong: check the numbers in your report.
MicroRes analyses microbiome tables in the browser: you load counts or profiles, clean and normalise them, then explore diversity, test differences and build a report. Nothing is queued: each step computes when you press its button, and every result adds a methods paragraph to the report. The science is in chapters 4–13.
The left menu has two modules that fold open: Taxonomic analysis in four phases (Data: Import, Quality control, Contaminants, Normalise & transform; Explore: Composition, Alpha diversity, Beta diversity, Phylogeny, Core microbiome; Test: Differential abundance, Associations, Co-occurrence networks; Predict: Biomarkers & prediction) and Functional metagenomics (F1 Import functions to F8 Antimicrobial resistance) and Viral metagenomics (V1 Import viral data to V7 Clinical report). Below them are Report, Guide & walkthrough and About. The CPU and RAM wheels in the menu show what MicroRes uses; click them to change the cores and memory budget.
Every step has a control deck on the left (settings, grouped and foldable; settings that apply to one choice appear only when it is chosen) and the results on the right. Once a result is drawn, changing a setting that feeds it redraws it. The How to read the results button at the top of each step explains every output; the ? on a card explains that card.
Open Import and choose the format under What are you loading?. Each format shows its own file boxes and a note:
.tsv, .txt, .csv or .xlsx. Say which column holds the IDs, whether there is a header row, and whether the taxonomy is one lineage string or one column per rank.--report, not the per-read output), Bracken outputs, per-sample taxonomy tables or a data matrix. Profiles of any number of samples are merged on the full lineage; a taxon missing from a sample becomes 0. A one-sample file is named after the file up to the first _.The Demo data ▾ menu loads a simulated diet study with known effects, so you can check that each method finds them; the demo input files can be downloaded to see the expected formats. The orange Clean session button in the top bar (after a confirmation) clears everything loaded and computed, including the reference, so that you can load new data; for signed-in users it also deletes the saved analysis, while uploaded files and downloads stay untouched.
Quality control removes features and samples that add noise: chloroplasts, mitochondria and unassigned features, chosen lineages, shallow samples, and features that are rare (prevalence and mean-abundance filters). Every change updates a live preview: the filter funnel (features and share of reads left after each filter), the prevalence–abundance map (removed features should sit low and to the left), sample checks (depth against Good's coverage; flagged samples are diamonds) and the list of removed features with the reason. Nothing changes until you press Apply quality control; Use all data keeps the data as it is. A single sample can be cleaned too: the filters only refuse when they would remove every sample or every feature. If your table holds relative abundances (every sample sums to 1 or 100) and the filters remove features, a Rescale switch appears: on, every sample is scaled back to its original total (100 % or 1) after the removal; off (the default), the remaining abundances are left as they are and no longer add up to the total.
Contaminants (decontam) finds reagent and kit contaminants from negative controls or DNA concentrations. Use it for low-biomass samples when you have controls or concentrations; skip it otherwise. Undo restores the data.
Sequencing depth differs between samples for technical reasons, so this step is required before any result: first a normalisation (correcting depth), then a transformation (changing the scale for the analysis). Choose the normalisation from your data and the transformation from the analysis you plan; Compare all transformations shows how each behaves on your data. The science is in chapter 7.
The functional module repeats the pattern for HUMAnN, PICRUSt2 or profiles: import (F1), clean and normalise (F2), then profiles, beta diversity, differential functions, who carries each function, and species–function networks.
As soon as your data are loaded, MicroRes asks how you want to continue. I'll analyse it myself takes you to quality control and every step with your own settings. Analyse it for me asks one question: is the study descriptive (describe the sample or samples) or comparative? For a comparison, choose the variable of interest (for example Diet), tick the groups to compare and the reference group they are compared with. Two more choices follow. Covariates are the other columns that could change the result, such as sex, age, BMI or batch: tick the ones to adjust for and the PERMANOVA, the association tests, differential abundance, functional beta diversity and differential functions include them (likely ones are ticked for you). A random effect is for repeated measures, when the same subject, patient or cage was sampled more than once: choose it and the alpha-diversity and functional models use a random intercept for it, so the repeated samples are not treated as independent. MicroRes then runs every suitable analysis with sound defaults (quality control, normalisation, composition, diversity, core, pangenome, differential abundance of each group against the reference with five methods, associations, network, prediction, and the functional steps when a profile is loaded, ending with the antimicrobial resistance screen) and tells you when the report is ready. Every result stays in its step, so you can open any of them, change a setting and run it again. The Analyse it for me button in the top bar brings the choice back. While it runs, Stop the analysis cancels it: the analyses that had finished are kept (and the report covers them), the rest are not run.
Display. Every figure follows the theme (Light, Plasma, Dark: dark background and light text in the dark themes, also in the report preview; downloads and the PDF stay light) and the size of its card. Wherever you choose how many taxa or functions to draw, the slider has a number box beside it and goes up to the number in your data (the label says all when everything is selected); a figure too tall or too wide for its card then scrolls inside the card. Colour legends keep a short, fixed length. The circular pangenome zooms with the mouse wheel and pans by dragging. Every report starts with the logo of the module that made it.
Is the sample free of antimicrobial resistance? Step F8 reads the names and identifiers of the functions in the profile you imported, recognises the resistance determinants with a curated rule table (beta-lactamases and carbapenemases, vancomycin Van ligases, PBP2a, tetracycline, macrolide, aminoglycoside, quinolone, sulfonamide, trimethoprim, phenicol, colistin MCR, fosfomycin and others) and gives every sample a verdict: no determinant found, only intrinsic or generic functions, determinants of common classes, determinants of clinical concern, or determinants against last-resort drugs. The result goes down to the drug class, the mechanism, the KEGG resistance pathway map where one exists, and the gene, with the taxa that carry it when the profile has per-taxon strata. The report has its own chapter with the same layers.
Read the verdict with its limits. AMR-safe here means no acquired determinant was found above the detection floor (1 per million annotated functions by default) in this profile. It needs a profile with gene-level names: KEGG orthologs, annotations or a named table give the full view; enzyme classes and reactions show only enzymatic resistance, and the verdict then says it cannot confirm the absence; pathway tables and unnamed UniRef gene families cannot show resistance genes at all, and the step says so instead of calling the sample safe. A gene that is present is not a resistant organism, point mutations (gyrA, parC, rpoB) are invisible, and generic functions such as multidrug efflux pumps and chromosomal AmpC are listed as context only. Confirm any finding on assembled contigs with or RGI/.
The pangenome idea from bacterial genomics applied to your communities: everything found across the samples (the pangenome) is split into a core shared by (almost) all samples and an accessory part that varies (shell and the rare cloud). Accumulation curves show whether each new sample still brings new taxa or functions (an open pangenome) or not (closed). With a variable of interest you get one pangenome per group, their shared core and the taxa or functions found only in one group. With a stratified functional profile every main species gets its own pangenome of functions across the samples where it occurs, which shows how variable its strains are. With one sample, the functions are compared across the species of that sample.
Two tables count the taxa (or functions) in each partition: one per group of your variable, one per sample (how many of the sample's taxa are hard core, soft core, shell or cloud, and the accessory share, which is high for unusual communities). The circular pangenome shows presence and absence at a glance: the most abundant taxa sit around the circle, ordered from hard core to cloud, and each ring is one sample, with the rings of a group side by side (the coloured strip in the gap names the group). Choose how many taxa to draw (10–500) and whether to draw features, species, genera or families before you press Build pangenome. The Core microbiome chapter of the report has the same tables for genera.
If you have a reference population (a healthy cohort, a product specification, a published study), load it in the Taxonomy reference and Functional reference fields, each in one of two ways: a JSON file ({"key-map": {"species": [{"name": …, "min", "q_10", "q_25", "median", "mean", "q_75", "q_90", "max", "sd"}]}}, optionally with alpha-diversity statistics of the cohort) or a cohort table with taxa (or functions) in the rows and the samples of the cohort, for instance 100, in the columns; the statistics are then calculated for you. Templates and a 50-sample example table can be downloaded from the step. Without a reference, Build your own reference makes one from the samples you have loaded (all of them or one group); it is switched off while you load your own files, and the result can be downloaded and reused. Every sample is placed in the reference feature by feature: below the range, low, typical (Q1–Q3), high or above the range, with its percentile. A position strip per taxon shows the reference cohort as a mini density spectrum from its minimum to its 90th percentile with an arrow where your sample lands, and the figures show reference boxes with your samples or group medians on top, a percentile map and the share of features in each zone. Choose how many features to show, or all of them; every setting updates the results at once.
The module under Viral metagenomics in the left menu analyses a virome: the abundance of viral genomes (vOTUs) in each sample. It loads its data into the same slot as a taxonomic dataset, so steps 04 to 13 (normalise, alpha, beta, core, differential abundance, associations, networks, prediction) run on the vOTUs unchanged, and the module adds the steps a virome needs. Read the chapters on viruses and viromes (V1 to V3 of the handbook) first: they explain why each step is there.
Figures export as PNG, SVG, PDF or HTML and tables as CSV from the card's own buttons; networks also for Cytoscape. Download ▾ in the top bar holds your data at each stage (after QC, contaminant removal, normalisation, functional cleaning).
Report in the left menu shows the report in the page before you download it. It reads your results the way an analyst would, in clickable chapters: an Overview with a verdict per group, a signal board (diversity, butyrate producers, pathobionts, short-chain fatty acid and inflammatory functions and more, each marked favourable or unfavourable against the reference group) and the main findings ranked by strength; then Data & quality, Who is there (with ecological guilds), Diversity, Core microbiome, What differs, Links to measurements, Networks, Prediction, Functions & health (vitamins, SCFA, fibre and mucus use, protein fermentation, sulfide, inflammatory signals, bile acids, TMA, GABA, resistance), Who does what (each function traced to the microbes that carry it and linked to the taxa that changed), Against the reference and a Conclusion with precautions and recommendations. Chapters appear only for analyses that were run. Every chapter starts with why it matters; Who is there introduces the main players and portrays each group and sample, and Pangenome shows what is shared and what is individual. How we classified lists every health module and guild with its matching rule, the functions or genera it matched in your data, and the references, so you can check each assignment. Every sentence is filled from your numbers with fixed thresholds (effect sizes, agreement between methods, p-values), so the same data always give the same text. Download it as HTML (interactive figures), PDF (every chapter, A4) or Markdown; the methods text alone is under the preview.
RNA-realm goes from RNA-seq reads (or a count table) to differentially expressed genes, pathways and co-expression modules, in eight numbered steps in the left menu. Each step writes its methods paragraph with references. The science is in chapters 24–29.
The left menu lists Data (00 Reads → counts, 01 Import counts, 02 Quality control, 03 Filter & normalise) and Analyse (04 Explore samples, 05 Differential expression, 06 Pathway enrichment, 07 Co-expression, 08 Methods report). Each step has its controls on the left and the results on the right; Continue moves to the next step. The top bar shows the loaded dataset and Download ▾. Demo data ▾ loads a demo study (12 samples, Control vs Treated, two batches) and offers its files so you can see the expected formats.
From reads (step 00): choose the FASTQ files (sent in 8 MB pieces), the reference transcriptome (human , mouse , or your own transcript FASTA) and the threads, then Process reads. trims and quantifies each sample; a report summarises the read quality. Then Load results, with sample metadata if you have it. Processing reads needs an account.
From counts (step 01): choose the count file(s), the sample metadata (first column: sample names exactly as in the counts) and, for transcript-level files, a transcript-to-gene table, then Import.
Run QC shows library sizes, detected genes, count distributions and sample correlations, coloured by a metadata column of your choice. Look for a sample that sits apart from its group, or a library far smaller than the others.
Filter & normalise: choose the groups for filtering (the smallest group sets the minimum number of samples), the minimum count and total count per gene, the normalisation (TMM recommended; median of ratios, upper quartile or library size) and the transformation for plots and modules (VST recommended, or log2 CPM). See chapter 26.
A PCA of the most variable genes, coloured and shaped by metadata, any two components, with an option to remove a batch for display and to show sample labels; press Draw. Samples should cluster by condition; a separation by batch or sequencing date is a confounder to include in the model.
Choose Compare groups of (a metadata column), the Reference and Test groups, covariates (such as batch), the methods (, -, and Wilcoxon for 8 or more samples per group), the FDR and the minimum |log2 fold change|, and whether to shrink fold changes; press Test. The results show each method's table, volcano and MA plots, and the agreement between methods; the Gene viewer plots one gene across samples. Read chapter 27.
Pathway enrichment: choose the gene-set library, the DE results to use, (preranked) and/or over-representation, the set-size limits and the permutations, then Run enrichment; the enrichment plot for a term shows where its genes fall in the ranked list. Co-expression (-style): the most variable genes, the soft-threshold power (empty: chosen automatically) and the minimum module size, then Find modules. It works best with 15 or more samples; with fewer, treat modules as hypotheses.
Methods report (step 08): Write the methods, then Download (.md) for a methods section with references. Download ▾ in the top bar holds the filtered counts (with gene symbols), the normalised expression, every DE method's table for all genes, the enrichment tables, the co-expression modules (gene, module, kME) and the methods text. Every figure has a camera icon to save it.
Every number a Bioinformaniac pipeline gives you, a Shannon index, a fold change, an ANI, a QV, is an estimate made from a sample. This part of the handbook teaches the statistics behind those numbers, so that you can read every result of MicroRes and the other tools the way a statistician would: what was estimated, how uncertain it is, and what it can and cannot support.
Biology varies. Two mice on the same diet carry different gut communities; two cultures of the same strain grow at slightly different rates; two libraries made from the same DNA give slightly different counts. Statistics is the discipline of drawing conclusions despite this variation: separating a difference that reflects the biology from one that the sampling alone could produce. Holmes and Huber put it plainly in the opening of Modern Statistics for Modern Biology: modern biology generates data whose variability is part of the signal, and the analysis has to model it rather than wish it away (Holmes & Huber, 2019, ch. 1).
In sequencing studies three layers of variation stack on top of each other. Biological variation is real difference between individuals, plants, cultures or time points: it is what we want to study. Technical variation comes from extraction, library preparation, sequencing depth and batches: it is noise, and sometimes bias. Sampling variation arises because a sequencer reads a random subset of the molecules in a library: a taxon at 0.01 % gives 2 reads in one library of 20,000 and 0 in another. Good design (chapter 2) controls the first two; the statistics in this part quantify all three.
The population is everything you want to make a statement about: all healthy adults, all batches of a yoghurt, all isolates of a lineage. A parameter is a number that describes it, such as the true mean relative abundance of Faecalibacterium in healthy adults. You never see the parameter. You see a sample, and compute an estimate, a statistic such as the sample mean. Writing μ (mu) for the parameter and x̄ ("x-bar") for the estimate keeps the two apart. Every downstream question, a test, an interval, a model, is about how far x̄ may lie from μ.
The sample has to represent the population for this to work. Twenty stool samples from one family represent that family, not "healthy adults". This is the single most common reason a statistically perfect analysis gives a wrong biological answer, and no method in this part repairs it (Whitlock & Schluter, The Analysis of Biological Data, ch. 1).
A distribution describes how likely each value is. Four families cover most of what the tools compute.
| Distribution | What it describes | Where you meet it |
|---|---|---|
| Normal (Gaussian) | sums of many small effects; symmetric, mean μ, standard deviation σ | means, log fold changes, CLR values, regression errors |
| Binomial | successes out of n independent trials with probability p | completeness, allele balance in a heterozygote, accuracy of a classifier |
| Poisson | counts of rare independent events, variance = mean | reads covering a base, technical replicates of a library |
| Negative binomial | counts with extra (biological) variance, σ² = μ + μ²/θ | taxa and gene counts across biological replicates (, ) |
Relative abundances, P values and proportions live on [0, 1]; counts are whole numbers ≥ 0; ANI sits near 100 %. Recognising the kind of number in front of you tells you which methods are allowed: a t-test on raw counts, or a Pearson correlation on proportions, ignores what the numbers are (chapters S4 and S7).
The standard deviation (SD, s) describes how much single observations vary. The standard error (SE) describes how much an estimate would vary if you repeated the whole study. For a mean of n independent observations:
SE(x̄) = s / √n
Four times more samples halves the uncertainty of a mean; it does not halve the spread of individuals. Confusing SD and SE is one of the commonest errors in figures: error bars must say which they show (Krzywinski & Altman, 2013). The central limit theorem says that means (and sums) of independent values are close to normally distributed even when single values are not, provided n is not tiny. Figure S1.1 shows it on a skewed variable typical of taxon abundances.
A 95 % confidence interval (CI) is a range built by a procedure that, in repeated studies, would contain the true parameter 95 % of the time. For a mean, x̄ ± t0.975, n−1·SE. It is not "a 95 % probability that μ is in this particular interval": μ is fixed, the interval is random. The practical reading is simpler: the interval shows which values of the parameter are compatible with your data, and its width shows how much the data can tell you. A narrow interval around a tiny effect is a precise "almost nothing"; a wide interval that spans zero is "we cannot tell", not "no effect".
The bootstrap builds intervals when no formula exists, for a median, a diversity index, a ratio of two estimates or an AUC (Efron, 1979; Efron & Hastie, 2016, ch. 10). Draw n samples with replacement from your n samples, recompute the statistic, repeat thousands of times; the spread of the recomputed values approximates the sampling distribution, and its 2.5th and 97.5th percentiles give a 95 % interval (right panel of Figure S1.1). The bootstrap resamples the independent units: whole subjects, not the reads inside a sample and not the repeated visits of one subject treated as separate people.
One. The replicates measure technical noise of that sample; they do not add independent biological information.
They show the standard error s/√n, the uncertainty of the mean, not the standard deviation of individuals.
A test answers one narrow question: could chance alone, as defined by a null hypothesis, have produced a difference at least this large? This chapter explains what a P value is and is not, which test fits which data, and why every P value in a report should travel with an effect size and its uncertainty.
Every test has the same four parts. A null hypothesis H₀ states "no difference" in a precise way, for example "the mean Shannon index is the same in both diets". A test statistic summarises the difference in your data, for example t = (x̄₁ − x̄₂)/SE. A null distribution says how that statistic would vary if H₀ were true. The P value is the probability, under H₀, of a statistic at least as extreme as the one observed.
A small P value says the data would be surprising if H₀ were true. It does not give the probability that H₀ is true, it does not measure the size or importance of an effect, and P > 0.05 does not show that there is no effect. These are the core points of the American Statistical Association's statement on P values (Wasserstein & Lazar, 2016). P values are also fickle: repeat a modest study and P can easily move from 0.01 to 0.3 (Halsey et al., 2015).
| H₀ true (no effect) | H₀ false (real effect) | |
|---|---|---|
| Test significant | Type I error (false positive), rate α | correct: a discovery (rate = power) |
| Not significant | correct | Type II error (false negative), rate β |
Power = 1 − β is the probability of detecting an effect of a given size. It grows with the effect size, the sample size and α, and falls with the noise. Figure S2.1 computes it exactly for a two-group t-test: with α = 0.05, 80 % power needs about 64 samples per group for a medium effect (d = 0.5) but only 26 for a large one (d = 0.8). Microbiome effects are often small, which is why many published microbiome studies are underpowered (Kelly et al., 2015); RNA-seq experiments need at least six biological replicates per group to find most changes reliably (Schurch et al., 2016).
| Data and question | Parametric test | Rank-based / resampling |
|---|---|---|
| Two independent groups, a continuous measure | Welch t-test | Wilcoxon rank-sum (Mann–Whitney U) |
| Paired measurements (before/after, same subject) | paired t-test | Wilcoxon signed-rank |
| Three or more groups | one-way ANOVA (Welch) | Kruskal–Wallis, then Dunn's test with adjustment |
| Two categorical variables (e.g. gene present × group) | χ² test | Fisher's exact test |
| Any statistic, any design you can permute | permutation test: shuffle the labels, recompute, compare | |
The Welch t-test does not assume equal variances and should be the default two-group t-test. The Wilcoxon rank-sum test replaces values by ranks; it is robust to outliers and skew, and tests whether values in one group tend to be larger than in the other (it is not, strictly, a test of medians). Kruskal–Wallis generalises it to several groups and only says that some group differs; a post-hoc test (Dunn's) says which pairs, with a multiple-testing adjustment. A permutation test builds the null distribution by shuffling the group labels many times: it needs only that samples are exchangeable under H₀, and it is how PERMANOVA (chapter S5) and many network and machine-learning checks obtain their P values (Good, 2005; Holmes & Huber, 2019, ch. 6).
With enough samples any trivial difference becomes "significant"; with too few, an important one does not. An effect size answers the question that matters to the biologist.
The habit to build: report the estimate, its uncertainty and the P value together, and judge importance from the first two (Sullivan & Feinn, 2012; Gelman, Hill & Vehtari, 2020, ch. 4).
A P value conditions on H₀; a decision often needs the reverse, the probability that a finding is real. Bayes' rule gives it from the prior probability. A pathogen marker with 95 % sensitivity and 95 % specificity, used where only 1 % of samples truly carry the pathogen, gives a positive predictive value of 0.95 × 0.01 / (0.95 × 0.01 + 0.05 × 0.99) ≈ 16 %: most positives are false. The same logic explains why many "significant" findings from small exploratory studies do not replicate (Ioannidis, 2005), and it is the basis of genotype calling (chapter S8). Bayesian models make the prior explicit and give posterior intervals for the quantities of interest (Gelman et al., 2013).
No. Absence of evidence is not evidence of absence; look at the confidence interval of the difference to see which effects remain compatible with the data.
It does not assume equal variances in the two groups, and loses almost nothing when they are equal.
A differential abundance analysis tests every taxon; an RNA-seq study tests every gene. With thousands of tests, a 5 % false-positive rate per test means hundreds of false discoveries. This chapter explains the adjustments every omics result relies on, and how to read a q value and a volcano plot.
Test 5,000 taxa that do not differ at all, each at α = 0.05, and about 250 will be "significant" by chance. The probability of at least one false positive among m independent tests is 1 − (1 − α)m, which is already 64 % for m = 20. Two kinds of control answer this, and they answer different questions.
The family-wise error rate (FWER) is the probability of making even one false discovery. Bonferroni controls it by testing each hypothesis at α/m; Holm's step-down version is uniformly more powerful and should replace it. FWER control is right when a single false claim is costly: a regulatory safety claim, a clinical variant, a small set of planned comparisons such as Dunn's pairwise tests.
The false discovery rate (FDR) is the expected proportion of false discoveries among the discoveries. Controlling it at 5 % accepts that about one in twenty listed taxa or genes may be a false lead, in exchange for far more power. For screening thousands of features it is the standard (Benjamini & Hochberg, 1995).
Sort the m P values: p(1) ≤ p(2) ≤ … ≤ p(m). Find the largest rank k with
p(k) ≤ (k / m) · q
and declare the k smallest P values significant. Equivalently, each test gets an adjusted P value (often called a q value in software): padj,(i) = min over j ≥ i of (m · p(j) / j), capped at 1; keep the features with padj ≤ 0.05. Storey's q value refines this by estimating π₀, the proportion of true nulls, from the flat part of the P-value histogram, and gains power when many features truly change (Storey & Tibshirani, 2003).
Always look at it before trusting the list of hits (Holmes & Huber, 2019, ch. 6). A flat histogram with a spike at zero (Figure S3.1) is healthy. A histogram with no spike means no detectable signal. A hump in the middle, or a pile-up near 1, means the test's assumptions are violated (wrong model, discrete counts, too few samples for the permutations), and adjusted P values are then not trustworthy either.
A volcano plot (Figure S3.2) places each feature by effect (log₂ fold change) and evidence (−log₁₀ P). The interesting features sit high and far from zero. Points far out but low are features with large but uncertain estimates, typically rare taxa with few reads: that is why and RNA-realm report shrunken fold changes, which pull noisy estimates towards zero in proportion to their uncertainty and so rank genes more reliably (Love, Huber & Anders, 2014; Zhu, Ibrahim & Love, 2019).
Every feature you test costs power. Removing features that could never be significant before testing, using a criterion that does not look at the groups, such as overall prevalence or mean abundance, raises power without biasing the FDR. This is independent filtering, done automatically by and by the prevalence filter of MicroRes (Bourgon, Gentleman & Huber, 2010). Filtering by the observed difference between groups is not independent and invalidates the adjustment.
About 10 (5 % of 200) on average; the FDR is a rate over the list, not a property of each gene.
When even one false positive is costly and the number of tests is small, e.g. a handful of planned comparisons or a safety claim.
Sequencing gives counts, and the counts of one sample add up to a total set by the machine, not by the biology. These two facts, overdispersed counts and compositions, explain why microbiome and RNA-seq data need their own statistics, and why MicroRes asks you to normalise and transform before anything else.
If a library were sequenced again and again, the count of one taxon would vary like a Poisson variable: variance equal to the mean. Across biological replicates the variance is much larger, because the taxon's true abundance itself differs between individuals. The negative binomial (NB) captures this with a dispersion parameter: σ² = μ + αμ² (α = 1/θ). For abundant features the αμ² term dominates (Figure S4.1, left). and model each feature as NB and share information across features to estimate α reliably from few replicates (Robinson, McCarthy & Smyth, 2010; Love, Huber & Anders, 2014). Holmes and Huber derive the NB as a Poisson whose rate is itself gamma-distributed, a "mixture" that is the natural model of heterogeneous biology (Holmes & Huber, 2019, ch. 4 and 8).
Most entries of a microbiome table are zero. A zero can mean the taxon is absent (biological zero), present but not sampled at this depth (sampling zero), or removed by a technical step. Sampling zeros are expected from the NB itself at low means; models that add a separate "zero inflation" component are needed only when there are more zeros than the count model predicts (Silverman et al., 2020). Practically: never read a zero as proof of absence, filter features that are zero almost everywhere (they cannot be tested), and be careful with any method that takes a logarithm (log 0 is undefined, so a pseudocount is added, and its size changes the results for rare taxa).
The total number of reads per sample is fixed by the sequencing run, so the counts only carry relative information: the data are compositions (Aitchison, 1982; Gloor et al., 2017). If one taxon blooms, the shares of all others must fall even if their absolute amounts did not change (Figure S4.1, right). Three consequences follow. Correlations between shares are biased, often negative (chapter S7). A taxon "decreasing" in relative abundance may not have decreased at all. And differences in total microbial load, which can be large between people or treatments, are invisible unless measured separately (qPCR, flow cytometry, spike-ins) (Vandeputte et al., 2017).
The remedy is to work with log-ratios, which do not depend on the total. The centred log-ratio (CLR) divides each count by the geometric mean of the sample:
clr(x)i = log( xi / g(x) ), g(x) = (x1·x2·…·xD)1/D
CLR values are centred on zero within each sample: positive means more abundant than the sample's typical taxon. The Euclidean distance between CLR vectors is the Aitchison distance, the natural distance between compositions. A difference in CLR between groups is a difference relative to the typical taxon, which is the honest statement compositional data allow.
| Method | Idea | Good for | Watch out |
|---|---|---|---|
| TSS (proportions) | divide by the sample total | composition plots, Bray–Curtis | dominated by abundant taxa; compositional |
| Rarefying | subsample every sample to the same depth | richness and other depth-sensitive diversity | throws data away; random; not for differential abundance (McMurdie & Holmes, 2014) |
| CSS | scale by the sum up to a data-driven quantile | sparse marker-gene data | designed for metagenomeSeq (Paulson et al., 2013) |
| TMM | trimmed mean of M-values against a reference sample | RNA-seq-like counts | assumes most features do not change (Robinson & Oshlack, 2010) |
| Median of ratios () | size factor = median ratio to a pseudo-reference | NB models | needs features present in all samples; struggles with very sparse tables |
| VST / rlog | variance-stabilising transformation of NB counts | ordination, clustering, heatmaps | not for testing; testing uses the model |
| CLR | log-ratio to the geometric mean | Aitchison distance, correlations, ML, CLR-based tests | pseudocount choice affects rare taxa |
No normalisation is right for everything, and benchmarks show that the choice changes results, especially for differential abundance (Weiss et al., 2017). MicroRes's Compare all transformations panel shows how each behaves on your data; the methods paragraph of the report records which one you used, because a reader cannot judge a result without it.
The true abundance itself differs between replicates, adding biological variance on top of sampling (Poisson) variance: overdispersion.
The taxon is e² ≈ 7.4 times less abundant than the geometric mean of that sample's taxa (natural log).
Diversity indices are estimates, distances are modelling choices, and an ordination is a projection. This chapter treats the ecological results of MicroRes as statistics: what each number estimates, how depth biases it, and how PERMANOVA and PERMDISP test differences between communities.
Observed richness counts the taxa seen, and the number seen grows with the number of reads (Figure S5.1, left). Comparing raw richness between samples of different depth compares sequencing effort as much as biology. Two remedies exist: compare at equal effort (rarefying, or reading the rarefaction curves at a common depth) or estimate the true richness including unseen taxa. Chao1 uses the taxa seen once (F₁, singletons) and twice (F₂, doubletons):
ŜChao1 = Sobs + F₁(F₁ − 1) / (2(F₂ + 1))
Many singletons relative to doubletons mean many taxa remain unseen (Chao, 1984). With denoised amplicon data singletons are often removed by the denoiser, which makes Chao1 unreliable: report it only when singletons are genuine. Willis argues that diversity should be estimated with a standard error and compared with models that use it, rather than computed from rarefied tables (Willis, 2019).
Shannon (H = −Σ pi ln pi) and Simpson (Σ pi²) weight abundant taxa more and depend less on depth. Hill numbers put them on one scale, the effective number of equally abundant taxa: q = 0 is richness, q = 1 is eH, q = 2 the inverse Simpson (Hill, 1973; Chao et al., 2014). A sample with Shannon H = 2.3 has the diversity of e2.3 ≈ 10 equally common taxa: a number a reader can picture.
An index per sample becomes an ordinary variable: compare two groups with Wilcoxon or a t-test, several with Kruskal–Wallis and Dunn (chapter S2), and adjust for covariates or repeated measures with a linear or mixed model (chapter S6). Diversity is often skewed and sometimes depends on depth even after rarefying; including log(depth) as a covariate is a simple check.
| Distance | Uses | Sensitive to |
|---|---|---|
| Jaccard | presence/absence | rare taxa, membership |
| Bray–Curtis | relative abundances | dominant taxa |
| Unweighted UniFrac | presence/absence + tree | rare lineages, deep differences (Lozupone & Knight, 2005) |
| Weighted UniFrac | abundances + tree | abundant lineages |
| Aitchison | CLR values | ratios; compositionally coherent |
Different distances answer different questions, so a difference seen with Jaccard but not with Bray–Curtis says the groups differ in which rare taxa they carry rather than in their dominant members. Choose the distance for the question before testing, or report all and interpret the pattern.
PCoA (principal coordinates) places samples in a few dimensions that best preserve the chosen distances, by an eigen-decomposition of the double-centred distance matrix; with Euclidean distances on CLR values it equals PCA. The percentage on each axis is the share of the total variation that axis shows. Two axes showing 31 % and 12 % (Figure S5.1, right) leave more than half the variation out of the picture, so a lack of visible separation is not evidence of no difference. NMDS preserves only the rank order of distances; its stress (below about 0.2) says how faithful the picture is. Ordinations are for looking; tests are for deciding (Legendre & Legendre, 2012, ch. 9; Holmes & Huber, 2019, ch. 9).
PERMANOVA partitions the sum of squared distances into a part between groups and a part within, exactly as ANOVA does with variances, giving a pseudo-F statistic and R², the share of variation explained by the variable. Its P value comes from permuting the group labels (Anderson, 2001). Because it uses distances, it works with any measure above. Covariates enter in order (sequential) or by margin; repeated measures need restricted permutations within subject.
PERMANOVA can be significant because group centroids differ or because groups differ in spread. PERMDISP tests the second: it compares the distances of samples to their group centroid (Anderson, 2006). Read them together: PERMANOVA significant and PERMDISP not → a shift in composition; both significant → the shift may be partly or wholly a difference in heterogeneity, which is itself a finding (dysbiotic communities are often more variable) but a different one (Anderson & Walsh, 2013).
Not necessarily; at equal depth B may be as rich or richer. Compare at a common depth or with an estimator that accounts for unseen taxa.
The groups differ, but at least partly in dispersion; a shift in average composition is not established.
A two-group test answers a two-group question. Real studies have ages, batches, antibiotics, repeated visits and several variables of interest at once. Regression models handle them, and every differential abundance and differential expression method in our tools is a regression model underneath.
A linear model writes the outcome as a sum of effects plus noise:
y = β₀ + β₁·diet + β₂·age + β₃·batch + ε
Each coefficient β is the expected change in y for a one-unit change in that variable with the others held fixed. That last phrase is the whole point: the model separates effects that travel together in the data. Group comparisons are linear models with indicator variables (the t-test and ANOVA are special cases). Assumptions to check: independent errors, roughly constant variance, and residuals without strong patterns (Gelman, Hill & Vehtari, 2020, ch. 6–11).
A generalised linear model keeps the linear predictor but links it to a mean that suits the data: a log link and a negative binomial distribution for counts, a logit link and a binomial for presence/absence or cases/controls. Sequencing depth enters as an offset (log size factor), so the model compares rates, not raw counts. and are NB GLMs fitted to every feature (McCarthy, Chen & Smyth, 2012; Love, Huber & Anders, 2014).
A confounder is associated with both the variable of interest and the outcome. If cases were sequenced in one batch and controls in another, batch and disease are inseparable, and no model can fix it afterwards: that is a design failure (chapter 2). If they are only partly entangled, including the confounder as a covariate recovers the effect of interest (Figure S6.1). Antibiotics, proton-pump inhibitors, metformin, age, BMI, transit time and sequencing batch are frequent confounders in microbiome studies (Leek et al., 2010; Vujkovic-Cvijin et al., 2020). Batch-correction methods such as remove batch means and variances when batches are balanced with the biology (Johnson, Li & Rabinovic, 2007); with unbalanced designs, model batch as a covariate instead of "correcting" the data first.
Ten visits of one person are not ten people (chapter S1). A mixed model adds a random effect for subject, a subject-specific intercept drawn from a distribution, so that within-person correlation is modelled and the effective sample size is honest. The same applies to cages of mice (mice in one cage share microbes), fields, fermenters and families. In formula notation: abundance ~ diet + time + (1 | subject). MaAsLin2 and LinDA fit such models for microbiome features (Mallick et al., 2021; Zhou et al., 2022); for distances, PERMANOVA with permutations restricted within subject does the equivalent.
| Method | Model | What it estimates | Typical behaviour |
|---|---|---|---|
| | NB GLM with size factors, Wald test, shrinkage | log₂ fold change of normalised counts | powerful; FDR can inflate with very sparse, compositional data (Love et al., 2014) |
| ANCOM-BC | log-linear model with a bias term for sample-specific sampling fractions | log fold change of absolute abundance, up to a constant | good FDR control; corrects compositional bias (Lin & Peddada, 2020) |
| ALDEx2 | Dirichlet Monte-Carlo instances → CLR → Welch/Wilcoxon, averaged | difference in CLR (relative to the geometric mean) | conservative; robust to sparsity (Fernandes et al., 2014) |
| LinDA | linear (mixed) model on CLR with a bias correction | log fold change, compositionally corrected | fast, allows covariates and random effects (Zhou et al., 2022) |
| Wilcoxon on CLR | rank test per feature | a shift in CLR distribution | simple, assumption-light; no covariates |
On 38 datasets, popular methods identified very different sets of significant taxa from the same data (Nearing et al., 2022). Disagreement is not a bug in one method; it reflects different targets (counts, CLR, absolute fold change) and different assumptions. A taxon found by most methods is a robust finding; one found by a single method needs caution. That is exactly what the consensus heatmap of MicroRes shows.
LEfSe combines Kruskal–Wallis and Wilcoxon tests with a linear discriminant analysis effect size to rank biomarkers (Segata et al., 2011). It is useful for description, but it does not adjust for multiple testing in the usual way and does not model covariates, so treat its list as exploratory.
No. Run and disease status are perfectly confounded; their effects cannot be separated by any model.
They estimate different quantities (normalised count fold change vs CLR difference) under different assumptions; compositional effects and sparsity hit them differently.
Which taxa go together? Which features predict a disease? Correlation and prediction are where compositional data and small samples cause the most convincing false results. This chapter explains the methods behind MicroRes's Associations, Co-occurrence networks and Biomarkers & prediction steps, and how to tell a robust result from an artefact.
Pearson's r measures linear association and is sensitive to outliers; Spearman's ρ is Pearson on ranks and measures any monotonic association. Both range from −1 to +1. Correlation is not causation, and in microbiome data it is often not even association between amounts: two taxa can correlate because both respond to pH, because both are divided by the same total, or because of a batch (Altman & Krzywinski, 2015).
The left panel of Figure S7.1 shows the compositional trap: taxa simulated independently become correlated once divided by a shared, variable total. Pearson described this "spurious correlation" of ratios in 1897; Aitchison's log-ratio framework is the cure (Aitchison, 1982; Lovell et al., 2015). SparCC estimates correlations of the underlying log-abundances from the variances of log-ratios, assuming the network is sparse (Friedman & Alm, 2012). SPIEC-EASI goes further: from CLR data it estimates a sparse inverse covariance (graphical lasso or neighbourhood selection), so an edge means conditional dependence, two taxa associated after accounting for all others, and the sparsity is chosen by stability selection (Kurtz et al., 2015). Spearman on CLR values is a simpler, compositionally aware option.
Network edges are hypotheses about interaction, not evidence of it. With n samples and p taxa, there are p(p−1)/2 possible edges: 4,950 for 100 taxa. Edges need multiple-testing control or stability selection, and a network inferred from 30 samples is fragile; MicroRes reports the number of samples and the method next to every network.
Testing asks whether a feature differs; prediction asks whether a set of features can classify a new sample. Many features that differ significantly predict poorly (Figure S2.1: large effects still overlap), and a model can predict well from features that are individually unremarkable.
With more features than samples, the usual case in omics, a flexible model can fit the training data perfectly and still predict nothing (Figure S7.2). k-fold cross-validation estimates performance on new data: split the samples into k parts, train on k − 1, test on the held-out part, repeat, average. The penalty of LASSO or elastic net is chosen by an inner cross-validation; the performance must then be measured by an outer loop that never saw that choice (nested cross-validation), or the estimate is optimistic (Varma & Simon, 2006; James et al., 2021, ch. 5).
Data leakage is the classic error: selecting the "significant" taxa on all samples, then cross-validating a model on those taxa. The test folds influenced the selection, so the AUC is inflated, sometimes from 0.5 to 0.9 on pure noise. Every step that looks at the outcome, including feature selection, normalisation fitted to the data and parameter tuning, must happen inside the training folds.
Sensitivity is the share of true cases called positive; specificity the share of true controls called negative. Changing the threshold trades one for the other; the ROC curve draws all trade-offs, and the AUC (area under it) is the probability that a random case scores higher than a random control: 0.5 is chance, 1 perfect (Figure S7.1, right). AUC ignores prevalence; when cases are rare, also report positive predictive value (chapter S2) or the precision–recall curve. A model is only as general as its samples: an AUC from one cohort often falls in another, so external validation is the real test (Pasolli et al., 2016).
Not necessarily. Compositional closure, a shared driver or a batch can produce the correlation; check with SparCC/SPIEC-EASI and design experiments to test interaction.
Otherwise the held-out samples influenced which features were chosen, and the performance estimate is biased upwards (leakage).
Genome tools look deterministic, but nearly every number they print is a probability or an estimate: a Phred score, a QV, a coverage depth, an E-value, an ANI, a percentage, a genotype quality, a bootstrap support. Knowing the statistics behind them is what lets you read BactFlow, Bactomaniac, FungFlow, RNA-realm, Humaniac and Insectra results with confidence.
A Phred quality is a log-scaled error probability: Q = −10 log₁₀ P, so Q20 is 1 error in 100 bases, Q30 1 in 1,000, Q40 1 in 10,000 (Figure S8.1, left) (Ewing & Green, 1998). The same scale describes a whole assembly: a consensus QV of 40 means about one error per 10,000 bases, so about 500 errors in a 5 Mb bacterial genome; QV 50 brings that to 50. estimates QV without a reference, from the k-mers of the assembly that do not occur in accurate reads: with Kasm-only such k-mers among Ktotal, the per-base error rate is E = 1 − (1 − Kasm-only/Ktotal)1/k and QV = −10 log₁₀ E (Rhie et al., 2020). In the FungFlow validation the simulated-read assemblies reached QV 45–58 against their known truth.
If reads start at random positions, the number covering a base follows a Poisson distribution with mean c, the average coverage (Lander & Waterman, 1988). The fraction of the genome with zero coverage is then e−c: 5 % at 3×, 0.005 % at 10×. Real libraries deviate (GC bias, repeats, amplification), so tools report the observed distribution; a heterozygous site in a diploid needs enough reads on each allele, which is why variant callers ask for roughly 15–30× and why Humaniac reports the share of the genome above 20×.
A E-value is the number of alignments with at least this score expected by chance in a database of this size: E = K·m·n·e−λS, where S is the score and m, n are the query and database lengths (Karlin & Altschul, 1990). It grows with database size, so the same hit has a larger E in a bigger database. For small values E ≈ P; E = 1e−50 is overwhelming, E = 0.1 is noise.
Average nucleotide identity (ANI) between two genomes clusters into within-species and between-species groups, with a gap that makes 95 % a practical species boundary (Goris et al., 2007; Jain et al., 2018). It is an estimate over the aligned fraction (AF), so an ANI of 99 % over 20 % of the genome is not the same evidence as 99 % over 90 %; requires both an ANI and an alignment-fraction threshold. Values near the boundary (94–96 %) deserve the word "borderline", not a confident name. The Bactomaniac validation found that the remaining identity "errors" were nomenclature conflicts of exactly this kind ( placing B. cereus AH187 in B. paranthracis), not wrong measurements.
completeness is the proportion of a set of near-universal single-copy genes found complete; with the 758 genes of fungi_odb10, 97.0 % is 735 of 758, and the binomial standard error √(p(1−p)/n) ≈ 0.6 points. Differences of a few tenths of a percent between assemblies are within that noise; differences of several points are not. A high duplicated share signals uncollapsed haplotypes or contamination rather than better quality (Simão et al., 2015).
A variant caller computes, for each possible genotype G, the likelihood of the observed reads, P(reads | G), from base qualities and mapping qualities, and combines it with a prior by Bayes' rule (Li, 2011). In a VCF, QUAL is the Phred-scaled probability that the site is not variant, and GQ the Phred-scaled probability that the reported genotype is wrong: GQ 20 means a 1 % chance. For a true heterozygote the alternative-allele fraction is binomial around 0.5; a "heterozygous" call at 10 % alternative reads in a haploid bacterium or fungus points to contamination, a mixed culture or a collapsed repeat, not to heterozygosity. Clinical interpretation adds population frequencies and evidence categories on top (Richards et al., 2015).
A phylogeny is an estimate. Bootstrap support resamples alignment columns, rebuilds the tree and counts how often a clade reappears (Felsenstein, 1985); it measures the stability of the clade given the data, not the probability that it is true. Standard bootstrap values of 70 % or more are usually read as good support; 's ultrafast bootstrap is less biased and is usually taken as strong from 95 % (Minh, Nguyen & von Haeseler, 2013; Hoang et al., 2018).
Over-representation analysis tests whether a gene set is more frequent among the significant genes than expected, with a hypergeometric (Fisher) test; its answer depends entirely on the background (all genes tested, not the whole genome). Gene-set enrichment analysis () uses the full ranked list and avoids an arbitrary cut-off (Subramanian et al., 2005); competitive tests that account for correlation between genes, such as camera, keep the error rate honest (Wu & Smyth, 2012). Both need multiple-testing adjustment over the sets tested.
"100 % sensitivity" on 18 expected resistance genes is an estimate with an interval: the exact (Clopper–Pearson) 95 % lower bound for 18 of 18 is 81 %. Report the number of cases behind each figure, fix the pass marks before seeing the results, and separate errors of the pipeline from errors of the truth set. Bioinformaniac's validation pages follow these rules; they are described in the customer validation report.
No evidence of one: a hit that good is expected about once by chance in this search.
e⁻⁵ ≈ 0.7 % under the Poisson model; more in practice because of bias.
The previous chapters gave the tools; this one applies them. Step by step, it lists what each MicroRes result estimates, which statistics produced it, what to check before believing it, and how to write it up. The same checklist closes for the genome and RNA tools.
| Step | What it estimates | Statistics | Check before believing | Chapter |
|---|---|---|---|---|
| 02 Quality control | which features and samples carry information | prevalence and abundance filters (independent of groups) | the funnel: share of reads kept; no group lost more samples than another | S3, 6 |
| 03 Contaminants | reagent contaminants | decontam frequency/prevalence models | negative controls present and sequenced; low-biomass samples flagged | 6 |
| 04 Normalise & transform | comparable values across samples | TSS, rarefying, CSS, TMM, median-of-ratios, VST, CLR | "Compare all transformations"; the choice stated in the methods | S4 |
| 05 Composition | relative abundance of top taxa | descriptive (shares) | shares are compositional; do not read a fall as a loss | S4 |
| 06 Alpha diversity | richness/evenness per sample | Chao1, Shannon, Simpson, Hill; Wilcoxon, Kruskal–Wallis + Dunn | equal depth or rarefaction curves flat; effect size with P | S5, S2 |
| 07 Beta diversity | differences in community composition | distances, PCoA/NMDS, PERMANOVA (R², P), PERMDISP | PERMDISP not significant before calling a shift; distance named | S5 |
| Differential abundance | which taxa differ, by how much | , ANCOM-BC, ALDEx2, LinDA, Wilcoxon-CLR; BH | consensus across methods; P-value histogram; covariates included | S3, S6 |
| Associations | taxa related to continuous metadata | rank correlation (CLR), BH | outliers, confounders (age, BMI, medication) | S7 |
| Networks | conditional co-occurrence | SPIEC-EASI, SparCC, Spearman-CLR | samples ≫ taxa; stability; edges as hypotheses | S7 |
| Biomarkers & prediction | can features classify new samples? | elastic net, LASSO, random forest; cross-validated AUC | selection inside folds; spread of AUC across folds; external cohort | S7 |
| 14 Pangenome | core and accessory taxa or functions | prevalence thresholds; accumulation curves | thresholds stated; sample size per group comparable | S5 |
| R1 Reference ranges | where a sample sits in a reference population | quantiles of the reference cohort | the reference was measured the same way (region, pipeline) | S1 |
A good analysis reads as one argument. Quality control shows the data are fit to use. Normalisation is stated. Alpha and beta diversity describe how much and how the communities differ overall, with effect sizes (Δ Shannon, PERMANOVA R²) and the dispersion check. Differential abundance names the taxa behind the difference, with fold changes, adjusted P values and agreement between methods. Associations and networks suggest mechanisms, flagged as hypotheses. Prediction says whether the difference is strong enough to classify individuals. Every number travels with its uncertainty and its n. MicroRes's narrative report is written in this order for that reason, and its methods paragraph lists every choice a reviewer needs.
Three questions keep the story honest: Could the design explain it? (batch, confounders, pseudoreplication; chapters 2 and S6). Could the composition explain it? (closure, load; S4). Could chance explain it? (multiple testing, power; S2–S3). An effect that survives all three is worth reporting with confidence.
These points match the STORMS checklist for microbiome studies, which journals increasingly ask for (Mirzayi et al., 2021).
As a method-dependent, tentative finding; the robust findings are those most methods agree on.
Could the design, the composition, or chance explain it?
Launching BactFlowEverything you uploaded and every result the tools wrote. Review it, download it, or delete it.
Your name and picture show at the top of every page.
Download everything the site stores about you, or delete your account and all its files. Deleting cannot be undone.
Sign in with an admin username (or e-mail) and password.
Use the recovery code you saved when the admin account was created, or get a reset link by e-mail.
portal/.venv/bin/python portal/manage.py reset-admin
Open your authenticator app and type the 6-digit code for Bioinformaniac.
1. Install an authenticator app (Google Authenticator, Microsoft Authenticator, Aegis, 1Password, Bitwarden ...).
2. In the app choose "Scan a QR code" and scan this:
No camera? Choose "Enter a setup key" instead and type this key (time-based, 6 digits):
Open it in an app on this device
3. Type the 6-digit code the app shows now.
This form only works on the computer that runs the site. Choose the username and password you will use at /admin.
Admins, user accounts, activity, data and machine use, and site settings.
Checking your link…
Questions, bug reports, collaborations or a dataset that does not load: send a message and you get an answer by e-mail.
Your uploads, saved analyses and account settings live here once you have an account.