Transformers and Attention in Microbiology
One-Sentence Definition
Transformers use attention to relate tokens across a sequence (or set), powering protein and DNA language models, structure modules, and many modern sequence-to-phenotype tools in microbiology.
Simple Explanation
Instead of reading a gene left-to-right only, attention lets every position “look at” relevant other positions — like spotting a catalytic triad or a resistance motif scattered along a protein.
Detailed Scientific Explanation
| Component | Role | Micro relevance |
|---|---|---|
| Token embedding | Amino acids / nucleotides / k-mers → vectors | Sequence alphabet choice |
| Self-attention | Pairwise relevance weights | Long-range contacts, motifs |
| Multi-head stacks | Hierarchical representation | Domains → full protein |
| Pretrain objectives | Masked LM, next-token, contrastive | UniRef / genomic corpora |
| Task heads | Classification, token tags, generation | AMR, function, variant effect |
How to use
- Proteins: embeddings or fine-tunes from Protein Language Models (ESM-family, ProtTrans) for function, solubility, binder design inputs
- DNA/genomes: DNA and Genome Language Models for promoter/AMR elements; still validate against gene databases
- Structure: attention-based modules inside AlphaFold in Microbiology
- Text/LLMs: lab SOP assistants and literature tools (Foundation Models and LLMs in Microbiology) — not AST oracles
- Prefer frozen embeddings + small classifier when labels are scarce (Transfer Learning in Microbiology)
Mechanism
Scaled dot-product attention computes content-based input combinations; depth + scale yield transferable representations.
Clinical Importance
- Promising for novel determinant discovery; not yet a general replacement for phenotypic AST
- Governance needed before LLM text answers enter patient reports
Research Importance
- Foundation of modern representation learning in molecular biology
Diagnostic Relevance
- Indirect today (annotation, triage research); imaging still mostly CNNs
AMR Relevance
- Sequence models can prioritize unknown ORFs near mobile elements; confirm with AMR Gene Databases + MIC
Related MOCs
Active Recall Questions
- Why do transformers fit proteins better than tiny CNNs on one-hot sequences?
- Frozen embedding + logistic head vs full fine-tune — when prefer each?
- What must never be skipped after an LM flags a “new” resistance gene?