Feature Engineering
ESM-2 + PAIRFORMERESM-2 650M frozen backbone (1,280-dim per residue) β Pairformer single/pair tracks (256-dim / 64-dim) β mean-pooled protein embedding for downstream tasks.
Amino acid sequence string. Variable length L β [30, 1011].
facebook/esm2_t33_650M_UR50D β 33 transformer layers, 20 heads, 650M parameters. Frozen during PaTz training. Context-aware residue representations.
Linear projection 1280β256 (d_single). One vector per residue fed into Pairformer as single track.
Outer product mean of projected residue features β LΓLΓ64 (d_pair). Encodes pairwise residue relationships.
4 Pairformer layers: triangle multiplicative updates on z_ij + multi-head self-attention (8 heads) on h_i. Dropout 0.1. Captures long-range co-evolutionary dependencies.
Mean-pool over sequence positions β single 256-dim protein vector used for functional annotation and diffusion conditioning.
| Property | Range | Use |
|---|---|---|
| Hydrophobicity (Kyte-Doolittle) | β4.5 to +4.5 | Membrane-binding prediction |
| isoelectric point (pI) | 2 β 14 | Charge-based interaction scoring |
| Molecular weight | kDa | Size normalisation |
| GRAVY score | β3 to +3 | Global hydrophilicity index |
| Aromaticity fraction | 0 β 1 | ΟβΟ stacking propensity |
| CΞ± bond geometry | Γ , Β° | Physics-informed energy terms |
33 transformer layers, 20 attention heads per layer. Pre-trained on 250M protein sequences from UniRef90 database. Self-supervised masked language model objective.
ESM-2 weights are frozen during PaTz training. Only the Pairformer, score network, and function head are trained (4,387,749 parameters). This constrains VRAM to 6GB.
ESM-2 650M embeddings encode evolutionary information, secondary structure, and contact maps. No explicit feature engineering required beyond the linear projection to d_single=256.