Attention & Residue Attribution

IG HEATMAP — LIVE DATA

Pairformer-derived residue-pair importance visualized as an outer-product IG attribution matrix. Actual model attention weights require post-training extraction — shown here as pairwise IG proxy.

🏗

Pairformer Layers

triangle + attention

👁

Attention Heads

per layer

↔

Pair Dim (d_pair)

z_ij per residue pair

💡

—

Proteins (XAI)

with real IG data

Visualization method: Pairformer L×L attention weights are not stored during training. This view shows the outer-product IG attribution map: pair_importance[i][j] = (IG[i] + IG[j]) / 2 — a proxy for which residue pairs jointly contribute to the GO prediction. High values (yellow) indicate co-important residue pairs.

Protein:

Pairwise IG Importance Matrix — …

L×L outer-product of per-residue IG attributions. Yellow = co-important pairs.

Select a protein with IG data.

Per-Residue IG Attribution Bar — …

Raw IG score per residue position. High = stronger contribution to GO prediction.

Select a protein with IG data.

Pairformer Architecture — How Attention Works

Triangle Multiplication (outgoing)L×L×64z̃_ij = σ(g_ij) ⊙ Σ_k (a_ik ⊙ b_kj)

Triangle Multiplication (incoming)L×L×64z̃_ij = σ(g_ij) ⊙ Σ_k (a_ki ⊙ b_kj)

Triangle Attention (start nodes)L×L×648-head attention along rows of pair matrix

Triangle Attention (end nodes)L×L×648-head attention along columns of pair matrix

Transition FFNL×L×642-layer MLP, ReLU activation per pair position

8 Attention Head Specializations (observed)

Local contacts (short-range < 6 Å)

Domain boundary detection

Secondary structure transitions

Long-range coevolution

Hydrophobic core packing

Surface exposure

Binding site residues

Disordered region detection

Specializations are patterns from protein LM literature — heads are not explicitly labeled during training.