Ablation Studies

FULL MODEL DONE — ABLATIONS PENDING

Systematic ablation of PaTz components. Full model Fmax=0.3212 (baseline). Each ablation removes one component and retrains from scratch on the same train/val/test split.

Note: Only the full PaTz model has been trained and evaluated. Ablation variants have not been run yet. Each ablation requires retraining from scratch (~50 epochs, ~2-3 hours per run on RTX 3050). Results will populate this table as they are collected.

📐

0.3212

Full Model Fmax

baseline for comparison

📈

0.8568

Full Model AUROC

test set n=525

🔬

Ablations Planned

not yet run

🧮

4.39M

Total Params

trainable (ESM-2 frozen)

Ablation Study Table

Configuration	Removed Component	Fmax	AUROC	ΔFmax	Status
PaTz (full model) Pairformer (4L×8h) + Physics loss (λ=0.01) + Score diffusion (T=1000) + MC Dropout (N=20) + Function head.	—	0.3212	0.8568	baseline	✓ DONE
w/o Physics Loss Tests contribution of differentiable Cα bond/angle/clash energy terms. Hypothesis: physics loss improves structure-guided GO predictions.	λ_phys=0 (loss_phys zeroed)	—	—	—	NOT RUN
w/o Score Diffusion Removes score-matching diffusion (T=1000). Protein embedding passed directly to function head. Tests structural conditioning contribution.	Diffusion head removed	—	—	—	NOT RUN
w/o Pairformer Pair Track Removes L×L pair track. Only single track h_i retained. Tests pairwise residue–residue reasoning contribution.	d_pair=0 (no z_ij)	—	—	—	NOT RUN
Pairformer 1 Layer (vs 4) Reduces Pairformer depth from 4 to 1 layer. Tests depth of triangle-update message passing.	n_layers: 4→1	—	—	—	NOT RUN
w/o MC Dropout Standard deterministic inference instead of N=20 MC Dropout passes. Tests uncertainty quantification contribution.	dropout=0 at inference	—	—	—	NOT RUN
Random ESM-2 Init (no pretraining) ESM-2 backbone with random (not pre-trained) weights. Extreme ablation: measures contribution of transfer learning from UniRef90.	ESM-2 weights randomised	—	—	—	NOT RUN

Ablation Progress

PaTz (full model)Complete

w/o Physics LossPending

w/o Score DiffusionPending

w/o Pairformer Pair TrackPending

Pairformer 1 Layer (vs 4)Pending

w/o MC DropoutPending

Random ESM-2 Init (no pretraining)Pending

Ablation Methodology

🔒 Controlled Variables

Same train/val/test split (split_manifest.json). Same random seed (42). Same hyperparameters (training/default.yaml). Only one component changed per ablation.

📐 Evaluation Protocol

CAFA Fmax evaluated on held-out test set (n=525 proteins, MMseqs2 30% cluster split). Same threshold sweep τ ∈ [0,1]. Macro-averaged over proteins.

⏱ Expected Runtime

~50 epochs × ~3 min/epoch = ~2.5 hours per ablation run on RTX 3050. 6 ablations = ~15 hours total. Run sequentially or overnight.