PaTz — Physics-Aware Transformer with eXplainability
● API Offlinepatz_env

Deep Learning Models

4,387,749 PARAMS

PaTz — Physics-Aware Transformer with eXplainability. Pairformer + Score-based Diffusion + FunctionHead.

PaTz Total Loss
Ltotal = Lscore + λ(t)·Lphys + Lfunc
Lscore = E[‖s_θ(x_t,t,c) + ε/σ(t)‖²]  ·  λ(t) = min(1, t/T_warmup)
L_func = multilabel BCE over 354 GO terms · L_phys = w_bond·E_bond + w_angle·E_angle + w_clash·E_clash
Model Architecture Pipeline
ESM-2 (650M)Input Encoder
33 transformer layers · 1280-dim per-residue embeddings · frozen (pretrained)
PairformerStructure Reasoning
4 layers · d_single=256 · d_pair=64 · 8 heads · OuterProductMean + TriangleMultiplication
Score-based DiffusionStructure Generation
T=1000 · linear β schedule · β₀=0.0001, β_T=0.02 · ScoreNetwork denoiser
FunctionHeadGO Annotation
354 GO terms · 2-layer MLP (512→354) · multilabel BCE · threshold-optimised at val
MC DropoutUncertainty
N=20 stochastic passes · p=0.1 · predictive entropy H[p] per protein
Parameter Budget
Total Parameters
4,387,749
PaTz (excl. ESM-2)
Pairformer Params
~3.2M
4 blocks · d=256/64
FunctionHead Params
~180K
512→354 MLP
ESM-2 (frozen)
650M
Not trained
Training Configuration
Epochs
50 (early stop patience=10)
Batch Size
8 (effective 32 via grad accum×4)
Learning Rate
1e-4 (cosine with restarts)
Precision
FP16 + gradient scaling
Optimizer
AdamW (weight decay 1e-4)
Hardware
RTX 3050 6GB VRAM