Recapitulating the ESMFold2 Ab–Ag results

[echen1214]

Hi ESM team,

Thanks for the great work and open-sourcing many of the methods and code. @thomas-a-neil during last week's office hours recommended that I open up an issue here.

I've been trying to recapitulate the ESMFold2 Ab–Ag results on the Foldbench dataset (Fig 2C) by running the results locally and had noticed some differences in the results. In my testing, most of the single-sequence results are within the lower end of the standard deviation, while the MSA performance is degraded. I also wanted to confirm that for ESMFold2-single seq that we should also use LM_mask_pct=0.1.

Category	Model	# loops	Diffusion steps	LM_mask_pct	Original DockQ rate	Recapitulated DockQ rate
Single Seq	ESMFold2-Fast	10	68	0.1	50 ± 2	45.4 ± 2.2
Single Seq	ESMFold2	10	68	0.1	50 ± 2	45.6 ± 2.0
Single Seq	ESMFold2	20	68	0.1	51 ± 2	47.1 ± 1.6
MSA	ESMFold2 with ColabFold-MSA	10	68	0	53 ± 2	47.0 ± 1.6
MSA	ESMFold2 with ColabFold-MSA	20	68	0	55 ± 2	49.7 ± 2.0

(25 seeds, bootstrapped 5-of-25 without replacement, n=200 resamples, best-ranked sample per interface by ipTM.)

I was wondering if you had any suggestions on how to reach the level reported by the paper, or whether you could share which random seeds you used.

Environment / commits used

• Biohub/esm: d96737e for the single-seq/fast runs and cf002f1 for the runs MSA. ( cf002f1 didn't have a significant improvement compared to the previous d96737e commit on the MSA results)
• Biohub/transformers: ef32577
• Precision: ESMC in fp8 (H100), everything else via the model's built-in autocast (bf16 trunk/pair-transition/confidence, fp32+TF32 diffusion), matching the paper's training config.
• ColabFold-MSA search on UniRef30 release 2023-02 + ColabFold envDB 202108
• Minimal artifact esmfold2_fold_minimal.py

Thanks again, and let us know if any of the above is useful or if we're missing a config knob somewhere.

[faustomilletari]

this issue is being investigated. stay tuned

[faustomilletari]

Hello,

Thanks again for the detailed report. So far we haven't been able to reproduce the gap on our side — running the exact stack using biohub/esm and biohub/transfomers repo as well as biohub/ESMFold2 + biohub/ESMC-6B models. The model loads and folds as expected and, on our benchmark inputs with our scorer, matches the paper.

That points to a possible difference in the inputs or the scoring rather than the model itself.

To pin it down, could you share (even for just 2–3 targets):

• the input data you fold (the sequences_json, and the .a3m files for the MSA runs) — or enough detail to confirm we're using the same inputs or even better same datasets.
• how you compute the DockQ scores / pass-rates.
• if possible, the predicted structures you get from the model for those targets.

With those we should be able to reproduce your numbers exactly and identify the discrepancy. Thanks!

[echen1214]

Hi Fausto,

Thanks for the prompt response and taking a look into this. I've provided a tarball with the minimal artifacts for how I build the inputs, MSAs and score the predictions. Let me know if you need any more information.

--

The results below are the from the best ipTM model from ESMFold2 with ColabFold-MSA (num_loops=20, num_sampling_steps=68, 25 seeds x 5 diffusion samples = 125 predictions/target).

Targets

pdb_id	best-sample DockQ
8jyr-assembly1	0.933
8opr-assembly1	0.434
8gho-assembly1	0.101

Regenerating predictions

esmfold2_fold_minimal.py script run once per target, pointing --sequences_json at that target's flat chain list and --msa_dir at its a3m dir:

python esmfold2_fold_minimal.py --mode msa --pdb_id 8jyr-assembly1 \
    --sequences_json sequences_flat/8jyr-assembly1.json \
    --msa_dir msa/8jyr-assembly1 \
    --num_loops 20 --seeds # 25 random seeds

Input and MSA generation

• Inputs (generate_inputs.py: sequences.json): for each target, we pull the full SEQRES sequence per chain straight out of the ground-truth bioassembly CIF via OpenStructure, keyed by label_asym_id, and emit chains label-sorted so antibody/antigen order stays deterministic.
• MSAs (generate_msa.py: msa/{pdb}/{chain}.a3m): per-chain ColabFold colabfold_search (MMseqs2, GPU) against UniRef30 2023-02 + ColabFold envDB 202108, default params. Hits come back with compact headers (>UniRef100_ACC stats...) carrying no taxid of their own, so we separately scan UniRef30's raw header DB for TaxID= (restricted to just the accessions we hit) to build an accession->taxid lookup, then use that to tag UniRef30 hits key=<taxid> (row 0 = query). envDB hits aren't in that lookup and are left untagged/unpaired. It seems that ESMFold2 does the actual multi-chain pairing itself at load time from those key=<taxid> tokens (via esm/models/esmfold2/paired_msa.py).

Bootstrap sampling and scoring

• We run scoring through FoldBench's own evaluate.py -> evaluation/eval_by_ost.py pipeline. Then, we locally change task_score_summary.py's to perform seed-level bootstrap
• Scorer: OpenStructure (ost compare-structures), version 2.8.0 (aivant::openstructure=2.8.0 conda package, pinned in abag_eval/environment.yml), per interface (antibody_chain vs antigen_chain from targets.csv), predicted CIF vs ground_truth/{pdb}.cif. Automatic chain mapping, with fallback to forced identity mapping only if ost's automatic mapper crashes.
• Per target: take the single sample (across all seeds x diffusion samples) with the highest ipTM, per interface
• Pass-rate tiers: acceptable ≥0.23, medium ≥0.49, high ≥0.80
• Reported ±1σ: bootstrap over seeds — resample 5-of-25 seeds without replacement, 200 resamples, recompute the pass rate each time.

Where the underlying datasets come from

Dataset	Source	Version / release
FoldBench Ab-Ag target list	FoldBench/targets/interface_antibody_antigen.csv	commit 4273f68 on main
UniRef30	ColabFold DB mirror (wwwuser.gwdg.de/~compbiol/colabfold, also colabfold.mmseqs.com)	release 2023-02 (uniref30_2302.tar.gz)
ColabFold envDB	same ColabFold DB mirror	202108 (colabfold_envdb_202108.tar.gz)

Tarball Contents

• sequences.json, sequences_flat/{pdb}.json — inputs (AF3-dialect and flat-chain-list forms).
• msa/{pdb}/{chain}.a3m — MSAs used.
• ground_truth/{pdb}.cif, targets.csv — native structures + chain mapping.
• predictions/{pdb}_best.{cif,json} — the single best-sample structure + full metrics behind the DockQ numbers above.
• scores_per_sample.csv — OST scores for those 3 highlighted samples.
• generate_inputs.py, generate_msa.py — how sequences.json/msa/ were built

Environment

• BEAM-Labs/FoldBench: 4273f68 on main
• OpenStructure (ost compare-structures), version 2.8.0 (aivant::openstructure=2.8.0 conda package)
• ColabFold search: colabfold v1.6.1 + MMseqs2 16-747c6 (static GPU/CPU binaries pinned from the MMseqs2 GitHub releases)