small_molecule_binding: fix guided-RFD3 crashes, deadlock, and stuck seeds

examples/small_molecule_binding/CLAUDE.md

@@ -11,6 +11,8 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 | 2026-09-09 | —      | Replaced broken RFD3 `scaffoldguided.target_pdb` scaffold feedback with real RFD3 partial-diffusion guidance (`partial.input`/`partial_t`); replaced AlphaFold2/ColabFold fold-validation step with Boltz-2 (protein+ligand co-folding) |
 | 2026-09-09 | —      | Fixed `analysis_sequence()` silently never comparing MPNN candidates (it only ever read each `.fa` file's first line — the un-designed template record — since real LigandMPNN writes multiple candidates into one file, not one file per candidate); now parses every candidate record and picks the true highest-confidence one |
 | 2026-09-09 | —      | Added a metric-agnostic non-improvement short-circuit to `fastrelax`/`interface` retry logic — escalates to a new backbone (`STEP_RFD3`) as soon as a retry fails to improve on the previous attempt, instead of always exhausting 5 resequencing retries on backbones that real data showed never recover |
+| 2026-09-09 | —      | Fixed guided-RFD3 ligand atom-name mismatch that crashed 4/4 pipelines in a real production run (job `21916521`) on their first guided-backbone-feedback attempt: `_normalize_ligand_id()` only rewrote the Boltz co-folded PDB's ligand *residue* name, not its Boltz-assigned *atom* names, so `select_exposed`/`select_buried` (copied verbatim from the base spec, keyed by canonical `.params` atom names) never matched and RFD3's validator rejected every guided run. New `_infer_ligand_atom_mapping()`/`_normalize_ligand_atom_names()` establish atom correspondence via element+connectivity graph isomorphism (rdkit) with a Kabsch-RMSD tie-break; `_write_guided_rfd3_json()` now also verifies atom-name coverage before writing. Adds an `rdkit` runtime dependency |
+| 2026-09-09 | —      | Fixed a second guided-RFD3 crash found in a real production run (job `21928556`, 3/3 pipelines that reached guided feedback crashed on their very first attempt): `_write_guided_rfd3_json()` copied the base spec's `partial.length` field verbatim into the guided JSON, but RFD3's `DesignInputSpecification` validator rejects `length` outright whenever `partial.input`/`partial_t` (partial diffusion) are set (`ValidationError: ... Length argument must not be provided during partial diffusion`) — length is inferred from the input structure in that mode. `_write_guided_rfd3_json()` now drops `length` from the guided spec |
 
 ## Context
 
@@ -120,12 +122,29 @@ RFD3 has no `scaffoldguided.*`-style CLI override (that was a leftover from an o
 
 When `rfd3_input_pdb` is set, `rfd3()`:
 1. Reads the ligand's literal residue name from `ligand_params`'s `NAME` record via `_ligand_resname_from_params()` — this is **not** always the params filename stem (e.g. `ALR.params`'s `NAME` is `A:R`, not `ALR`; the colon is a deliberate workaround for RFD3 misresolving the bare `"ALR"` literal — never hardcode or "clean up" this value).
-2. Calls `_normalize_ligand_id()` to rewrite the Boltz model's ligand HETATM residue name (via gemmi, no coordinate transform — Boltz already places the ligand correctly relative to the protein it just co-folded) to match that literal, writing `{taskdir}/in/guided_scaffold.pdb`.
-3. Calls `_write_guided_rfd3_json()` to copy the base `ALR_binder_design.json`'s `ligand`/`length`/`select_exposed`/`select_buried` fields verbatim into a new spec with `input` pointed at the normalized PDB and `partial_t` set, writing `{taskdir}/in/guided_binder_design.json`.
-4. Passes that guided JSON (instead of the base one) as `rfd3.sh`'s `inputs=` argument. If normalization fails (no ligand found in the Boltz model), falls back to the base, unguided JSON rather than erroring.
+2. Calls `_normalize_ligand_id()` to rewrite the Boltz model's ligand HETATM residue *name* (via gemmi, no coordinate transform — Boltz already places the ligand correctly relative to the protein it just co-folded) to match that literal, writing `{taskdir}/in/guided_scaffold.pdb`.
+3. Calls `_normalize_ligand_atom_names()` to rewrite that same PDB's ligand *atom* names to the canonical `.params` names (see "Guided-RFD3 ligand atom-name mapping" below) — Boltz assigns its own arbitrary atom names during co-folding, unrelated to the params file, so this is a separate fix from step 2.
+4. Calls `_write_guided_rfd3_json()` to copy the base `ALR_binder_design.json`'s `ligand`/`select_exposed`/`select_buried` fields verbatim (dropping `length` — RFD3's validator rejects it during partial diffusion, since length is inferred from the input structure) into a new spec with `input` pointed at the normalized PDB and `partial_t` set, writing `{taskdir}/in/guided_binder_design.json` — after first verifying every `select_exposed`/`select_buried` atom name is actually present in the normalized PDB.
+5. Passes that guided JSON (instead of the base one) as `rfd3.sh`'s `inputs=` argument. If any of steps 2–4 fails (no ligand found in the Boltz model, no full atom-name mapping found, or the coverage check fails), falls back to the base, unguided JSON rather than erroring.
+
+### Guided-RFD3 ligand atom-name mapping
+
+Boltz-2's co-folded ligand output uses its own arbitrary atom names (e.g. `C41`, `O24`, ...) that have nothing to do with the canonical names in the ligand's `.params` file (e.g. `C18`, `O3`, ...). Since `select_exposed`/`select_buried` are copied verbatim from the base spec and are keyed by those canonical names, a guided PDB with unrenamed atoms fails RFD3's own input validation (`ComponentValidationError: Number of atoms must be a multiple of the requested names`) — this was the confirmed root cause of a real production run (job `21916521`) crashing all 4 pipeline instances on their first guided-feedback attempt.
+
+`_infer_ligand_atom_mapping()` establishes the correspondence via element + heavy-atom-connectivity graph isomorphism (rdkit), ignoring bond order throughout (the `.params` format has none):
+- **Reference graph**: heavy atoms + bonds parsed directly from the `.params` file's `ATOM`/`BOND` records (`_params_heavy_atom_graph()`) — exact, no perception needed. Reference *coordinates* come from the real, correctly-named structure the base spec's own `partial.input` already points at (`_resolve_reference_pdb_path()`) — no synthetic conformer is built.
+- **Query graph**: the Boltz ligand's connectivity, perceived from 3D distances via `rdkit.Chem.rdDetermineBonds.DetermineConnectivity()` (Boltz's output carries no CONECT records for the ligand).
+- **Isomorphism + tie-break**: `GetSubstructMatches()` enumerates every graph-valid atom correspondence; local topological symmetry (e.g. a sulfonate's three interchangeable terminal oxygens) can yield more than one. Each candidate is Kabsch-superposed (reusing `_kabsch_rmsd()`) against the reference coordinates, and the lowest-RMSD mapping wins — grounded in real geometry rather than an arbitrary tiebreak. When more than one isomorphism exists, the best-vs-next-best RMSD gap is logged.
+
+This is **not a generic guarantee for every future ligand**: it's only provably safe for a ligand whose "sides" (whatever `select_exposed`/`select_buried` partition into) aren't themselves graph-isomorphic to each other — true for `ALR` (a monocyclic benzene-sulfonate ring isn't isomorphic to a fused naphthalene-sulfonate ring), but not checked automatically for `IND`/`RED`/`IAI` or any future ligand. Run `scripts/check_ligand_atom_mapping.py` against a new ligand's `.params` + reference PDB before trusting guided feedback with it.
+
+`_write_guided_rfd3_json()` adds a final defensive check: before writing, it verifies every `select_exposed`/`select_buried` atom name is present in the (now atom-renamed) guided PDB, returning `False` (fail safe, fall back to unguided) rather than reproducing RFD3's rejection in a new form if not.
+
+`scripts/check_ligand_atom_mapping.py` validates this mapping logic against real crash artifacts rather than synthetic test data: the job-`21916521` `guided_scaffold.pdb` files under `logs/p{1..4}/*_rfd3/in/`, with the committed `p1_in/ALR.params` + `p1_in/input_pdbs/scaffold-with-ALR.pdb` as ground truth. **Those crash artifacts are not committed** — `logs/` is gitignored — so on a fresh checkout every check skips and the tool reports `INCONCLUSIVE` (exit 2), not `PASS`. Pass `--base-path <completed run's output tree>` to actually exercise it. `scripts/validate_run.py`'s check 8 (`check_guided_ligand_atom_names`) regression-tests the same invariant against any completed run's output tree.
 
 Ensemble similarity utilities (all in `small_molecule_binding.py`):
 - `_ca_rmsd(path1, path2)` — Kabsch-aligned CA RMSD between two PDB files
+- `_kabsch_rmsd(coords1, coords2)` — the underlying generic Kabsch-alignment RMSD, also reused by `_infer_ligand_atom_mapping()`'s isomorphism tie-break
 - `_seq_identity(fasta1, fasta2)` — fraction matching residues over shorter sequence
 - `_ensemble_selective_avg(current, prior, sim_fn, similar_if_low)` — returns `(overall_avg, selective_avg, has_data)` for scores of entries whose similarity is on the "similar" side of the mean pairwise similarity
 
@@ -207,3 +226,5 @@ Each pipeline instance (named e.g. `p1`) expects a `{name}_in/` directory contai
 - `<ligand_name>.params` — Rosetta ligand params file (default `ALR.params`)
 - `<ligand_name>.smiles` — ligand SMILES string, read by the `boltz` task to build its co-folding input; derive it once with `scripts/derive_ligand_smiles.py <ligand>.params <reference>.pdb` (RDKit bond-order perception from the params file's exact connectivity + the reference structure's 3D coordinates — there is no SMILES in a Rosetta `.params` file itself)
 - Optionally `common_filenames.txt` — used by `filter_energy` for cross-filtering
+
+`rdkit` is a runtime dependency (installed by `delta_env_setup.sh`'s Step 10, pinned `2024.9.6`) used both by the offline `derive_ligand_smiles.py` tool above and, per-cycle, by `rfd3()`'s guided-backbone-feedback atom-name mapping (see "Guided-RFD3 ligand atom-name mapping" above). `scripts/check_ligand_atom_mapping.py` and `scripts/validate_run.py` (check 8) validate that mapping against real fixtures/completed runs, respectively.

examples/small_molecule_binding/delta_env_setup.sh

@@ -200,26 +200,47 @@ echo ""
 echo "── Step 9: gemmi ──"
 "${PIP}" install -q "gemmi==0.6.5"
 
-# ── 10. Additional dependencies ───────────────────────────────────────────────
+# ── 10. rdkit — ligand atom-name graph-isomorphism mapping for guided RFD3 ────
+#
+# Used by rfd3()'s guided-backbone-feedback path (_infer_ligand_atom_mapping
+# in small_molecule_binding.py) to reconcile Boltz-2's arbitrary ligand atom
+# names against the canonical names in the ligand's .params file, via
+# element+connectivity graph isomorphism (rdDetermineBonds.DetermineConnectivity
+# + GetSubstructMatches) with a Kabsch-RMSD tie-break. Without this, RFD3's
+# input validator rejects every guided run (ComponentValidationError) --
+# confirmed as the root cause of 4/4 pipeline crashes in a real production
+# run (job 21916521).
+#
+# Pinned to 2024.9.6, the same version already used by the offline
+# scripts/derive_ligand_smiles.py tool in this repo (rdkit has no
+# dependency on numpy/gemmi's own pins, so it should not disturb Step 7's
+# numpy<2.0/gemmi==0.6.5 resolution -- `pip check` after this step should
+# stay clean; re-investigate only if it doesn't).
+#
+echo ""
+echo "── Step 10: rdkit ──"
+"${PIP}" install -q "rdkit==2024.9.6"
+
+# ── 11. Additional dependencies ───────────────────────────────────────────────
 echo ""
-echo "── Step 10: pandas + biopandas ──"
+echo "── Step 11: pandas + biopandas ──"
 "${PIP}" install -q pandas biopandas
 
-# ── 11. PyRosetta ─────────────────────────────────────────────────────────────
+# ── 12. PyRosetta ─────────────────────────────────────────────────────────────
 echo ""
-echo "── Step 11: PyRosetta ──"
+echo "── Step 12: PyRosetta ──"
 "${PIP}" install -q pyrosetta-installer
 "${PY}" -c "import pyrosetta_installer; pyrosetta_installer.install_pyrosetta()"
 
-# ── 12. Boltz-2 model weights ─────────────────────────────────────────────────
+# ── 13. Boltz-2 model weights ─────────────────────────────────────────────────
 #
 # Boltz has no dedicated "download weights" subcommand — weights auto-download
 # on first `boltz predict` call.  Warm the cache with a trivial CPU prediction
 # on a login node so compute nodes (no internet) find them already present at
 # BOLTZ_CACHE.
 #
 echo ""
-echo "── Step 12: Boltz-2 model weights (cache warm-up) ──"
+echo "── Step 13: Boltz-2 model weights (cache warm-up) ──"
 BOLTZ_CACHE="${BOLTZ_CACHE:-${SCRATCH}/${USER}/.cache/boltz}"
 mkdir -p "${BOLTZ_CACHE}"
 _WARM_DIR=$(mktemp -d)
@@ -237,9 +258,9 @@ YAML
     || echo "WARNING: boltz cache warm-up failed — check login-node internet access"
 rm -rf "${_WARM_DIR}"
 
-# ── 13. Verify ────────────────────────────────────────────────────────────────
+# ── 14. Verify ────────────────────────────────────────────────────────────────
 echo ""
-echo "── Step 13: Verifying installation ──"
+echo "── Step 14: Verifying installation ──"
 _check() {
     local label="$1"; shift
     if out=$("$@" 2>&1); then
@@ -256,6 +277,7 @@ _check "impress"           "${PY}" -c "import impress; print('ok')"
 _check "torch"             "${PY}" -c "import torch; print(torch.__version__)"
 _check "boltz"             "${PY}" -c "import boltz; print('ok')"
 _check "gemmi"             "${PY}" -c "import gemmi; print(gemmi.__version__)"
+_check "rdkit"             "${PY}" -c "import rdkit; print(rdkit.__version__)"
 _check "pyrosetta"         "${PY}" -c "import pyrosetta; print('ok')"
 _check "ProDy"             "${PY}" -c "import prody; print(prody.__version__)"
 _check "LigandMPNN"        test -d "${MPNN_DIR}" && echo "present"

examples/small_molecule_binding/delta_gpu_run.sh

@@ -71,7 +71,7 @@ dragon-config add --ofi-runtime-lib="${FAB_LIB}"
 export MPNN_DIR="${MPNN_DIR:-${SCRATCH}/${USER}/LigandMPNN}"
 
 # Boltz-2 model weights cache — kept on scratch to avoid home quota exhaustion.
-# Pre-warm once on a login node via delta_env_setup.sh's Step 12 (boltz has no
+# Pre-warm once on a login node via delta_env_setup.sh's Step 13 (boltz has no
 # dedicated "download weights" subcommand; weights auto-download on first
 # `boltz predict` call).
 export BOLTZ_CACHE="${BOLTZ_CACHE:-${SCRATCH}/${USER}/.cache/boltz}"