[DimaMolod]

The MSA cache key records each database's identifier, not its content, and a completed shard is never re-validated against the databases. So a database rebuilt or re-copied in place under the same identifier kept serving MSAs from the old build, while proteins added later were searched against the new one. AlphaPulldown's own index_size check never ran, because no search job is scheduled on a cache hit.

This implements option 1 from the issue, adding a durable witness so that the check also covers a complete DAG.

Changes

• validate_database_provenance() in rules/mmseqs2_gpu.smk runs at parse time whenever local MMseqs2 has proteins to search:
  • Reads each {path}.index size. An unreadable index fails with a message saying indexes must be visible on the submit host and on the workers.
  • Writes a witness on first use. {msa_cache}/.database_provenance.json records an identifier and index size per database. It is published atomically with os.link, so a racing parse checks the winner's witness instead of overwriting it.
  • Checks it on every later parse, including when everything is complete. On a mismatch it stops with, for example, uniref90 (index size 6 -> 12) and "Assign a new identifier to the rebuilt database and rerun."
  • Refuses a mixed legacy cache. A cache created before this change is adopted only after every completed, valid bundle's recorded provenance matches today's indexes. An existing mixture of old and new builds is refused rather than blessed.
• The cache key is unchanged, so every parse still agrees on the cache path (the cross-parse constraint behind feat: schedule local MMseqs2 features for AlphaFold 2 runs #55).
• config/config.yaml states the new requirement and the witness's limits; the README is unchanged.

Testing

• Reproduced first. A dry-run test builds a complete cache (bundle, completion summary, feature symlink), changes uniref90.index and runs again.
  • On main the second dry run printed "Nothing to be done".
  • Now it exits non-zero and names the database.
  • After the identifier changes, the dry run schedules create_mmseqs2_gpu_msa_shard and finalize_mmseqs2_features again.
• Further tests:
  • the witness keeps the cache key stable and accepts an identical copy with a newer mtime;
  • a legacy cache mixing two builds is rejected before adoption;
  • a missing index fails before anything is scheduled;
  • a run using only precomputed features does not need the indexes.
• Full suite: 181 passed, 1 skipped; with Container rules inherit a host TMPDIR that isn't bound inside the container, so XLA's ptxas fails #65 and Inference memory settings are coupled: base RAM is also the unified-memory margin, the host coefficient also routes GPUs, and retries barely escalate #66 merged on top, 213 passed, 1 skipped.

Compatibility

• Databases visible only on the workers' local disks are no longer supported. The submit host must be able to read the .index files at the configured paths. Node-local staging still works if those paths also resolve on the submit host to the same build.
• Index size is not a content hash. A rebuild that leaves every .index the same size, or changes data without changing the index, is not detected. Identifiers still have to change on a rebuild; this check makes the common case fail loudly instead of silently.
• Upgrading reads the cache once. The first parse on an existing cache reads every completed bundle before writing the witness. On a large campaign over NFS this can take a while, once.
• Option 2 (a witness in the shard registry, compared by the search job) was not taken. It cannot catch a fully complete cache, because no search job runs.

Closes #56.

🤖 Generated with Claude Code