[DimaMolod]

A SLURM prolog can set TMPDIR to job-local scratch (here /scratch/jobs/<jobid>) that is not bound into the container. Python's tempfile silently skips an unwritable TMPDIR, but XLA's ptxas writes its tmpxft_* files there directly and aborts. In the October campaign that failed 151 AF3 inference jobs, and every retry inherited the same path.

This implements option 1 from the issue.

Changes

• container_tmpdir_guard() in rules/common.smk returns a bash prologue that runs inside the container:
  • it keeps $TMPDIR when it is a writable, searchable directory;
  • otherwise it switches to container_tmpdir (default /tmp) and logs one [container-tmpdir] line;
  • if the fallback is unusable too, it exits 1 before any work starts.
• All six container rules run it first: create_mmseqs2_gpu_msa_shard, finalize_mmseqs2_features, create_features, structure_inference, analyze_structure, recursive_report. It is passed as a callable param, because Snakemake applies wildcards to string params and would try to format ${TMPDIR:-}.
• container_tmpdir in config/config.yaml, and a README troubleshooting entry for ptxas fatal: Could not open output file … tmpxft_*.

Testing

• Reproduced first. A new test renders the real structure_inference shell with a dry run, then runs its prologue under set -euo pipefail with TMPDIR below an unbound path and writes a tmpxft_* file natively.
  • On main it fails with …/job-tmp/tmpxft_regression: Not a directory.
  • With the guard it passes and logs the fallback once.
• Strict-bash tests of the guard:
  • unset or empty TMPDIR;
  • usable scratch kept without a log line, whether it exists or can be created;
  • read-only and unsearchable directories fall back;
  • an unusable fallback (a file, read-only, unsearchable) stops before any work;
  • a fallback path containing spaces, quotes, $, backticks and braces stays literal.
• Coverage test. A dry-run test parses the Snakefile for every container: rule and checks that each renders with the guard first, in both feature modes. A new container rule without the guard fails it.
• Full suite: 188 passed, 1 skipped; with Local MMseqs2: a database rebuilt in place under the same identifier is never re-searched #56 and Inference memory settings are coupled: base RAM is also the unified-memory margin, the host coefficient also routes GPUs, and retries barely escalate #66 merged on top, 213 passed, 1 skipped.
• Not run: containers or GPU inference on the cluster.

Notes

• Option 2 (APPTAINERENV_TMPDIR=/tmp) was not used. It always overrides, including a bound and faster job-local TMPDIR; the guard leaves those alone.
• container_tmpdir adds no bind mount. A fallback outside the container's default mounts has to be bound by the site.

Closes #65.

🤖 Generated with Claude Code