TACC/Vista eval: multi-operator support - #91
Open
mrinal-bespoke wants to merge 5 commits into
Open
Conversation
Same fix as the Leonardo $USER commit, for the vista (TACC) eval cluster view. The write/code paths (project_root, hf_cache, eval_jobs_dir, harbor_src, datasets_dirs, secrets_file) hardcoded penfever's /scratch/10635/penfever, so no other operator could run the listener there (another user's dirs aren't writable; secrets_file pointed at penfever's keys.env). Parameterize them with $SCRATCH, which to_eval_cluster_view()/load_cluster_config() expandvars-expand at load time. $SCRATCH (not $USER) because TACC scratch is /scratch/<allocnum>/<user> and the alloc-number differs per user — only $SCRATCH encodes both. For penfever $SCRATCH=/scratch/10635/penfever, so this renders byte-identical to the prior hardcoded values (verified: view rendered as penfever == old literals). conda_envs.otagent and hardware.cuda_home stay pinned to penfever's prefix on purpose — it's world-readable and shared read-only, so a new operator borrows it without rebuilding the aarch64 env. Point at your own once built. hpc.py 'tacc' view and the eval/clusters/tacc.yaml shim kept consistent (paths parity verified rendered as mkumar73). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ety) Port the leonardo sbatch's EVAL_SKIP_UPLOAD guard to the TACC path — main's TACC sbatch had neither EVAL_SKIP_UPLOAD nor EVAL_SKIP_DB. Without it a subset/canary run would call upload_eval_results() (HF upload + Supabase register in one call) and either register a below-gate canary as a leaderboard result or crash on the bare os.environ["HF_TOKEN"] when no token is set. The case-guard exit 0s before the uploader, preserving results on disk. Needed for any TACC smoke/canary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…CH has no otagent env Job 886379 died 9s in with EnvironmentLocationNotFound: Not a conda environment: /scratch/11694/mkumar73/miniconda3/envs/otagent The operator had a bare $SCRATCH/miniconda3 with no otagent env inside it. The listener had ALREADY passed a perfectly good shared env in OTAGENT_DIR (the vista eval_cluster_view pins conda_envs.otagent to the world-readable /scratch/10635/penfever/miniconda3/envs/otagent, precisely so a borrowing operator does not have to build their own), but the sbatch never read that var — it defaulted OTAGENT_CONDA_PREFIX on the variable being UNSET, so a present -but-empty miniconda root beat the env that actually existed. Resolve in order OTAGENT_CONDA_PREFIX > OTAGENT_DIR > $SCRATCH default, and select on the env EXISTING (<prefix>/bin/python) rather than on the var being set. Unchanged for an operator who has built their own env. Same defect class as 2442d5e (evalchemy: validate the ENV, not just the miniforge3 root). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…OAD_HF_ORG)
The TACC uploader hardcoded DCAgent2/<run_tag>. An operator without DCAgent2
write gets a 403 ('Make sure your token has the correct permissions'), the leg
logs 'HF upload failed but DB records uploaded successfully', and hf_traces_link
is never set -- so the leaderboard row lands with no traces.
Leonardo has had EVAL_UPLOAD_HF_ORG since 06ef8f9; TACC was missed. Same default
(DCAgent2), so this is byte-identical for anyone who already had write there.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
TACC Vista's eval path was hardcoded to one operator's
/scratch/10635/penfever/...for every write and code path — project root, HF cache, eval_jobs, harbor src, secrets.
A second operator could not run an eval on Vista at all.
Change
$SCRATCH-parameterized, not$USER-parameterized. Each operator's$SCRATCHalready encodes
/scratch/<allocnum>/<user>, and the alloc number differs per user, so$USERalone is insufficient. Expanded viaexpandvarsat load time on the login node,where
$SCRATCHis always set.Byte-identical for the original operator: with
$SCRATCH=/scratch/10635/penfeverevery path renders exactly as before.
conda_envs/cuda_homestay pinned to penfever's prefix on purpose — that env isworld-readable and shared read-only, so a new operator borrows it rather than rebuilding
a ~20 GB env. Point at your own once you have one.
Fixes included
Conda resolution (
OTAGENT_DIRfallback). A borrowing operator can have a bare$SCRATCH/miniconda3with nootagentenv inside. The listener already passes a goodshared env in
OTAGENT_DIR(from the vistaeval_cluster_view'sconda_envs), but thesbatch defaulted on the variable being unset, so a present-but-empty miniconda root beat
the env that actually existed. Job 886379 died 9 seconds in with
EnvironmentLocationNotFound. Now resolvesOTAGENT_CONDA_PREFIX > OTAGENT_DIR > $SCRATCH default, selecting on the env existing (<prefix>/bin/python).Trace-upload org (
EVAL_UPLOAD_HF_ORG). The uploader hardcodedDCAgent2/<run_tag>.An operator without DCAgent2 write gets a 403, the leg logs "HF upload failed but DB
records uploaded successfully", and
hf_traces_linkis never set — so the leaderboard rowlands with no traces. Leonardo has had this since
06ef8f9; TACC was missed. Same default(
DCAgent2), so no change for anyone who already had write there.EVAL_SKIP_UPLOADguard for local-only smoke/canary runs.Testing
Running in production on Vista for several days across ~25 registered eval legs — paths
resolve to the submitting operator's
$SCRATCH, conda borrows the shared env, and tracesupload to the configured org.