Add Harbor Framework Support

[hrdkbhatnagar]

Adds Harbor framework support to PostTrainBench, enabling anyone to run our benchmark on cloud GPUs (Modal, Daytona) without needing access to our internal HTCondor cluster.

At the moment:

• Generate Harbor-compatible task directories from PostTrainBench benchmarks
• Almost full parity with original pipeline

Tested:

• Generated task for gsm8k + qwen3-1.7b
• Ran 1-hour test with Claude Code on Sonnet 4 on Modal
• Verified end-to-end pipeline (including eval + contam judge)
• Confirmed accuracy metrics extracted correctly

Usage

  cd src/harbor_adapter
  uv sync

  # Generate a task
  python run_adapter.py --benchmark gsm8k --model qwen3-1.7b --output ./tasks

  # Run with Harbor
  harbor run \
      --path ./tasks/posttrainbench-gsm8k-qwen3-1.7b \
      --agent claude-code \
      --model anthropic/claude-sonnet-4 \
      --env modal

See src/harbor_adapter/README.md for detailed parity tracking. Key points:

• Agent timeout, GPU access, evaluation: Full parity
• Contamination judge: Parity
• Agent duration: Tracked by Harbor in result.json
• timer.sh: Minor difference (created at task generation vs job start)

Note: Right now I have skipped the installation of flash-attn in the container as we need to have a CUDA runtime for it. In modal the GPU is attached to the sandbox after the container is built, so installation doesn't occur.

Note: I have added a uv environment for us to use in PTB. This is used for using modal and harbor, and is useful in general for reproducibility

Todos:

• directly before agent is run, install flash_attn and build timer.sh
• huggingface cache in a modal storage
• before running evaluation, uninstall and reinstall major dependencies (like transformers, inspect-ai, ...). Make sure NOT to use the cache. ((Alternatively we can look into docker in docker))

[hrdkbhatnagar]

pin versions like the current images

[hrdkbhatnagar]

things that are remaining to get full parity with the original PTB implementation:

1. Separate verifier container (eval integrity)
   In our setup, we evaluate the agent's post-trained model in a different container than the one the agent trained in. This prevents reward hacking, the agent could modify eval files in its workspace to inflate its score. In Harbor, the verifier runs in the same sandbox as the agent, so it uses whatever files the agent may have tampered with. We need a way to run the verifier in an isolated environment.

2. Pre-agent shell command inside the container
   We have a timer script that the agent calls to check remaining time (out of 10 hours). It needs to know when the agent actually started. In our original setup, the host orchestrator writes the start timestamp before launching the agent container. In Harbor, I see lifecycle hooks on the Trial object (TrialEvent.AGENT_START etc.), but those run on the orchestrator side, not inside the sandbox. we need a way to execute a shell command inside the container right before the agent starts , like a pre-agent hook.

3. Downloading additional directories after a run
   After the agent finishes, we'd like to download its full workspace (/home/agent/workspace/), including the code it wrote and the fine-tuned model weights. Currently Harbor only downloads /logs/agent and /logs/verifier.

[hrdkbhatnagar]

after discussing with Alex from Harbor/tbench:

1. we could put the verifier in the tests/ directory which only gets uploaded after the agent runs

2. this is not yet supported be we should look into this, potentially make a PR to harbor

3. Artifact collection is supported now in harbor, so we should use that

[rank-and-file]

Added modal storage for hf-cache for harbor in the branch add_harbor_support_with_hfcache. This is necessary to have the pre-cached models and datasets. This is necessary for full parity.

Although there are some other changes as well, so you can probably clone the repo with this branch in another directory and ask your agent:

Consider PostTrainBench_harbor_modal and port all changes which are related to the modal storage integration for the huggingface cache to PostTrainBench_harbor.

[rank-and-file]

Also there is an upcoming change to the judge which will need to be integrated. Will post here.

[rank-and-file]

We need to hardcode baseline values to a json, instead of fetching them from the POST_TRAIN_BENCH_RESULTS_DIR.

This is needed for harbor integration (harbor should output the baseline value, in case the judge flags the run).

[hrdkbhatnagar]

Merged main into Harbor branch, @rank-and-file maybe we should push the new judge to main soon so we can pull it here. The new judge would require some major changes for Harbor

[lewtun]

Apologies for lurking, but I noticed this comment:

Right now I have skipped the installation of flash-attn in the container as we need to have a CUDA runtime for it

In case it's useful, we have flash-attn kernels available on the Hub (link) which are matched to the hardware at runtime and skip the annoying / long / brittle install of flash-attn itself. If you want to run PostTrainBench on cloud GPUs, this could be the fastest way to support such kernels across e.g. different CUDA versions

[hrdkbhatnagar]

Apologies for lurking, but I noticed this comment:

Right now I have skipped the installation of flash-attn in the container as we need to have a CUDA runtime for it

In case it's useful, we have flash-attn kernels available on the Hub (link) which are matched to the hardware at runtime and skip the annoying / long / brittle install of flash-attn itself. If you want to run PostTrainBench on cloud GPUs, this could be the fastest way to support such kernels across e.g. different CUDA versions

Hey Lewis, thanks for your comment! I didn't know about this, it would be very useful for us, especially when running on cloud providers and having full parity with the local version. Adding it to our todo for Harbor :)

[hrdkbhatnagar]

update

Pushed a substantial round of changes to the harbor adapter since last review. end to end run is now working at parity with our local pipeline for everything except verifier sandbox isolation, which is being addressed natively upstream (harbor-framework/harbor#1607).

main changes

• Build mirrors the image we used for the leaderboard
• Live log streaming (agent + verifier) via PID-1 entrypoint
• Timer parity (healthcheck-driven, absolute path)
• system_monitor.sh ported (GPU/CPU/mem/disk every 60s)
• /tests/ placement closes file-tampering attack vector
• final_model/ + agent workspace/ downloaded as artifacts after each trial
• Single sandbox model leaves package tampering open; [verifier_environment] support upstream (Feature: isolated verifier sandbox ([verifier_environment]) harbor-framework/harbor#1607) lets us drop in a fresh sandbox for verification — fork branch validated, will integrate once landed

build

template/environment/Dockerfile is now a faithful port of the production apptainer image with two intentional changes:

• --torch-backend=cu128 instead of auto — Modal's build VM has no nvidia-smi, so auto resolves to CPU torch and breaks vllm's CUDA-only xformers requirement.
• I decided not to go with the HF kernels as suggested by Lewis since that would deviate from our original setting, as the kernels lib does not re-export under the flash_attn module name AFAICT. Agents either route through kernels directly or use the transformers attn-implementation string. But this is very useful for the future versions of PTB we build!

log streaming

template/environment/entrypoint.sh runs as PID 1 and:

1. backgrounds tail -F -q /logs/agent/*.txt /logs/verifier/*.txt so agent and verifier output stream live to the Modal dashboard via the sandbox's main process stdout, mirroring our local setup.
2. backgrounds system_monitor.sh (port of src/utils/system_monitor.sh), which writes GPU utilization / memory / disk to /logs/agent/system_monitor.log every 60s.

timer (now reliable)

Old design was START_FILE="$(dirname "$0")/.timer_start" — dirname resolves relative to invocation cwd, so each new directory the agent cd'd into got a fresh sentinel and the timer reset.

New design (native Harbor healthcheck):

• [environment.healthcheck] in task.toml writes /timer_start once before agent setup begins:

  [environment.healthcheck]
  command = "pgrep -f 'tail -F' > /dev/null && (test -f /timer_start || date +%s > /timer_start)"                                            
  interval_sec = 2                  
  timeout_sec = 5              
  start_period_sec = 5                    
  retries = 3                      

tamper resistance: /tests/ placement

The single sandbox means evaluate.py originally lived in the same writable directory the agent worked in: agent could trivially modify it to inflate scores. To match condor's "verifier reads from a separate trusted source" pattern within harbor's constraints:

• Adapter's _copy_eval_files() helper now populates both environment/ (so agent has copies for iterative testing during training,
  matching condor) and tests/ (so verifier reads untampered copies).
• Harbor uploads tests/ to /tests after the agent process exits — the agent has no opportunity to write there.

Closes: agent modifying evaluate.py, templates/, evaluation_code/, contamination_judge.py, metadata.json.

Doesn't close: agent pip installing tampered packages, addressed natively in harbor-framework/harbor#1607.

artifact collection

task.toml now declares two artifacts:

[[artifacts]]
source = "/home/agent/workspace/final_model"
destination = "final_model"
                                                                                                                                              
[[artifacts]]
source = "/home/agent/workspace"                                                                                                             
destination = "workspace"
exclude = ["final_model", "__pycache__", "*.pyc", ".git", ".venv", "venv"]

After each trial, <trial_dir>/artifacts/ contains final_model/ (the trained weights the verifier evaluated) and workspace/

tested e2e on modal with 1hr tasks (bfcl, gsm8k on qwen3 1.7b)