Agentic Coding Test-Time Scaling Reproduction

This repository is an open-source reproduction effort for Meta's paper "Scaling Test-Time Compute for Agentic Coding". It focuses on the Terminal-Bench setting and implements a practical PDR-style loop around mature terminal agents such as Terminus:

• parallel rollout generation
• structured semantic summaries
• recursive tournament voting (RTV) select-K
• final RTV selection
• quota-aware pause/checkpoint behavior
• task/summary/judge pool decoupling

This project is not affiliated with Meta, Terminal-Bench, Harbor, Terminus, or any model provider. Terminal-Bench source files, benchmark data, task files, and local experiment traces are intentionally not included in this repository.

Repository Layout

scripts/
  run_harbor_rollout.sh      Single Harbor rollout wrapper
  run_pdr_task_pool.py       End-to-end PDR task-pool runner
  semantic_summarize_job.py  Structured rollout summary agent
  rtv_select.py              Recursive tournament voting judge
  compare_pdr_results.py     Iteration-level metric comparison
  pdr_utils.py               Shared parsing and aggregation helpers
src/
  json_command_agent.py      Early JSON-command agent prototype
experiments/
  tb40_stratified_tasks.txt  Example stratified Terminal-Bench task list

Prerequisites

• Linux server with Docker installed and usable by the current user
• Python 3.12+
• uv or pip
• A working Harbor/Terminal-Bench installation
• A model API key available through LiteLLM-compatible environment variables

The experiments in this repo used a GLM code model through a provider-compatible LiteLLM route. The scripts default to anthropic/glm-5.2-code, but you can override this with MODEL or --model.

Install

git clone https://github.com/zy95-12/Agentic-Coding.git
cd Agentic-Coding

python3.12 -m venv .venv312
source .venv312/bin/activate
pip install -r requirements.txt

Install Harbor and Terminal-Bench according to their official documentation. The scripts assume the harbor CLI is on PATH and can run a command like:

harbor run -d terminal-bench/terminal-bench-2 -l 1 -k 1 -n 1 -y

Configure Model Access

Create a local .env file. Do not commit it.

cp .env.example .env

Fill in the API variables required by your LiteLLM route. For example, if your GLM endpoint is exposed through an Anthropic-compatible route:

ANTHROPIC_API_KEY=...
ANTHROPIC_API_BASE=...
MODEL=anthropic/glm-5.2-code

For a different provider or model, update MODEL and the corresponding LiteLLM environment variables.

Configure Terminal-Bench

This repository does not vendor Terminal-Bench. Configure it locally:

1. Install Docker and verify docker ps works.
2. Install Harbor and Terminal-Bench using their official setup instructions.
3. Make sure Harbor can locate or download the terminal-bench/terminal-bench-2 task package.
4. Run a one-task smoke test:

HARBOR_ENV_FILE=.env \
TASKS=1 ATTEMPTS=1 CONCURRENT=1 \
scripts/run_harbor_rollout.sh

If this creates a runs/glm_rollouts/.../result.json, the basic environment is ready.

Run the PDR Pipeline

Small smoke experiment:

source .venv312/bin/activate

python scripts/run_pdr_task_pool.py \
  --root-dir runs/pdr_smoke \
  --tasks polyglot-c-py make-mips-interpreter \
  --rollout-pool-size 2 \
  --attempts 4 \
  --attempt-concurrent 2 \
  --summary-pool-size 2 \
  --judge-pool-size 2 \
  --rtv-judge-concurrency 2 \
  --rtv-votes 4 \
  --k 2 \
  --model "${MODEL:-anthropic/glm-5.2-code}"

Larger experiment using the example stratified task list:

python scripts/run_pdr_task_pool.py \
  --root-dir runs/tb40 \
  --tasks $(cat experiments/tb40_stratified_tasks.txt) \
  --rollout-pool-size 8 \
  --attempts 16 \
  --attempt-concurrent 2 \
  --summary-pool-size 4 \
  --judge-pool-size 4 \
  --rtv-judge-concurrency 4 \
  --rtv-votes 8 \
  --k 4

Outputs are written under runs/, which is ignored by Git because Terminal-Bench traces can contain benchmark content and provider-specific logs.

Useful Parameters

• --attempts: number of rollouts per generation, default 16
• --max-turns: Terminus max turns, default 8192
• --agent-timeout-multiplier: Harbor agent timeout multiplier, default 0.70
• --rollout-pool-size: number of tasks with active rollout stages
• --summary-pool-size: concurrent summary jobs
• --judge-pool-size: concurrent RTV jobs
• --rtv-votes: votes per pairwise judge decision
• --k: selected candidates passed from generation 0 to generation 1

Small-Scale Experiment Log

The experiments below were small-scale engineering checks. They are included to show that the workflow is executable end to end, not to claim full reproduction of the paper's headline benchmark numbers. The original paper evaluates at much larger scale with its own agent stack, model choices, and infrastructure.

What this repository has validated so far:

• Harbor/Terminus can be used as the task agent for Terminal-Bench rollouts.
• Independent rollouts can be collected with AgentTimeoutError counted as a normal failed sample instead of crashing the experiment.
• Rollout trajectories can be converted into structured semantic summaries.
• RTV select-K can choose candidate summaries and pass a refinement context into a second generation.
• Final RTV and metric comparison can run after the second generation.
• Rollout, summary, judge, and metrics stages can be decoupled through a task pool so long-running rollouts do not block all other pipeline stages.

Early Agent Check

An early prototype used a hand-written JSON-command terminal agent. We abandoned that path because Terminal-Bench is a heavy interactive setting: the agent must maintain shell state, recover from partial output, handle long-running commands, and reliably submit completion. In small tests the JSON-command agent produced too many format and state-management failures, so the workflow moved to Harbor/Terminus.

PDR/RTV Smoke Runs

Small PDR loops were run with GLM code models through a LiteLLM-compatible route. The main configuration was:

attempts = 16
T = 2 generations
K = 4 selected candidates
RTV group size = 2
RTV votes = 8
rollout input/output budget = 128k / 8k
agent_timeout_multiplier = 0.70

These runs confirmed that semantic summaries and RTV selection can produce a usable refinement context for later rollouts. They also exposed several system issues that matter for Terminal-Bench:

• Some failures are not caused by max_turns; wall-clock AgentTimeoutError and provider response latency can dominate.
• Structured summaries can preserve useful partial progress, but verifier failure output must be included explicitly or Judge can select a superficially good but invalid candidate.
• Final workspace hygiene matters in Terminal-Bench. For example, self-test artifacts left in the target directory can fail hidden tests even when local validation passes.

Stratified Terminal-Bench Pilot

A broader pilot used a stratified set of Terminal-Bench tasks from experiments/tb40_stratified_tasks.txt. Only a subset completed cleanly because some tasks were interrupted by quota limits, semantic-summary failures, or long agent timeouts. Among the completed cases, the workflow showed that second generation rollouts can improve over first generation rollouts on some tasks. Representative improvements observed during development included:

financial-document-processor: 0.1875 -> 1.0
regex-log:                    0.1875 -> 0.8125
reshard-c4-data:              0.3750 -> 1.0
build-cython-ext:             0.1875 -> 0.75

These numbers are from local small-scale engineering runs and should be treated as workflow evidence only. They are not a statistically powered Terminal-Bench score.

Max-Turn Probe

We also ran a focused probe with max_turns=8192 on:

polyglot-c-py
polyglot-rust-c
make-mips-interpreter

The probe showed that increasing max_turns alone does not solve the current failure modes:

• polyglot-c-py generated working-looking solutions, but hidden tests failed because self-test artifacts such as cmain were left in /app/polyglot.
• polyglot-rust-c reached a semantic-summary failure before completing the full PDR loop.
• make-mips-interpreter remained dominated by AgentTimeoutError in both generations.

This was useful because it separated true turn-limit issues from wall-clock timeout, verifier-feedback, and summary/Judge-input issues.

Current Engineering Gaps

The current implementation should be viewed as a working reproduction scaffold. Known gaps before claiming a stronger reproduction result:

• Summary inputs should include verifier stdout/stderr as first-class evidence, not just agent self-validation.
• Judge cards should expose hidden-test failures and final workspace state more directly.
• Provider quota and empty-output behavior need more robust checkpoint/resume handling.
• Long tasks need better timeout-aware scheduling or task-specific execution budgets.
• The system needs larger, repeatable experiments with fixed task splits before reporting benchmark-level conclusions.

Secret Handling

Never commit .env, provider API keys, Terminal-Bench task data, or raw runs/ traces. The repository includes .env.example only.