Make the coding-agent RL loop train on ROCm/MI355X

[lizamd]

Includes @aoshen02's local multi-turn smoke test (aoshen02#4) with his agreement, then makes the loop actually train on ROCm/MI355X.

Why it couldn't train

The smoke test runs end to end but cannot learn on any platform:

--n-samples-per-prompt 1
--rollout-temperature 0.0

GRPO computes its advantage within a prompt's sample group. One sample gives an advantage of identically zero; greedy decoding makes n samples identical, so group variance is zero too. Both have to change — fixing either alone still gives no gradient.

What's in here

• Working GRPO defaults, with the measurements behind them in the commit messages
• Prompt truncation instead of returning an empty TurnRecord on overflow (the empty reply made the CLI exit 1, so the overflow and the agent_exit_code=1 after it were one bug)
• Tool-call repair before vLLM's parser runs — its Hermes parser doesn't raise, it logs and returns tools_called=False, so post-hoc recovery hid the failure instead of fixing it
• max_turns_per_sid wired through; it was fully implemented but generate.py never passed it
• Dockerfile.sympy-23950 fixed — it no longer builds anywhere since bullseye went EOL and its security Release file expired
• IMAGE and WEIGHT_TRANSPORT parameterised

Context budget

Turn 1 spent 24118 of a 40960-token window before any conversation accumulated: 6315 system prompt, 18730 tool schemas across 19 tools. A SWE task uses three (Bash, Read, Edit = 3848). Dropping the rest takes turn-1 to 9878 and usable turns from 13 to 59. A 262144-token window hides this; it appears immediately on a smaller model.

Verification

vllm/vime-rocm:coding-agent-rl, 8×MI355X, Qwen3-32B: 4 episodes, nccl weight sync 6.5s, 0 context overflows, 0 agent_exit_code=1, 0 OOM.

The training behaviour behind the defaults was measured on the pre-rebase tree — a 30-step run where pass rate rose 0.254 → 0.596 against a 0.3125 baseline (t = +4.19, df=28), entropy rising rather than collapsing. That has not been re-collected on this rebase; the run above only confirms the branch starts, rolls out, and syncs weights.

🤖 Generated with Claude Code

[read-the-docs-community]

Documentation build overview

📚 vime | 🛠️ Build #34463578 | 📁 Comparing 56959c4 against latest (ce92eff)

  🔍 Preview build  

45 files changed · ± 45 modified

± Modified

• genindex.html
• index.html
• search.html
• advanced/arch-support-beyond-megatron.html
• advanced/delta-weight-sync.html
• advanced/external-rollout-engines.html
• advanced/fault-tolerance.html
• advanced/low-precision.html
• advanced/megatron-config.html
• advanced/observability.html
• and 35 more...

[gemini-code-assist]

Code Review

This pull request introduces a local multi-turn smoke test environment for coding agents, including local Docker sandbox execution, run scripts, and enhanced tool-call parsing to handle invalid JSON escapes. The review feedback highlights several critical areas for improvement: addressing hardcoded paths to ensure portability, preventing subprocess leaks on cancellation, ensuring failed executions are properly traced, fixing potential negative budget calculations for small context windows, storing session-specific state correctly, and refining token count calculations and regex patterns for robust parsing.

[gemini-code-assist]

Parameterize or generalize the hardcoded path /mnt/m2m_nobackup/lizli102/vime-agent-smoke to make the script portable for other users.

Suggested change

	ROOT=${ROOT:-/mnt/m2m_nobackup/lizli102/vime-agent-smoke}
	ROOT=${ROOT:-${HOME}/vime-agent-smoke}

[gemini-code-assist]

Avoid hardcoding the user home directory /home/lizli102 in volume mounts. Use ${HOME} or a parameter instead to ensure portability.

Suggested change

	-v /home/lizli102/vime:/root/vime \
	-v "${ROOT}/models:/work/models" \
	-v "${ROOT}/assets:/work/assets" \
	-v "${ROOT}/tasks:/work/tasks:ro" \
	-v "${ROOT}/runs:/work/runs" \
	-v /home/lizli102:/host \
	-v "${HOME}/vime:/root/vime" \
	-v "${ROOT}/models:/work/models" \
	-v "${ROOT}/assets:/work/assets" \
	-v "${ROOT}/tasks:/work/tasks:ro" \
	-v "${ROOT}/runs:/work/runs" \
	-v "${HOME}:/host" \

[gemini-code-assist]

If _run is cancelled (e.g., due to a timeout in asyncio.wait_for inside exec), the spawned subprocess is not terminated and will leak as a zombie or background process. Use a try...finally block or catch asyncio.CancelledError to terminate the process.

Suggested change

	async def _run(*argv: str, check: bool = False) -> ExecResult:
	process = await asyncio.create_subprocess_exec(
	*argv,
	stdout=asyncio.subprocess.PIPE,
	stderr=asyncio.subprocess.PIPE,
	)
	stdout, stderr = await process.communicate()
	result = process.returncode, stdout.decode(errors="replace"), stderr.decode(errors="replace")
	if check and process.returncode != 0:
	raise RuntimeError(f"command failed ({process.returncode}): {' '.join(argv)}\n{result[2]}")
	return result
	@staticmethod
	async def _run(*argv: str, check: bool = False) -> ExecResult:
	process = await asyncio.create_subprocess_exec(
	*argv,
	stdout=asyncio.subprocess.PIPE,
	stderr=asyncio.subprocess.PIPE,
	)
	try:
	stdout, stderr = await process.communicate()
	except asyncio.CancelledError:
	try:
	process.terminate()
	await process.wait()
	except ProcessLookupError:
	pass
	raise
	result = process.returncode, stdout.decode(errors="replace"), stderr.decode(errors="replace")
	if check and process.returncode != 0:
	raise RuntimeError(f"command failed ({process.returncode}): {' '.join(argv)}\n{result[2]}")
	return result

[gemini-code-assist]

When check=True is passed to exec, any command failure will raise a RuntimeError inside _run, bypassing the self._trace call entirely. This prevents failed executions from being logged in the sandbox events trace. Call _run with check=False and manually check the return code after tracing.

Suggested change

	result = await asyncio.wait_for(self._run(*argv, check=check), timeout=timeout)
	self._trace(
	"exec",
	user=user,
	cmd=cmd,
	returncode=result[0],
	stdout=result[1],
	stderr=result[2],
	)
	result = await asyncio.wait_for(self._run(*argv, check=False), timeout=timeout)
	self._trace(
	"exec",
	user=user,
	cmd=cmd,
	returncode=result[0],
	stdout=result[1],
	stderr=result[2],
	)
	if check and result[0] != 0:
	raise RuntimeError(f"command failed ({result[0]}): {' '.join(argv)}\n{result[2]}")
	return result

[gemini-code-assist]

If session.max_context_tokens is small (e.g., less than 256), _reserve will be larger than max_context_tokens, resulting in a negative _budget. This causes incorrect slicing and potential errors. Cap _reserve at half of max_context_tokens to ensure _budget remains positive.

        _reserve = min(1024, max(256, session.max_context_tokens // 8))
        _reserve = min(_reserve, session.max_context_tokens // 2)
        _budget = session.max_context_tokens - _reserve

[gemini-code-assist]

_logged_schema is set on the adapter instance, which is a shared singleton across all sessions. This prevents the schema from being logged for any subsequent sessions/tasks that might have different tools or system prompts. Store _logged_schema on the session object instead.

Suggested change

	if _n <= 1 and not getattr(adapter, "_logged_schema", False):
	adapter._logged_schema = True
	if _n <= 1 and not getattr(session, "_logged_schema", False):
	session._logged_schema = True

[gemini-code-assist]

_sys is a string representing the system prompt. Using json.dumps(_sys) wraps it in quotes and escapes characters, which inflates the token count and makes the logged system prompt size inaccurate. Encode the raw string directly.

Suggested change

	_st = len(adapter.tokenizer.encode(json.dumps(_sys))) if _sys else 0
	_st = len(adapter.tokenizer.encode(_sys)) if _sys else 0

[gemini-code-assist]

If VIME_LOCAL_SANDBOX_TRACE_DIR is not set in the environment, _trace_dir() will raise a KeyError and crash the script. Handle this gracefully by providing a default fallback path (e.g., /tmp/vime-trace).

Suggested change

	def _trace_dir() -> Path:
	path = Path(os.environ["VIME_LOCAL_SANDBOX_TRACE_DIR"]) / _instance_id.get()
	path.mkdir(parents=True, exist_ok=True)
	return path
	def _trace_dir() -> Path:
	path = Path(os.environ.get("VIME_LOCAL_SANDBOX_TRACE_DIR", "/tmp/vime-trace")) / _instance_id.get()
	path.mkdir(parents=True, exist_ok=True)
	return path

[gemini-code-assist]

The _ESCAPE_FIX_RE negative lookahead prevents matching \u entirely. However, if the model outputs an invalid \u escape (such as \user or \utility in file paths), json.loads will still fail with a JSONDecodeError. Update the regex to only skip \u if it is actually followed by 4 hex digits.

Suggested change

	_ESCAPE_FIX_RE = re.compile(r'\\(?!["\\\\/bfnrtu])')
	_ESCAPE_FIX_RE = re.compile(r'\\(?!["\\\\/bfnrt]|u[0-9a-fA-F]{4})')