recursive-lm-harness

Find where a coding agent went wrong across a transcript far too big to read, then propose how to fix it.

An auto-improve harness built on the Recursive Language Models (RLM) idea, using the OpenAI Agents SDK.

You hand it a coding agent's transcript (trace.jsonl) — millions of lines, hundreds of millions of tokens, far larger than any context window. The harness builds an agent that mines that transcript to find recurring failure modes (where the coding agent went wrong) and proposes concrete, testable improvements to the coding agent's policy. There is no answer key. It is doing real diagnosis.

root agent ──drives──► windowed trace tools ──over──► trace.jsonl (byte-indexed)
     ▲                          │
     │  small structured        │  search / read rows / coverage gaps
     │  result only             ▼
     └──────── recursive sub-agents (depth-capped) ──reduce──► ImprovementsToBeMade

Why RLM

A multi-million-line transcript can't be pasted into a model. The RLM move: the root agent never reads the whole transcript. It drives windowed tools over a byte-indexed view of the file, slices it into ranges, and spawns recursive sub-agents on each range, then merges their findings. Tool output back to the model is hard-truncated to a few KB — the "narrow waist". You can't dump the transcript, so you must compute a small result (matched rows, a count, a reduced summary).

Everything runs on the OpenAI Agents SDK (agents.Agent + agents.Runner); there is no raw chat.completions.

Background reading: Zhang & Khattab, "Recursive Language Models" — the environment-plus-sub-model-calls pattern this mirrors.

What's in the transcript

A mostly-benign coding-agent log (read_file / apply_patch / run_tests / run_cmd / final). Woven into it are recurring failure modes you hunt:

Failure mode	Token that marks it	What went wrong
patch-without-read	PATCH REJECTED	patched a file never read in that task (stale hunks)
done-despite-failing-tests	despite failing tests	declared a task complete with tests still red
retry-thrash	thrash	retried the identical failing command over and over
hallucinated-api	frobnicate	called an API/symbol that doesn't exist
no-verification	(unverified)	declared "done" without running anything to verify

Architecture

Module	Responsibility
generate_trace.py	Synthesise the coding-agent transcript (50K–2M lines).
trace_view.py	TraceView: byte-indexed windowed access over the rows.
repl.py	ReplEnvironment: persistent, memory-capped Python scratchpad.
models.py	ImprovementsToBeMade + deterministic merge + HarnessContext.
tools.py	Trace/REPL tools (narrow-waisted) + the depth-capped recursive sub-agent.
harness.py	Root agent + run_harness() entry point + report formatter.

Three design decisions are the whole point:

1. The narrow waist. Every tool truncates its output to ~6KB (MAX_OUTPUT_CHARS). The model physically cannot pull the trace into context; it has to drive computation and accept small results.
2. The merge is deterministic. merge_improvements reduces a whole tree of findings in pure Python — dedupe by failure_mode, sum occurrences, union evidence, keep the highest severity. The LLM only ever produces leaf findings; it never does the reduce.
3. Recursion is capped in Python, not the prompt. delegate_to_sub_agent reads depth from a HarnessContext threaded through every Runner.run. At MAX_DEPTH the tool refuses and tells the agent to do the work itself. The model cannot recurse past the cap no matter what it decides.

Setup

Requires uv and Python 3.11+.

git clone https://github.com/jamesaphoenix/recursive-lm-harness.git
cd recursive-lm-harness
uv sync                                 # install deps + dev tools

# OPENAI_API_KEY via 1Password (edit the op:// path to wherever your key lives)
op inject -i .env.dev -o .env
# ...or just: echo "OPENAI_API_KEY=sk-..." > .env

Generate a trace

uv run rlm-trace --quick                # 50,000 lines, fast — start here
# uv run rlm-trace                      # 2,000,000 lines — the real thing

Run the harness

uv run rlm-harness
uv run rlm-harness "Focus on apply_patch failures across the whole trace."
uv run rlm-harness --trace ./trace.jsonl --max-turns 30

It prints the merged ImprovementsToBeMade — one entry per failure mode, with evidence (row indices / task ids), an occurrence count, a severity, and a concrete proposed policy change. The live recursion tree streams to stderr.

Self-check (after you've done the analysis)

uv run rlm-trace --quick --reveal       # prints planted counts + canonical fixes

The planted counts in the quick trace: PATCH REJECTED ×12, "despite failing tests" ×7, thrash ×9, frobnicate ×5, (unverified) ×11.

Use as a library

import asyncio
from rlm_harness import merge_improvements
from rlm_harness.tools import init_state
from rlm_harness.harness import run_harness

init_state("trace.jsonl")               # load the trace, build the byte index
report = asyncio.run(run_harness("Diagnose this transcript."))
print(report.model_dump_json(indent=2))

Development

This project uses just as the command runner.

just            # list recipes
just trace      # generate the quick trace
just test       # run the test suite with coverage
just lint       # ruff check
just typecheck  # mypy --strict
just check      # lint + typecheck + test (what CI runs)

Or call the tools directly: uv run pytest, uv run ruff check ., uv run mypy.

The test suite covers the deterministic core (the byte-indexed TraceView, the memory-capped REPL, the merge reducer, the trace generator, and the Python-enforced recursion gate) and does not require an API key — those parts are where the real engineering lives, so they're where the tests live.

License

MIT — see LICENSE.