Hi — I'm the author of #194/#195. While reading the support matrix and the Care Prep page I kept coming back to one gap, and I'd like to propose filling it. Per CONTRIBUTING, describing before building.
Where this starts
The Care Prep page already shows that on one and the same task, harness × model configurations differ by 3.2× in end-to-end latency and 99.8% in cost, and it says more tasks and evaluation methods will follow. I went looking for the piece that would let someone reproduce that measurement, or run it on a task set they didn't write themselves. As far as I can tell it doesn't exist yet: scripts/support-matrix measures whether a harness works (five scenarios, four rules, all enforced in code), protocol/conformance measures whether an implementation speaks UHP, and the Care Prep task and its scripts aren't in the repository. The README's benchmark figure rests on a single private task.
I think this is exactly the moment to build a unified, standard way of evaluating harnesses. The reason is not hypothetical any more. Harness-Bench (arXiv 2605.27922, May 2026) ran 106 oracle-checked tasks across model–harness pairings and found "substantial variation in completion, process quality, efficiency, and failure behavior"; its conclusion is that capability should be reported at the model–harness configuration level rather than attributed to the base model alone. Other write-ups this year put the harness effect at 10–20 points on identical weights. Everyone measures models; almost nobody measures the harness, because almost nobody has eleven of them behind one API with the served model stamped on every turn. HarnessRouter does. The infrastructure is already there — the sandbox, the canonical event stream, the relay's served-model stamp, the artifact capture, the checkpoint/recycle path — and I'd like to put it to that use.
How I chose the task suites
To evaluate a harness (not just the model inside it) the tasks have to exercise the things a harness is responsible for: memory across a long session, long-horizon work that doesn't fit in one shot, tool calling, self-checking (does the agent verify its own output before declaring done), and reasoning efficiency (tokens, turns, wall time to get there). At the same time this must not turn into a second project inside the repository — every suite has to run in the existing sandbox image (python 3.12 + node 22 + git) or on the sandbox's docker network, grade deterministically with no LLM judge, and reuse the matrix's plumbing rather than bring its own. That ruled out a lot: GDPval (pairwise human/LLM grading), SWE-bench Verified (saturated, and one Docker image per instance), τ-bench/AppWorld/WebArena (user simulators, browsers), MLE-bench (GBs of data per task). What survived, and what each one is there to measure:
| Suite |
What it exercises |
Why it fits the sandbox |
Grading |
Headroom |
| DABstep hard (Adyen/HF) |
multi-document reasoning: rules live in Markdown manuals, data in CSV/JSON, and the answer needs both; reasoning efficiency baseline |
a handful of files, no image change |
factoid answers, scorer ships with the suite |
not saturated |
| SpreadsheetBench 2 (2026-06) |
artifact production + self-checking: cross-sheet workflows that must come out structurally coherent |
xlsx in, xlsx out; the artifact row already captures the file; needs only openpyxl |
cell comparison |
new, low scores |
| RoadmapBench (2026-05), Python + TypeScript subset |
long-horizon + memory: implement the next release from a V_old snapshot, median oracle patch ~3,700 lines across modules |
repo snapshot + tests into the workspace |
test.sh → reward in [0,1], plus resolved rate |
best published model 39.1% |
| TheAgentCompany (NeurIPS 2025) |
tool use across systems + memory: GitLab, OwnCloud, RocketChat, Plane, with NPC "colleagues" |
four services via docker-compose on the sandbox's network, no external internet |
checkpoint-based, with partial credit |
best model 30% |
| Toolathlon (ICLR 2026), optional |
tool calling through MCP across 32 real apps — this maps directly onto the Plugins sub-protocol shipped in #175/#181/#185 |
app containers via compose |
state checks |
frontier <55% |
Five packs, each answering a different question about the harness. I'd start with DABstep hard because it is the cheapest to run, needs zero image work, and its scorer is already the kind the matrix trusts; RoadmapBench and TheAgentCompany are where the harness differences should actually show.
What I'd build
scripts/benchmark/, beside support-matrix, reusing what's there: run.mjs for driving sessions, samemodel.py for the served-model rule, render.py for the table. New parts:
- Task packs — a loader per suite that stages the task's files into the workspace, sends the instruction as the first turn, collects the answer/artifacts, and runs the suite's own grader. Follow-up turns where the suite defines them (RoadmapBench's test feedback, TheAgentCompany's multi-step tasks) go through the same session so memory is actually tested, not simulated.
- Per-run metrics from the turn record — resolved/reward, wall time, input/output tokens, turns, tool calls, served model, and the failure line on failure. All of it is already stored; nothing new is instrumented.
docs/benchmark.md in the matrix's shape: one row per harness × model × pack, notes per column, the rule that produced each verdict stated next to it. Matrix rules 1 and 2 apply unchanged — a turn served by another connection or another model is a finding, never a score.
Questions before I write code
- Is this welcome in the repository, or do you see it living on the harnessrouter.ai side with Care Prep?
- Any objection to the five packs, or a preference on order? I'd run DABstep hard first and bring the numbers here before anything is written up as a result.
- RoadmapBench and Terminal-Bench 2 are both published in Harbor format. A Harbor agent adapter for HarnessRouter (external-driver mode, like Terminus-2) would make every Harbor-format suite runnable at once — worth its own proposal, or out of scope?
I'd run the pilot on a self-hosted instance at my own cost.
Hi — I'm the author of #194/#195. While reading the support matrix and the Care Prep page I kept coming back to one gap, and I'd like to propose filling it. Per CONTRIBUTING, describing before building.
Where this starts
The Care Prep page already shows that on one and the same task, harness × model configurations differ by 3.2× in end-to-end latency and 99.8% in cost, and it says more tasks and evaluation methods will follow. I went looking for the piece that would let someone reproduce that measurement, or run it on a task set they didn't write themselves. As far as I can tell it doesn't exist yet:
scripts/support-matrixmeasures whether a harness works (five scenarios, four rules, all enforced in code),protocol/conformancemeasures whether an implementation speaks UHP, and the Care Prep task and its scripts aren't in the repository. The README's benchmark figure rests on a single private task.I think this is exactly the moment to build a unified, standard way of evaluating harnesses. The reason is not hypothetical any more. Harness-Bench (arXiv 2605.27922, May 2026) ran 106 oracle-checked tasks across model–harness pairings and found "substantial variation in completion, process quality, efficiency, and failure behavior"; its conclusion is that capability should be reported at the model–harness configuration level rather than attributed to the base model alone. Other write-ups this year put the harness effect at 10–20 points on identical weights. Everyone measures models; almost nobody measures the harness, because almost nobody has eleven of them behind one API with the served model stamped on every turn. HarnessRouter does. The infrastructure is already there — the sandbox, the canonical event stream, the relay's served-model stamp, the artifact capture, the checkpoint/recycle path — and I'd like to put it to that use.
How I chose the task suites
To evaluate a harness (not just the model inside it) the tasks have to exercise the things a harness is responsible for: memory across a long session, long-horizon work that doesn't fit in one shot, tool calling, self-checking (does the agent verify its own output before declaring done), and reasoning efficiency (tokens, turns, wall time to get there). At the same time this must not turn into a second project inside the repository — every suite has to run in the existing sandbox image (python 3.12 + node 22 + git) or on the sandbox's docker network, grade deterministically with no LLM judge, and reuse the matrix's plumbing rather than bring its own. That ruled out a lot: GDPval (pairwise human/LLM grading), SWE-bench Verified (saturated, and one Docker image per instance), τ-bench/AppWorld/WebArena (user simulators, browsers), MLE-bench (GBs of data per task). What survived, and what each one is there to measure:
test.sh→ reward in [0,1], plus resolved rateFive packs, each answering a different question about the harness. I'd start with DABstep hard because it is the cheapest to run, needs zero image work, and its scorer is already the kind the matrix trusts; RoadmapBench and TheAgentCompany are where the harness differences should actually show.
What I'd build
scripts/benchmark/, besidesupport-matrix, reusing what's there:run.mjsfor driving sessions,samemodel.pyfor the served-model rule,render.pyfor the table. New parts:docs/benchmark.mdin the matrix's shape: one row per harness × model × pack, notes per column, the rule that produced each verdict stated next to it. Matrix rules 1 and 2 apply unchanged — a turn served by another connection or another model is a finding, never a score.Questions before I write code
I'd run the pilot on a self-hosted instance at my own cost.