A gate harness for coding agents. Gatebench wraps a project in deterministic gates, evidence-chain manifests, and independent verifiers, so a fixed model ships work that passes its checks and records the passing as evidence.
The design bet is that reliability can come from outside the model. Hold the model constant, add enforcement around it, and measure whether the output gets more reliable. On this repo's own benchmark it does, by four times.
Same model (Haiku), same 50 policy-trapped drafting tasks, two arms. OFF ships the first draft. ON feeds fatal-policy findings back into a bounded repair loop (at most two retries), then ships the final draft. Both arms are graded by the same gold checks.
| arm | ships a passing artifact | 95% CI |
|---|---|---|
| OFF, first draft | 22% | 12.8–35.2% |
| ON, gates + repair | 86% | 73.8–93.0% |
McNemar exact p = 4.7e-10, discordant pairs b=0 and c=32. The criterion was pre-registered before the run (c > b and p < 0.05) and the git history carries the timestamp. The run cost $2.89 notional, 1.29M tokens, 37 minutes of wall clock. Full method, per-case table, and caveats live in docs/DESIGN.md; the numbers reproduce from eval/ab_report.json.
Two honest readings travel with the headline. The bare model leaks a trapped token almost every time it is tempted (0 of 5 on seven of eight trap cases): when the facts hold a sensitive value or the prompt asks for it, the first draft includes it. One trap case fails 5 of 5 in both arms, because the task itself demands the violation and repair cannot converge; in production the harness refuses the gate and escalates instead of shipping, so this benchmark undercounts the enforcement value.
Gates. A project declares a linear chain of gates. Each gate advances only
when the tool that produced its evidence sets it, so no step is skippable by
assertion. make check is the only definition of done. The reference policy
pack stamped in eval/ runs a five-gate chain.
Evidence-chain manifests. A gate records a sha256 of every file it examined. Edit one of those files afterward and the gate reopens; add a file the gate never saw and it reopens the same way. A gate earned over zero files is refused. The manifest is the receipt, and it is checked, not trusted.
Write-time policy. A PostToolUse hook runs the project's policy checks on
every write. A banned pattern (a national ID on disk, a disclosed client name
in a rendered document) fails at the moment it is written, well before review.
The exit-code contract is strict: 0 pass, 1 fatal, 2 warn, and any other
exit counts as fatal "the check did not run", so a gate is never earned on a
check that crashed.
Independent verifiers. A blinded reviewer subagent scores an artifact against a rubric with no access to the author's justification, then writes a schema-valid, sha256-anchored review into the unit. The reviewer that grades the work never hears the case for it.
Core and adapter. core/ is stamped verbatim into every project as
harness/core/ and stays in sync through gatebench update (hash-tracked,
never hand-edited). harness/adapter/ holds the project's own enforcement and
update never touches it. The harness grows with the project: every observed
failure becomes a check, and a check general enough to reuse gets promoted back
to the template with gatebench upstream.
git clone https://github.com/landtml/gatebench
cd gatebench
make test-core # fast inner loop, 385 tests in ~17s
python3 gatebench.py init ../my-project # interview, then stamp
python3 gatebench.py init ../my-project --defaults # accept every default (CI + agents)
python3 gatebench.py init ../my-project --retrofit # existing or dirty repo, never overwritesA stamped project gets:
harness/core/, the gatebench-owned skeleton, hash-tracked and synced bygatebench updateharness/adapter/, your enforcement:adapter.yaml, executable checks, an oracle, a review rubric- session hooks wired into
.claude/settings.json make checkas the single gate on done, and aPROGRESS.yamlwhose items pass only on a green check
gatebench.py, the CLI:init,update,diff,upstream,gen,add-check,set-gate-commandcore/, the harness skeleton stamped into<project>/harness/core/, pluscore/eval/for the measurement machineryseed/, adapter starters, stamped once at init and never updated aftereval/, the reference corpus (54 synthetic cases) and the policy pack it runs againsttests/, the template's own suite;core/tests/, the core suitedocs/, DESIGN.md, ADAPTER_GUIDE.md, RETROFIT.md, EVAL.md, THREAT-MODEL.mdplan.md, the spec this repo implements
make test-core # fast inner loop, 385 tests in ~17s
make test # test-core plus the template suite, which stamps real projects end to end (~8 min)
make eval # deterministic corpus, free, a regression net
make eval-ab # the paired A/B, spends worker quota, pre-register first
make chaos # fault-injection suite (mock legs), run alone
make fuzz # boundary fuzz suite, run alonemake eval is expected to pass at 100%: the corpus is authored green against
the current policy pack, and its job is to catch regressions. make eval-ab
spends real model time and is never run implicitly; pre-register the criterion
in docs/DESIGN.md before the run.
Linux and macOS (POSIX). The core relies on POSIX semantics: os.replace
atomicity, fcntl advisory locks, and signal-based process control. On Windows
the doctor refuses and WSL is the supported path. macOS is expected to work and
awaits one full suite run as its evidence, since evidence over opinion applies
to platform claims too.
Python 3 standard library and PyYAML. The core takes no other dependency and imports no Claude Code plugin or skill, so it stays self-contained by design.
0.4.0-dev. The core suite and the deterministic eval are green, and the A/B
has one confirmatory run on record (above). Interfaces can still move before a
1.0 tag. Changes are logged in CHANGELOG.md.
MIT, © 2026 landtml. See LICENSE.