TallyGuard

Process-aware agent harness and evaluation for τ²-bench.

Status: in progress. The harness is being built; no experimental results exist yet. Nothing in this README reports a measurement. Claims will appear here only when they are backed by raw traces committed to results/.

---

The problem

τ²-bench scores an agent on its end state — does the final database match the target, plus (for retail) an LLM-judged natural-language assertion. That is a sound and deliberate design choice, but it is blind to how the agent got there.

Upstream documents the gap themselves in sierra-research/tau2-bench#298: across 2,624 simulations, 11 of 1,755 (0.6%) trajectories that scored a perfect reward had violated the domain's explicit "obtain user confirmation before mutating the database" rule. The end state was right; the process was not.

This gap was found and reported by others. TallyGuard reproduces it and builds the machinery to measure it — it does not claim the discovery.

TallyGuard adds a process layer on top of τ² without forking it:

• a deterministic action gate — schema validation and an independent, no-drafting-authority veto that runs before any tool call is emitted
• semantic idempotency (tally) over the 7 state-mutating retail tools
• a policy oracle that judges trajectory prefixes, not just final state
• fault injection wired so that τ²'s own evaluation replay stays consistent

The question it answers

Does a cheap, rule-bound, non-drafting independent vetoer reduce the rate of policy-violating actions at runtime — and what does it cost in success rate, money and latency?

The control arm is no vetoer, not a strong-model critic. Both arms run the identical planner, validator, tally, budget and trace machinery; only the veto implementation class differs. A negative result is a valid result and will be reported as one.

Boundaries

Stated up front, because they limit what this project can be cited for:

• The experimental domain is τ² retail customer service. It is not a creator/seller or affiliate setting. What transfers is the runtime and evaluation layer, not the domain policy or task distribution.
• No production deployment, no GMV or business metrics, no multilingual claims.
• τ² owns the environment, the database, the user simulator and reward computation. TallyGuard owns everything inside the agent turn. The architecture diagram makes the seam explicit (see docs/).
• The run ledger is a session state ledger. It is not called "memory" anywhere in this repository until the memory stress test defined in the spec actually passes.

Layout

src/tallyguard/     harness: planner, validator, veto, tally, ledger, trace, replay
vendor/tau2-bench/  upstream, as a pinned submodule — never modified
scripts/            freeze.py (pins upstream state), inventory.py (task/tool census)
docs/adr/           architecture decision records, each backed by raw run output
FROZEN.md           every pinned hash, regenerable by one command
results/            raw JSONL traces and summary CSVs

Reproduce

git clone --recurse-submodules https://github.com/yhnapoleon/TallyGuard.git
cd TallyGuard
uv sync --extra dev
uv run pytest

uv run pytest needs no API key, no network and no spend. That is enforced, not merely intended: tests/conftest.py disables outbound sockets for the whole default suite, and tests assert both that the socket guard still bites and that live-marked tests really are deselected. Tests that genuinely need a provider or a full τ² simulation run only under -m live.

Offline runs are served by a recording provider. Responses are keyed on the full request — model, messages, tools, schema, temperature, seed — and stored one JSON file per key, so a cassette can be read, diffed, or hand-written to construct a case a live model would be hard to coax into producing:

uv run pytest                                          # replay; the default
TALLYGUARD_PROVIDER_MODE=record uv run pytest -m live   # call out, save responses

A cassette miss in replay mode is an error, never a fallback to the network. A silent fallback is how a free test suite starts costing money without anyone noticing, and how a deterministic run quietly stops being one.

Upstream state is pinned in FROZEN.md and verified by:

uv run python scripts/freeze.py --check

Design provenance

The control structure is modelled on a specific historical arrangement: the Ming grand secretariat drafted edicts (piàonǐ), while the Six Offices of Scrutiny held a low-rank power to reject but not to redraft (fēngbó), checking proposals against a frozen body of precedent. The idempotency-plus-authorization token (kānhé, a split tally) comes from the same period's maritime trade permits.

That framing produced exactly one mechanism that a purely technical design would have been unlikely to reach: pairing the semantic idempotency key with an authorization token issued by a party that cannot itself act. Everything else here stands on its own engineering rationale, and every argument in the spec holds with the historical names deleted.

License

MIT. Upstream τ²-bench is MIT-licensed and is included unmodified as a submodule.