Model-agnostic eval harness for tool-calling agents. Runs the same state-verified task suite against any chat backend — a local 8B model over Ollama, Claude, or anything that speaks text — and reports pass@k (capability) and pass^k (reliability).
Zero runtime dependencies. TypeScript + bun.
pass@k asks "can the agent do this at least once in k tries?" — the classic capability metric. pass^k, introduced with τ-bench, asks the production question: "does it succeed every time?" An agent that books the right flight 4 times out of 5 has strong pass@k and terrible pass^k — and it's the second number your customers experience.
Both are computed with unbiased estimators over n trials with c successes:
pass@k = 1 - C(n-c, k) / C(n, k)
pass^k = C(c, k) / C(n, k)
--trials 4 --k 2, temperature 0, five built-in tasks:
| provider / model | pass@2 | pass^2 | trials |
|---|---|---|---|
| claude-code / haiku | 1.000 | 1.000 | 20/20 pass |
| ollama / llama3.1:8b | 0.600 | 0.600 | 12/20 pass |
Findings from these runs, all caught by state checks rather than transcript reading:
- Refusal is hard for small models. On
cancel-shipped-refusal(a shipped order that cannot be cancelled — the correct move is to say so), llama3.1:8b issued an unrequested refund in every trial rather than refuse. A transcript reader would call those runs helpful. Haiku refused cleanly 4/4. - Action hallucination. On
cancel-pending-lookup, llama3.1:8b gathered all the right information, never calledcancel_order, then told the customer "Your order O1003 has been cancelled." The database says otherwise; the database is what gets scored. - Prompt brittleness. Adding two contract lines to the system prompt flipped llama3.1:8b on that same task from 4/4 to 0/4 at temperature 0. Small-model evals are as much a measurement of the prompt as of the model — version your prompts like code.
- Task ambiguity punishes careful models. Before the task prompts stated the customer's identity, haiku scored 0/4 on the easiest task — it refused to refund an unauthenticated stranger, which is arguably correct — while llama paid anyone who asked. One identity line later, haiku went 20/20. When a strong model fails your easiest case, suspect the eval before the model.
The first Claude run through the CLI produced a bizarre result: 0/4 on the easiest task. The teed transcript showed why — the Claude Code CLI injects the logged-in account's context (email, date) into every invocation, and the model concluded the operator's identity didn't match the customer's order, so it refused the refund. Perfectly reasonable behavior, completely invalid eval. The fix is --exclude-dynamic-system-prompt-sections in the provider spawn plus an explicit identity rule in the task contract. If you eval through any CLI that decorates prompts, assume contamination until you've read a raw transcript.
git clone https://github.com/MaxHarar/passk && cd passk
bun install
# no model needed — deterministic reference agent, proves the harness
bun src/cli.ts run --provider scripted --trials 3 --k 2
# local model via Ollama (https://ollama.com)
ollama pull llama3.1:8b
bun src/cli.ts run --provider ollama --model llama3.1:8b --trials 4 --k 2
# Claude via the Claude Code CLI (uses your existing `claude` login)
bun src/cli.ts run --provider claude-code --model haiku --trials 4 --k 2
# Anthropic API directly
ANTHROPIC_API_KEY=... bun src/cli.ts run --provider anthropic --model claude-haiku-4-5 --trials 4 --k 2The provider contract is one method. Implement it and the harness runs your model:
interface Provider {
name: string;
chat(messages: Msg[]): Promise<string>;
}Actions are a uniform JSON protocol, not vendor-native function calling. Each turn the model must emit exactly one of:
{"tool": "refund_order", "args": {"order_id": "O1001", "amount": 79.99}}
{"answer": "I've refunded your $79.99 in full."}This is a deliberate v1 design choice: it makes model-agnosticism real (any text backend works, including small local models), keeps the harness dependency-free, and folds format discipline into the measured reliability — an agent that can't hold an output contract is an unreliable agent. Native function-calling adapters are a natural v2.
Tasks are judged on final environment state, not transcripts. Each trial gets a fresh seeded order-database; success means the database ended in the right state (and, for refusal tasks, that nothing was mutated). The built-in suite is an order-support desk with five tasks:
| task | tests |
|---|---|
| refund-delivered | direct tool use with correct amount |
| cancel-pending-lookup | multi-step lookup, acting on the right record |
| refund-damaged-shipped | policy-conditional action (damaged ⇒ full refund) |
| address-change-pending | mutation guarded by order status |
| cancel-shipped-refusal | the trap: correct answer is a refusal + zero state change |
A task is a prompt, an environment factory, and a state check — see src/tasks/index.ts. A provider is one chat() implementation — see src/providers/. The scripted provider replays known-good action sequences and runs in CI on every push, so the harness itself is continuously eval-tested without model costs.
MIT