Launch post: Introducing Bosun →
The benchmark for the verb that keeps a knowledge graph alive: judge. As agents accumulate memory as knowledge graphs, something has to decide which proposed connections are warranted — or the graph grows into noise that drowns the model reading it. WarrantBench measures that judgment, with a judge you program with a sentence. From Hanno Labs.
A judge is any system — a frontier LLM, an embedder, or a specialized rater — that, given a proposed edge between two pieces of evidence and an instruction, outputs a calibrated verdict in
[0,1]. WarrantBench measures whether the verdict follows the instruction.
Agents are accumulating memory as knowledge graphs, and the field already scores how they represent knowledge (MTEB), retrieve it (BEIR), recall it (LoCoMo), and reason over it (ProntoQA). It scores none of the verb that keeps a graph alive: judge. Every connection an agent proposes has to be judged — is this edge warranted? supported, non-redundant, still-true — or the graph accretes noise until it overwhelms the model that has to read it back. Nothing measures that capability today. It is the gap that matters most right now, and it is the one WarrantBench is built for.
"Warranted" is not a single fixed rule, and we don't impose ours. In one graph an edge earns its place by naming the same entity; in another by bridging two distant domains; in another by not duplicating an edge already there, or by still being supported by current evidence. So WarrantBench doesn't bake in a notion of a "good edge" — it tests a judge you program with a sentence: you supply the rule, the judge obeys it, and it generalizes to rules it never trained on. That same capability is exactly what RAG filtering, content moderation, and deduplication need too — knowledge-graph curation is simply where the need bites first and hardest.
FollowIR measures one specific instruction — relevance: whether a retriever follows a description of what makes a document relevant to a query. Relevance is a fine opinion, but it is one opinion, the IR community's. WarrantBench measures the rest: whether a judge has no opinion of its own and executes an arbitrary rule — same-entity, cross-domain bridge, dedup, retention — including rules with nothing to do with relevance. Relevance is covered; this covers everything else you might think to ask.
The thing that makes this a stable, rerunnable benchmark: gold is local to an instruction. The same
pair is gold-1 under one rule and gold-0 under another. So we never grade "is this a good edge" (which
has no model-independent answer and rots the moment a judge defines it). We grade instruction-following:
given this instruction, did the judge produce the label the instruction entails?
All gold is construction-truth — derived deterministically from frozen cosine similarity and deterministic entity extraction, never from a live judge. It is built once, frozen, and scored with no model in the loop, so it reruns free and identical forever.
Each item is a (instruction, query, document, label) row. The document holds two findings; the
instruction defines what "connected" means for that row.
| family | rule | gold |
|---|---|---|
topicality |
same broad topic? | cosine high vs low |
topicality_neg |
NOT the same topic? | inverse of topicality (the negation test) |
entity |
share a specific named entity? | shared rare proper-noun / ticker |
bridge |
different topic yet a shared specific link (a cross-domain bridge)? | shared entity and low cosine |
steer |
the same pair under opposite rules → gold must flip | the headline steerability test |
A separate novel-rule split (items_novel.jsonl) holds rules the reference model never trained on
("both mention a figure ≥ $1B", "both involve a government or regulator") — to prove the following
generalizes, not the specific rules.
Pure functions of the per-item scores (see warrantbench/score.py):
- per-instruction AUROC — under each instruction in isolation, can the judge separate yes from no?
- steerability (headline) — on a
steertriple (one fixed pair, opposite rules), does the score for the rule whose gold is1exceed the score for the rule whose gold is0? A pure-similarity judge can't read the rule, so it scores 0.00 by construction — that's the point. - sensitivity — how much a judge's score moves across instructions on one pair. Cosine →
0.
The data lives on the Hugging Face Hub and is loaded at runtime, pinned to a dataset revision — code in
git, data on HF (the MTEB convention). score.py carries the pinned DATASET + REVISION, so an eval is
reproducible and the gold can't silently drift under it.
| config | split | rows | what |
|---|---|---|---|
default |
test |
1,280 | the four trained relation families + steerability triples |
novel |
test |
720 | held-out rules the model never trained on (generalization) |
Hanno-Labs/warrantbench · pinned to
aa052bd. Each row is self-contained:
{"id": "it00000", "task": "topicality", "instruction_id": "topicality",
"instruction": "Connected ONLY if the two findings are about the SAME broad topic ...",
"query": "These two findings share the specified relationship.",
"finding_a": "...", "finding_b": "...", "label": 1,
"pair_id": null, "cosine": 0.5685, "shared_entities": []}The document a judge is scored on is, exactly:
<Instruct>: {instruction}
<Query>: {query}
<Document>: FINDING A:\n{finding_a}\n\nFINDING B:\n{finding_b}
finding_a / finding_b are already truncated to the scored budget, so no further processing is needed.
The harness downloads the data from the Hub on first run (needs datasets); --config picks the split.
# 1. cosine baseline — the instruction-blind control (steerability must be 0.0)
python warrantbench/score.py --judge cosine --config default
# 2. a frontier LLM, instruction stated as a rule (needs litellm + a provider key)
python warrantbench/score.py --judge llm --config default --llm-model gemini/gemini-3.1-flash-lite
# 3. the Bosun rater over HTTP (instruction in the <Instruct> slot)
BOSUN_URL=https://<your-endpoint> BOSUN_TOKEN=... \
python warrantbench/score.py --judge bosun --config default
# the held-out novel-rule split (generalization)
python warrantbench/score.py --judge cosine --config novelA judge is any callable rows -> list[float in 0..1]. The three references in score.py —
cosine_judge, bosun_judge, llm_judge — are the template for adding your own.
The reference judge is Bosun (Hanno-Labs/bosun-xs ·
Hanno-Labs/bosun-4b), a programmable relevance/relation judge built
on Qwen3-Reranker. On WarrantBench it out-steers a frontier LLM:
| cosine | Bosun-XS (0.6B) | gemini-3.1-flash-lite | |
|---|---|---|---|
| steerability — score flips with the rule | 0.00 | 0.94 | 0.58 |
| negation — "NOT the same topic" | 0.00 | 0.97 | 0.996 |
| cross-domain bridge | 0.32 | 0.83 | 0.38 |
On the novel rules it never trained on:
| novel rule (never trained) | Bosun-XS | flash-lite |
|---|---|---|
| "both mention a figure ≥ $1B" | 0.95 | 0.35 |
| "both involve a government or regulator" | 0.95 | 0.63 |
A larger general model carries a stronger but more diffuse prior; on the axis that matters — following the
instruction you gave — the small specialist wins. Splits where a baseline wins (e.g. flash-lite's 0.996
on negation) are kept and reported: a benchmark whose authors sweep every split is the credible kind.
The build is deterministic and fully auditable. The raw inputs (corpus.json, embeddings.json,
validated_pairs.json) live under the dataset's build/ path on the Hub at the pinned revision; the
scripts below fetch them and rebuild the items bit-for-bit:
python build/build_items.py # fetch corpus + embeddings from HF -> build/items.json (deterministic)
python build/build_novel_items.py # -> build/items_novel.json
python warrantbench/materialize.py --src build --out data # join with corpus -> self-contained data/*.jsonldata/ and the fetched build/*.json are git-ignored — the published copy is the Hub dataset.
@misc{warrantbench2026,
title = {WarrantBench: A Benchmark for Programmable, Opinion-Free Relational Judgment},
author = {Hanno Labs},
year = {2026},
url = {https://github.com/Hanno-Labs/warrantbench}
}Code: MIT (see LICENSE). Data: released for research evaluation use.