Progress log

Session-by-session record of what was actually done, newest first. README.md is the plan; this file is the history.

Append an entry before ending any working session. The reasoning behind a decision is what gets lost over 22 weeks, not the code — so write down why, not just what. The entry format is in CONVENTIONS.md.

Sessions

2026-10-31 (late night) - the human loop, K = 6 contexts, first live G1, and a rehearsal of everything after annotation

Done (1,953 tests collected, full suite green)

benchmark/human_loop.py: propose, apply reviewed verdicts, merge, collect-candidates. Only a person's accept or edit changes the corpus; a second annotator can only hold an item back; nothing is overwritten. Review items for the real corpus are ready in benchmark/data/review_items/ (18 branch proposals, 35 persona proposals, 10 unreviewed candidates).
python -m benchmark.contexts: expands a corpus to K = 6 with same-site BM25 hard negatives (70 of 128 instances; the 54 synthetic splits are not traceable to a document and are left as they are).
First live G1: B0 596, B3 509, B7c 1,078, O3 2,749 tokens per instance; the planned matrix is about 111K tokens, 0.6 days of quota. The gate stays open (3 instances, proposed branches, reasoning setting not frozen).
The client now waits out Groq's own "try again in Ns" hint; before, four short backoffs failed inside one minute.
A rehearsal of the whole post-annotation chain on scratch copies (accept-all stand-in verdicts) ran end to end: apply, merge, re-split, K = 6, four local cells on S1 dev, make_results.

Decisions / findings (D7.3-D7.5)

The first persona proposals stated the answer about half the time; prompt and validator fixed (0 of 67 now).
G1 required personas it never uses; fixed. A payment-due paraphrase class was added to the value matcher (health-surcharge outcomes); annotators are told how to phrase outcomes so they compare safely.

Blocked / open

Unchanged: humans have not started. Review the three item files, apply, merge; then re-split, G1 on the real S1/dev set (freeze the reasoning setting), V1 on real dev data, the V2 audit, the freeze.

2026-10-31 (night) - scoped precision, the V1 and V2 drivers, the release bundle, the generated datasheet

Done (1,911 tests collected, full suite green, exit 0)

metrics/scoped.py: each claim in an answer is labelled aligned / misattached / unsupported / extraneous against the gold branches (same value matcher as CCA plus an attachment test); an optional EVAL-tier judge may only rescue a claim the matcher could not read. metrics/validity.py now evaluates three scoped criteria (gold over every-passage concatenation, over the grounding passages, over gold plus filler) where it had a placeholder.
experiments/validity_battery.py: the V1 driver. Dev or train only (the test split is refused), and any instance lacking personas, branches or reference answers is an error naming its id.
experiments/reader_audit.py driver: sample runs the real reader over the battery outputs (and any cell files), draws the stratified sample and writes blind review items; evaluate scores V2 against two annotators, leaves disputes for adjudication and calibrates tau. Run live on four mock instances: 96 decisions explained.
release/bundle.py: text of non-open passages is withheld and recorded as URL, offsets and SHA-256; rehydrate restores it from documents the user fetches and reports drift instead of accepting changed text. Run on the real corpus: 48 of 256 passages withheld, 42 restorable, 6 not locatable in the stored documents (listed as unrestorable; the build refuses them unless explicitly allowed). No quoted spans leaked.
benchmark/datasheet_v7.py generates strata, tiers, label provenance, agreement, annotation hours and release counts from the data. benchmark/manual/review_guide_v7.md covers the five human tasks.
benchmark/human_loop.py closes the loop between proposals, review verdicts and the corpus (propose, apply, merge, collect-candidates). Review items for the real corpus are ready in benchmark/data/review_items/: 18 branch proposals, 35 persona proposals, 10 unreviewed mined candidates. Inspecting the first persona batch showed about half the statements stated the answer; prompt and validator fixed (D7.3), 0 of 67 statements leak now.
README section 6 gains a Phase 6 status table. Retrieval-miner output committed.

Decisions / findings (reasoning in docs/v7_decisions.md, D7.1)

Running V1 with the real reader found three defects in CCA-strict (D7.2): a wrapped NLI hypothesis scored correct default decisions at 0.2 (gold scored 0.00 strict); the hypothesis used the reader's paraphrase instead of the gold outcome the audit asks about; and a reader quoting only the tail of a hedged sentence passed strict. All fixed and regression-tested; with them every V1 criterion passes with the real reader and NLI on four mock dev instances (templated data: wiring is right, validity is not yet shown). Suite: 1,915 tests, green.
The end-to-end test found a real bug: the attachment pattern missed "for signature-tier cardholders", so the mock gold answer scored 0.79 against itself. The restricting-phrase test is now structural, not a word list.
Real-corpus release check found passages that differ from their documents only in typography (curly quotes, non-breaking hyphens, a replacement character from a bad PDF decode); location now folds typography and whitespace.
Retrieval miner: 110 candidates produced 3 verified proposals. S1 will not reach 60 by mining. The narrowing rule stands; the generated datasheet shows S1 test = 1 only because the re-split has not been applied.

Blocked / open

Humans have not started. Nothing is verified, branch-annotated, persona-checked, VACS-labelled or audited by a person. After that: benchmark.strata --resplit --incremental --apply, G1, the freeze.
Not run on real data: G1, V1 with the real reader on real dev instances, V2, the matrix. The NLI threshold is untuned.
Six corpus passages cannot be rehydrated from the stored documents (re-fetch or drop those instances before release).

Next up

Annotate (benchmark/manual/review_guide_v7.md), then experiments.validity_battery and experiments.reader_audit.

2026-10-31 (later) - v7/v8 build: systems, detection, gates, analysis and the claims ledger

Done (1,791 tests collected, full suite green; every module below has tests, and the riskiest guards were mutation-tested)

Systems: baselines/registry.py (B0, B3 with oracle conflict groups, B7c, B8, shared verbaliser, prompt hash), pipelines/scopecompose.py (O3: parse + frames + partition composition, repairs logged, falls back to its own parse), detection/frames.py (batched extraction, verbatim gate, one re-ask), detection/qnfa.py (D5).
Experiment machinery: experiments/run_matrix.py (resumable cells, one model per cell), scheduler.py (quota-aware, unattended), g1.py (token measurement, cut order), staging.py + prereg/freeze.py (test runs refuse before a tagged, committed, hash-matched freeze), analysis.py (exact McNemar forced, Holm, BCa, achieved power), detection_eval.py (E2/H2), vacs_eval.py + benchmark/vacs.py, oracle_cascade.py, make_results.py, reporting/claims.py + registry.py (numbers reach the manuscript only through results).
Human pipeline: benchmark/review.py (candidates, branches, personas, VACS; timed, independent), preannotate.py, personas.py, chunker.py, mining/retrieval_miner.py, govuk_search.py.
Models: scopecompose-reader (llama3.1:8b) and scopecompose-qwen7b (qwen2.5:7b) built with pinned num_ctx; both exercised for real.
24 commits pushed to origin/master.

Decisions / findings (reasoning in docs/v7_decisions.md)

The first reader prompt scored the gold answer zero (an 8B reader refused to apply a default); fixed and guarded by experiments/reader_smoke.py.
B3 on two passages is degenerate (margin ~0 or ~1), so it often presents both sources: H1 may fail honestly.
O3 first merged distinct decisions into one list and rendered one as the other's exception; fixed.
mcnemar approximated at >= 25 discordant items; confirmatory path now forces the exact test.
Mining that can reach the test set was routed to the EVAL tier (it had been on an evaluated model).
Whole-document mining yielded 8 candidates however many pairings were added (0.12 per pairing).
First live call: full-context B0 on a clean two-passage context stated the exception correctly, so the paper cannot claim full-context models fail on short clean contexts.

Blocked / open

Humans have not started. S1 is still 14 instances; nothing has been verified, branch-annotated or persona-checked by a person. The review tool is ready; hours are the bottleneck.
The retrieval miner's first 110-candidate run is still in flight; its yield is not yet known.
Not run on real data: G1 (token measurement), V1/V2 on real instances, VACS labelling, D5 vs baselines, the matrix itself. The NLI entailment threshold is untuned (flagged in every output).
Not built: K = 6 contexts (contexts.py), the external-annotator audit, release bundle/datasheet updates.
plan_v7/ is still untracked; corpus.jsonl has not been re-split (dry run only).

Next up

Verify mined candidates with python -m benchmark.review --kind candidates review --items <proposals> --annotator <you>.
Pre-annotate branches (EVAL), correct them, generate personas, verify them; then re-split with --incremental.
Run experiments/g1.py on dev to fix the reasoning setting; run V1/V2; pick the H3 comparator on dev; freeze.

2026-10-31 - v7/v8 build, day 1: contract v1.3.0, CCA, the validity battery, Propositions 2-3, signatures

Done (every item has tests; 540 -> 1,439 collected; full suite green, exit 0)

contract/ v1.3.0, purely additive: Opposition, Persona, PersonaPair (gold_changes is checked against the branches, not trusted), contract/frames.py (Signature, NormativeFrame, Modality). All 128 v1.2.0 records still validate. Schemas regenerated.
benchmark/relabel_v7.py (derives, never rewrites gold): 18 natural conditional conflicts = 14 S1 + 4 S2. benchmark/strata.py: S1-S5, stratified leak-free re-split with an incremental mode, and the narrowing rule computed from the data. Re-split NOT applied to corpus.jsonl yet (dry run only).
metrics/values.py (74 value cases, 42 modality cases), metrics/case_accuracy.py (CCA-strict/lenient, four named failure conditions), metrics/validity.py (8 outputs; V1 criteria fixed as constants + hash), metrics/counterfactual.py (V2 sensitivity + system consistency), metrics/coverage_confrag.py (H-K).
composition/partition.py: Propositions 2-3, exact region satisfiability; checked against an independent extensional oracle on 120 random frame sets; mutation-tested.
qnfa/schema/{banking,immigration}.yaml, detection/signature.py: unify (adds OVERLAP), normalisation, fold into applicability. The proposal's three worked fee variables reproduce.
baselines/conflictrag_topsis.py: Entropy-TOPSIS (hand-verified), sufficiency margin, ablatable specificity.
metrics/power_sim.py (exact), metrics/cluster.py (cluster bootstrap, BCa, one-sided bound, sign-flip).
benchmark/mining/govuk_search.py: 476 distinct items, 218 pairings; 186 verified distinct (vs 97 mined).
Tiers READER / EVAL / BACKBONE_PRIMARY / BACKBONE_LOCAL, pinned (fallback: false); Modelfiles pin num_ctx.
docs/v7_decisions.md records every decision below with its evidence.

Decisions (full reasoning in docs/v7_decisions.md)

The 7-day plan's "19 natural conditionals" was wrong; the figure is 18. S1 starts at 14, so the narrowing rule is the expected outcome.
B7 as released cannot run on Groq's free tier: NormBench's stage-1 prompt is ~18-27k tokens against an 8k tokens/minute limit. B7 splits into B7-full (local only) and B7-compact (both backbones, "SG-DT-style").
mcnemar silently used chi-square at >= 25 discordant items; confirmatory tests must pass force_exact=True.
Mining that can reach a test set must use the EVAL tier (BULK is an evaluated detector).

Blocked / open

Ollama pulls (llama3.1:8b, qwen2.5:7b) failed once with EOF and were restarted; reader and local backbone are not yet usable. No reported number may use a stand-in model.
The S1 mining run (60 of 218 pairings, EVAL tier) is in flight; human verification has not started.
Not yet built: detection/frames.py (LLM extraction), D5 detector, VACS, B0/B7/B8/O3 + registry, personas generator, prereg/, scheduler, claims ledger.
The v8 timeline needs ~135 annotation hours; this schedule has 25-40, so S1 will land below the floor of 60.

Next up

Create scopecompose-reader / scopecompose-qwen7b once pulls finish; smoke-test the reader on 5 hand-made answers.
Verify the first batch of mined S1 candidates (benchmark/data/llm_proposals_govuksearch_a.json); queue the next 65 pairings (--skip 60).
detection/frames.py then D5 and VACS; baselines/registry.py with B0/B3; measure tokens per call for gate G1.

2026-10-30 - The pipeline runs end to end, live, as a product: the Studio and the engine server

What exists now that did not. Nothing ran one query from retrieval to a verified answer. run_pipeline.py stopped at A4, B1–B3 were separate classes, and every B1 experiment borrowed the default branch from gold. service/ closes that:

orchestrator.run_query chains A1 → A2 → A3 → A4 → B1 → B2 → B3 → faithfulness, then baselines on the same input, reader verdicts and gold scoring. It reports every step as an event {run_id, seq, t_ms, stage, kind, title, payload}.
A tracing LLM client (an LLMClient subclass installed with set_client) attributes every model call to the stage that made it, with tokens, latency, cost and cache status.
A FastAPI server streams runs over server-sent events and stores every event in SQLite, so history, replay, comparison and export cost nothing extra.
Collections (text, URL, txt/md/pdf/html, chunked into passages), a conflict audit over a collection, batch evaluation, a Python client.
The site's #/studio is the client. Without a server it runs the verified browser engines and replays recorded server runs (site/public/data/runs/, written by python -m service.record).

Two pieces of glue, both reported, never silent.

The default branch is assembled from A4's default side (service/branches.py). B1 only ever extracts exceptions. Offline, the rule descriptor's outcome is the passage's first 80 characters, which cut answers mid-word ("… made in a f"), so a clipped outcome is replaced by the passage's own first sentence.
A detector miss either stops the run (as-run) or continues under an override policy, badged everywhere downstream. On a benchmark record the override uses the gold type and pair ("oracle: gold label"). This exists because stage 1 finds almost nothing (below), and without it the demo could never show composition.

The first live evaluation of the whole chain. Dev split, 25 instances, as-run (no override), DeBERTa NLI, Contriever, Groq for stage 2, descriptors and B1. 221 s, 20 LLM calls, $0.00 (free tier).


Stage-1 detector (DeBERTa)	TP 2, FP 1, FN 13, TN 9: precision 0.67, recall 0.13 (offline heuristic: 1 of 15)
Routing	22 pass-through, 2 composed, 1 prior work
Preservation, branch-bearing (n = 12)	ScopeCompose 0.29 · selection 0.38 · concatenation 0.67
Faithful answers	24 of 25

As it runs, the pipeline preserves less than selection. It is the bench's finding, now measured live and end to end: the detector is the bottleneck, and a missed conflict is answered with the general rule alone.
The one unfaithful answer is the most informative row. wp1_fin_exp_034 was detected and composed, but live B1 returned a garbled condition ("for the 8.00 per day overdraft fee does not apply"). The faithfulness check rejected it, which is the check doing its job on a real B1 output for the first time.
On wp1_fin_imp_003, live A4 (LLM descriptors) calls the pair opposed where gold says refinement, so it selects. The recording says so on its label.

The audit's ranking needed a better signal than stage 1. Ranking candidate pairs by the stage-1 score put a separate cash-withdrawal fee above the Premium Plus waiver of the card fee. A "contrast" term was added: one passage negates and the other does not, over shared, stem-normalised content (fee ≈ charge). On the test policy text the two genuine rule/exception pairs are now high risk and the two lookalikes medium.

Visibility is now measured, not judged by eye. site/tools/contrast.mjs screenshots each act twice, once with every glyph transparent, and computes the WCAG ratio of each text run against what is actually behind it (?still freezes the world). It found:

the amber brand on the paper header (1.48:1);
the truth table's unlit cells (2.29:1);
the corpus explorer's tags on paper (1.88:1);
inactive relation tabs over the scope volumes;
the composed-answer captions over the glass sphere;
the phone drag hint.

All are fixed. The final run: 2,386 text runs, three viewports, every one at or above 4.5:1 (body) and 3:1 (large). Two bugs in the tool itself were caught on the way: it measured text hidden by overflow and by a closed <details>.

Also finished. The lab views animate like the story (figures decode, score bars grow, the architecture draws itself). That first shifted the diagram's boxes up into their labels, because a GSAP transform replaced the SVG translate(), so the boxes now fade only. site/tools/studio_smoke.mjs drives every Studio tab against a real engine server (16 checks).

Gates.

Gate	Result
pytest	539 (20 new)
vitest	73
Build, `check_dist` (now covering nested data files), size budgets	pass (Studio chunk 32.8 KB gzip, lazy)
Smoke	pass, including a no-server Studio section
Studio smoke	pass
Contrast	pass
Engine replay	pass

Still not deployed. Same reason as before: the supervisor's call.

2026-10-22 - The site rebuilt as a 3D scroll story; the engines untouched and now guarded four ways

site/ was a dependency-free single page. It is now a Vite + TypeScript + Three.js + GSAP/Lenis build. The page tells the project as eleven acts in one persistent WebGL world:

the collision, and the deletion of a true branch;
the empty socket;
the four scope relations as volumes;
the engine as an instrument;
a live reasoner;
the recomposition;
the research;
the honest frontier.

Branch site-immersive. Nothing outside site/ changed except the CI workflow and the README section.

The engines were moved, not touched. git mv put them into site/public/engine/, which Vite copies verbatim, and the renames are 100%.

verify_engines.js now loads them from there.
Four layers keep them honest:
- Port check: the existing replay against Python.
- Byte check: check_dist.mjs proves dist/ ships them byte-identical and in order, and that no bundle contains a second copy.
- Single door: only bridge.ts may name an engine global, enforced by a test.
- Golden replay: the site's own orchestration replays the Python reference.
The retrieval baselines existed twice: inline in the verifier and in the bench. They are now one file, public/engine/baselines.js, loaded by both. So the copy that ships is the copy that is checked.

A real inconsistency removed. The old hero derived the default branch by array position; the bench sorted by passage id, mirroring ScopeAnalyser.analyse. Only the bench version is order-invariant, and it is now the only version (deriveRelation in src/engine/pipeline.ts).

Two genuine analyser-vs-gold disagreements surfaced. These are wp1_fin_imp_019 (gold refinement, analyser redundant) and wp1_imm_opp_055 (gold opposed, analyser redundant). The second is the authored opposed case already flagged for verification.

The golden test pins both, so a change is a deliberate review rather than a silent drift.
The reasoner shows them as disagreements.
The old README's "39/39 relations derived match gold" was stale.

One HCR misdescription corrected on the page. The bench prose said "HCR catches only invented figures". The code's HCR counts predicted branches citing a passage that does not exist (metrics/preservation.py:363).

Numbers on the page.

Everything is computed from corpus and engines, or imported from the artifact (benchmark/data/decisive.json).
Stage-1 precision 0.833 and recall 0.064 were hand-typed and are now computed.
What cannot be computed in a browser sits in src/data/facts.ts with a source the page shows. This covers the κ pilots, the Tier-1 pilot, and two figures re-run today:
- 520 Python tests passing. The site had said 518 and PROGRESS 501.
- The order-invariance gate passes at 1.0000: 60 records, 72 permutations, 66 non-positional.

Why the world maps scroll the way it does. Each act holds its own keyframes while pinned and travels to the next act only while scrolling out (cursorFor). The first version transitioned during the pinned phase, so the scope volumes slid into the failure act while its copy was still on screen.

What was found by looking. Headless Chrome with the real GPU found:

an overflowing wordmark;
copy and objects colliding;
textures decoded as sRGB in a raw ShaderMaterial, which rendered dark;
the router passing arbitrary hashes to querySelector, which throws;
lab views locking page scroll, because lenis.stop() hides overflow.

All are fixed.

Not deployed. The deploy job is if: false: publishing makes unpublished results public, which is the supervisor's call. CI now verifies (Python reference and corpus staleness, engine replay), then builds and uploads the artifact.

2026-10-02 - Everything buildable is built. 501 tests.

Four gaps closed. The repo now holds every component the proposal specifies; what remains is annotation and runs.

1. experiments/wp1_probe/ -- Member B's gating experiment. The directory held only a README, so B1 was a method with no evidence it worked.

baseline.py -- direct zero-shot extraction, the ConditionalQA-style comparison. Deliberately not a straw man: same passages, same model, a competent prompt. The probe's claim is that asking contrastively recovers what a direct ask misses, and that means nothing against a weakened baseline.
run_probe.py -- the go/no-go. Recall and HCR print in one block and never separately, because the spec is explicit that a gain bought with fabrication is not a success. verdict() either returns GO or names the agreed fallback, so a negative result is a finding rather than a scramble.
Explicit and implicit recall are never averaged -- the average is dominated by the easy half, and would let a method that does nothing for unmarked conditions look like it works.
A corpus with no unmarked conditions returns "NO UNMARKED CONDITIONS IN THIS DATA" rather than "no useful margin". Blaming the method for a property of the data would be the wrong conclusion.

2. detection/llm_nli.py -- API-backed entailment satisfying the NLI protocol, so the grounding gate can be scored either way. Built to be ablated against, not to replace the local scorer: the gate is the pipeline's highest-volume consumer and the cross-encoder is free per call. A failed API call scores NEUTRAL, never entailment -- the one direction a gate must not fail in. A missing index in a batched reply stays a gap rather than shifting later results into earlier slots, which would score each hypothesis against the wrong premise with nothing downstream to show it.

3. The LLM judge (proposal 6.1) is implemented. It was deliberately unreachable while nothing could validate it; validate_judge and gold instances now exist. One call per gold branch rather than several per reply, which invites the model to spread its judgements. A failed call returns SUPPRESSED, not PRESERVED -- the conservative direction, since a judge that inflates the headline metric on error leaves no trace.

The guard moved from blocking to provenance. A judge cannot police what a caller does with its output, so PreservationScores.judged_by records which of the three paths ran, and render() states the validation requirement beside every judged number.

4. Tier-2 construction and relation authoring.

split_constructor.py turns a rule and its carve-out stated together into two standalone passages -- the WP1 finding used in reverse. 256 splittable sentences in the mined corpus. Three guards: every figure must appear in the source, a half still opening with "except" means the cue merely moved rather than the halves separating, and both halves must state an OUTCOME rather than only a condition. That last one came from inspecting the first batch.
relation_author.py authors redundant and opposed, which nine pilots and 91 pairings never produced. Every candidate is verified against its own definition using the same compare and outcomes_match the pipeline uses, and a failure is discarded rather than relabelled -- relabelling would fill the refinement bucket, already full, and leave these two empty.

Building the author exposed a modelling error of mine: for redundant the general branch must be the DEFAULT. Giving both branches a categorical attribute makes them two single-valued sets, which compare correctly calls DISJOINT, and no redundant case could ever have verified.

Also: datasheet tier counts are generated rather than typed (corpus_plan --datasheet), with a third row for same-guide that did not exist when the datasheet was written.

Two tests updated deliberately, both having caught real changes: the judge no longer refuses, and the probe set is no longer unverified.

Next

Annotation: tier2_batch.jsonl (44 remaining); the authored redundant/opposed cases need the probe-set review treatment.
run_probe --live once enough instances carry branch structure.
Supervisor: the same-guide target needs ~250 pairings for 45 instances.

2026-09-30 - Corpus at 128 of 300; Tier 1 done; kappa settled at p_e 0.54

Pilot 9 is the strongest measurement of the nine. kappa = 1.000 on both axes over 39 instances, and the number that matters is p_e = 0.540, not the 1.000. Earlier pilots sat at 0.689 and 0.847 where kappa swings on one label; at 0.54 it is stable. Both passes plausible (20.7s and 25.3s medians, none under five seconds).

Corpus merged to benchmark/data/corpus.jsonl -- 128 instances, 128/128 valid.

row	have	target
natural cross-document	59	60
synthetic split	54	180
same-guide retrieval split	15	60

Tier 1 is effectively done. The row WP1 predicted would be hardest, at ~0.34 candidates per pairing across 91 gov.uk pairings, is one instance from target.

The probe set is corpus. Its 54 human-verified instances are tagged construction=split / separation=synthetic_split and were merged in. They are the only source of redundant and opposed in the project -- 2 of each -- and 42 of them already carry branch structure, which no mined instance does.

Provenance differs and must be disclosed: probe instances are model-proposed plus one human adjudicator (annotator_a="model-proposed", annotator_b="human-verified-2026-09-25"), while pilot instances are two independent annotators. Both fields are recorded per instance so the datasheet can report the split rather than pooling them silently.

Nine pilots, five mining batches, 91 pairings: redundant and opposed have never appeared in mined data. That is settled rather than small-sample. redundant needs a narrower restatement that agrees with the general rule; opposed needs two crossing scopes that disagree. Organisations publish neither on purpose -- the first is redundant to write, the second is a drafting error. They can only be constructed.

A distractor came back conditional from both annotators, which is why distractors are never marked as known negatives at build time. A random pair from one provider occasionally is a rule and its exception, and pre-labelling would have written a wrong gold label neither annotator would have seen.

Blocked on / next

Supervisor decision: same-guide is 45 short at 0.18 candidates per pairing -- roughly 250 more pairings for a descriptive row. Worth rebalancing the 60/60/180 target before spending that.
126 more synthetic-split instances, which is construction rather than mining, and the only route to the redundant/opposed floors of 20 each. Tooling for it does not exist yet.

2026-09-25 (evening) - Kappa pilot 1 failed on the type axis, and diagnosed cleanly

axis	kappa	n	95% CI	verdict
five-class conflict type	0.326	54	[0.168, 0.484]	fails 0.61
four-way scope relation	0.775	23	[0.395, 1.000]	passes, provisionally

Bulk annotation is not cleared. Manual revised; the pilot must be re-run on a fresh batch. Full write-up in docs/kappa_pilot_1.md.

The axis expected to be harder -- scope relation -- is the one that held. The five-class type failed, concentrated enough to name the cause. Finding this on 54 instances cost an afternoon; on 300 it would have cost the corpus.

Finding 1: the manual's own ordering produced the largest cell

conditional -> no_conflict, 7 of 23 disagreements. Three of those seven carry an explicit only if or unless -- the easiest cases in the set, which rules out "the hard boundary is subtle" as the explanation.

Section 3 said "work through these in order" and listed no_conflict first, defined as "about different subjects, or they agree", with "do not force a conflict label". For "interest is paid at 1.5%" against "interest is payable only if two direct debits are active", both statements are true, they do agree in the ordinary sense, and the reader has just been told not to force a label. They stop at the first entry and never reach conditional. The discriminating test existed only for factual-vs-conditional, further down the page.

The project's central conceptual move was assumed rather than stated: a passage that qualifies another is a conflict even though nothing contradicts. That sentence was nowhere in the manual. Section 3 now opens with the discriminating question -- does the second passage change the answer for anyone covered by the first? -- before the list, with both directions worked through.

Finding 2: temporal fires on any time expression

Six disagreements involve it. Dates and durations appear throughout conditional instances as properties of the case, not versions of the rule: "loans taken out on or after 1 April 2024", "closed within 14 days", "two years outside the country". The old entry said look for dates, "effective", "as of" -- a keyword instruction, and keywords are what misfires here.

Both annotators fell into it in opposite directions, so it is the manual rather than one reader. Rewritten around a question -- is the earlier statement still in force for anybody? -- with the rule of thumb that a superseded rule has a date attached to the rule, a conditional rule a date attached to the case.

Finding 3: two labels look like a keypress

Two numeric-disagreement cases (fee 1,500 vs 1,846; threshold 26,200 vs 38,700) were labelled opinion, which means neither passage is checkable -- impossible when both quote figures that disagree. On the menu 3 is temporal and 4 is opinion. Fixed in the tool rather than the manual: the CLI now detects numeric disagreement and, if opinion is chosen, says so and offers factual / temporal / opinion-anyway. Reuses numbers_conflict from the metric so the guard and the scorer cannot drift.

Two things about the pilot itself

Pace. Starker than the medians (5.5s vs 20.3s) suggest: febin judged 41 of 54 in under ten seconds and 18 in under five, against johann's 3 of 54 under ten. Fast is not careless, and the manual defects stand independently -- but 18 instances in under five seconds is hard to reconcile with reading a query, two passages and applying a two-step test. Treat it as a contributor of comparable size to the manual defects, not a footnote.

The batch was not fresh for one annotator. johann had verified all 54 of these during the probe-set review days earlier. Kappa between them remains valid -- neither saw the other's labels -- but he was re-labelling material he had personally adjudicated while febin came cold. Two consequences: the agreement-with-gold figures (85.2% vs 55.6%) are not comparable, since johann signed off that gold and agreeing with it is near-circular; and the exposure asymmetry could itself depress kappa. The re-run must use a batch neither has seen -- the first ~30 WP2 instances as they are built.

Also fixed: a gap this pilot exposed in the tool

The scope axis reported 0.775, "substantial", and cleared to proceed -- on 23 instances with an interval running down to 0.395. is_acceptable looked only at the point estimate. It now emits a caution when the estimate clears the bar and the interval does not, because "not shown to be unreliable" is not the same claim as "shown to be reliable".

Next up

Both annotators re-read section 3; re-run on ~30 instances new to both.
If the type axis still fails, the remaining disagreement is conceptual rather than editorial, and the next move is the third-reviewer route rather than another manual edit.
Scope axis needs re-measuring on more instances whichever way it lands.

2026-09-25 (end of day) - WP0 closed, replacement opposed case, second annotator secured

Tasks 2, 7 and the task-1 follow-up all done. All four gating checks are now closed, and none of them killed the project -- two narrowed it.

WP0 closed-venue sweep: prior art found, claim narrowed

The four-way relation is not a new formalism, and the paper must stop implying it is.

specificity-based override (Poole; Bochman) already formalises what this project calls refinement. Bochman states the principle in near-quotable terms: more specific default rules override less specific ones in conflict, pattern A -> C against A and B -> not C.
rule-analysis literature already uses disjoint and redundant as rule relations; KB refinement already uses refinement for condition specialisation.
default logic (Reiter 1980), defeasible logic, argumentation (Dung 1995) and AI & Law lex specialis all sit adjacent.

Verified directly in the proceedings, full author lists recovered:

A Rule-Based Approach to Specifying Preferences over Conflicting Facts and Querying Inconsistent Knowledge Bases -- Bienvenu, Bourgaux, Inoue, Jean. KR 2025.
Reasoning in Defeasible Description Logics with System W and Lexicographic Inference -- Casini, Haldimann, Meyer. KR 2025.
Deontic Reasoning Based on Inconsistency Measures -- Arieli, van Berkel, Raddaoui, Strasser. KR 2024.

What survives. No prior work found that says: given two retrieved passages, classify their relation as refinement / disjoint / redundant / opposed. The vocabulary is old; the operationalisation for separately retrieved documents, and the annotated benchmark, are not.

The strongest single part is opposed. Specificity answers B subset of A. It does not answer A ∩ B != {} with A ⊄ B and B ⊄ A and conflicting outcomes -- that is a structural gap in specificity-based override, not an application of it.

Decisions

Stop claiming the taxonomy; claim the operationalisation. The memo now carries the exact framing sentence plus four phrasings to avoid and four to use. That framing survives even if a paper enumerating all four cases turns up later, which is the reason for adopting it now rather than defending the taxonomy.
Residual named rather than hidden. TPLP, AIJ and JAR were not searched exhaustively. Recorded in the memo with the reason it no longer matters much.

Replacement opposed case: wp1_imm_opp_055

Dropping 054 had left opposed with one instance -- thin coverage of the relation hardest to tell from refinement. The replacement:

p0: five years continuous lawful residence -> may apply for ILR
p1: Youth Mobility Scheme participants -> may not apply for ILR

Fixes what killed 054: both passages speak to exactly one predicate, eligibility to apply for ILR, and assert opposite answers. The scopes cross without nesting -- another route can reach five years, a YMS participant can have fewer, and someone who switched routes sits in both.

Both passages are government guidance carrying the same date, so neither credibility nor recency can break the tie. A system reaching for either is answering a question the data does not pose. Verified: compare returns overlapping, outcomes do not match, no default branch, B2 routes to selection.

Set is back to 54 instances, opposed back to 2. Needs human verification like any other case -- added to HIGH_RISK with the predicate check spelled out.

Task 2 done -- second annotator secured, so the kappa pilot is unblocked.

Next up

The kappa pilot (task 5), then bulk annotation (task 6). Nothing else blocks.
Verify wp1_imm_opp_055.
Tidy-up only in WP0: read ArbGraph, confirm ConflictRAG's mechanism, read CARE-RAG for the distinguishing paragraph.

2026-09-25 (later still) - SG-DT verified against the source. WP0's biggest risk closed, and an overclaim corrected.

Task 4 done. SG-DT sections 3-4 read against the actual paper, not an abstract.

The bridging claim survives -- their task is defined as given a provision X, output a Span-Grounded Deontic Tree, with nodes grounded to spans of that current text scope, and section 4 describes NormBench as benchmarking intra-provision defeasible scope parsing. Competing separately retrieved documents are never the input unit. This was the single largest risk in WP0 and it is closed.

But this memo had overclaimed, and the human read caught it. The model pass concluded "the pipeline never has two documents, because the retrieval in its Auditability Trap is retrieval of spans inside the provision." That is wrong: the paper discusses RAG and retrieval directly in its motivation, noting retrieval can determine whether a decisive exception is visible at all, and its downstream statutory-reasoning experiment supplies a statute snippet as evidence. Writing "SG-DT does not involve retrieval" would have handed a reviewer a free correction.

The surviving distinction is about what the benchmark task takes as its input unit, not about whether retrieval is mentioned anywhere. The memo now carries the exact sentence to use and the two not to.

This is a clean worked example of the limit I flagged when writing the memo: a model reviewing a model's citations shares its failure modes. The structural defects it found were real; the framing it produced was too strong, and only a human reading the source would have caught that.

37% confirmed, with a wording constraint Verified against the dataset-statistics table: Level 1 simple 834 (36%), Level 2 nested 610 (27%), Level 3+ recursive 846 (37%). The paper states directly that a significant portion is recursive at Level 3+, 37%. Citable.

The constraint is phrasing. Their table classifies items by level; it does not say 37% require deep resolution to answer.

WRONG: "37% of their benchmark needs deep exception nesting"
RIGHT: "37% of NormBench items are classified as Level 3+ (recursive), corresponding to deep/counter-exception structures"

New finding: the Structure-Grounding Gap Their results frame a dissociation between finding the relevant span and attaching it to the correct logical parent -- models have the first ability without the second. That is this project's thesis observed inside a single provision, by an independent group. It is the strongest external support available for the argument that locating evidence and composing it correctly come apart, and it should be cited as support rather than treated as competition.

Decisions

Positioning reframed from "they do not retrieve" to "their structural target is an already-supplied provision". Defensible against the paper's own text; the earlier version was not.
Cite Chen et al. for Silent Scope Omission in section 4, in the paper's own voice. Reads as scholarship when volunteered, as concealment when found.

Risk register updated Largest remaining WP0 risk is now the closed-venue sweep (KR, TPLP, AAAI, AIJ, JAR). If classical KR already formalises the four-way relation, the contribution narrows from defining it to operationalising it for retrieval.

Next up

Second annotator for the kappa pilot; closed-venue sweep.
Open: OPPOSED has one probe instance after 054 was dropped.

2026-09-25 (later) - Corpus composition settled. Contract v1.2.0.

Task 3 closed. The WP1 gating question now has a decision attached, and the decision is encoded rather than left in prose.

The target: 300 instances

row	target	role
natural cross-document	60 (20%)	evidence for the natural multi-document setting
synthetic split	180 (60%)	main controlled benchmark
same-guide retrieval split	60 (20%)	real retrieval separation within one document

The 150-instance Tier-1 requirement is dropped. At ~0.08 plausible pairs per curated pairing it would need ~1,900 pairings for no proportional scientific return. Tier 1 is now evidence for the natural setting, not the main corpus.

Done

benchmark/corpus_plan.py - the target as executable code, with progress reporting: python -m benchmark.corpus_plan --data <corpus.jsonl>.
contract.models.Separation - new optional enum (cross_document | synthetic_split | same_guide) on both QueryRecord and GoldInstance.
metrics.classification.split_for_reporting() - the three-way split the paper reports. split_by_construction() keeps the old two-way behaviour.
Probe set and mock generator both now label their separation.
13 tests added (460 -> 473). Gate still PASS at 1.0000.

Decisions

Same-guide needed a field, not just an intention. The agreed plan was "store as Tier 2, report separately" to avoid schema risk. Storing it was free; reporting it separately was not -- split_by_construction partitions on construction, so a same-guide instance stored as split would have been silently folded into Tier 2 by every table, and the distinction would have existed only in prose. The additive optional separation field is the mechanism. Construction stays a two-way tag, so nothing that routes or validates on it changes.

Contract v1.1.0 -> v1.2.0. Additive, optional, backwards compatible -- every existing instance carries separation=None and still validates. Schema regenerated; mock updated in the same commit per the freeze rule.

Unlabelled instances fall back to synthetic split, not Tier 1. An unlabelled instance promoted to Tier 1 would inflate the claim that matters most, so the fallback goes the conservative way and the count of unlabelled instances is reported rather than hidden.

Tier 2 is stratified, not counted. TIER2_STRATA sets floors across all four relations, both explicitness values, seven condition devices, and distractors. A test demonstrates the failure mode directly: 180 instances that are all refinements meet the count target and still leave the plan incomplete, because the refinement/opposed boundary the paper reports would be untested.

The pilot yield cannot become a prevalence rate. 38 curated pairings across two domains supports a design decision, not a population claim. prevalence_estimate() exists only to raise, with the two sentences -- the one that is supportable and the one that is not -- written into the error. Cheaper than remembering.

LLM proposer is the primary discovery mechanism for WP2. On the banking corpus the lexical miner found 1 candidate against the proposer's 4, of which 3 held up. tier1_pilot.py now prints this at the end of every run, so the decision is visible where it applies rather than only in a document.

Next up

Remaining human tasks: second annotator (kappa pilot), SG-DT sections 3-4, closed-venue literature sweep.
Open from the probe verification: OPPOSED has one instance, thin coverage of the relation hardest to tell from refinement.

2026-09-25 - WP1 probe set human-verified. Gating item closed.

All 54 cases reviewed case by case by a human. 50 accepted as proposed, 3 changed, 1 dropped. The set is 53 instances and is usable as gold.

This closes the last gating item that was blocking Member B's feasibility probe.

Changed

wp1_imm_imp_016 - re-encoded from two branches to three: two years generally, five with settled status, four for Swiss nationals holding settled status. The Case schema gained sub_condition / sub_outcome / sub_applicability / sub_attrs to express a branch nested inside an exception. The instance now carries gold_multi_exception_flags.nested=True and B2 routes it to selection with the flag, which is the correct first-order behaviour -- exception-to-exception precedence is out of scope and is flagged, not guessed.
wp1_imm_imp_026 - refinement -> disjoint. The text never establishes that someone who has lived in the UK for six months is a subset of those resident in a listed country, and refinement requires that nesting. Re-encoded with both branches separately scoped and no default, per v1.1.0.
wp1_imm_opp_054 - dropped. p0 concerns "the skill requirement", p1 "the requirements of this route" -- different propositions, so a 46-year-old graduate satisfies both and there was never a contradiction. Dropped rather than reworded: changing p0 would have produced a valid case, but a different one from the case reviewed, and editing evidence to fit a label is the wrong habit to build into a benchmark.

A fourth matcher bug, exposed by the 016 re-encoding Encoding three branches immediately disagreed with B2, which read "the period is five years" and "the period is four years" as the same outcome. numbers_conflict extracted digits only, and policy text spells small quantities out constantly. The two strings share "period" and "years"; the only tokens that differed carried the whole meaning.

Now handles spelled cardinals including compounds, so "twenty-eight" and "28" resolve to one figure rather than conflicting. Gold and B2 now agree on 016.

This is the third bug of the same shape after negation-as-stopword and passage-ids-as-figures: two outcomes differing in one small token that carries all the meaning. Worth watching for a fourth.

Provenance and the kappa guard Every instance now carries annotator_a="model-proposed" and annotator_b="human-verified-2026-09-25". Both are kept: the first records that a model drafted the case, which is exactly why this set can never enter a Cohen's kappa -- the two passes share a starting point, so their agreement would measure the proposal rather than the task.

MODEL_ANNOTATORS was missing the hyphenated spelling "model-proposed", which is the exact string this repo writes. The guard would have let the project's own model-proposed labels straight through. Fixed, and a test now pins the refusal against that literal string.

test_probe_instances_are_marked_unverified failed, as designed -- its docstring said that if it ever failed with a real annotator name, the set had been verified and the assertion should be changed deliberately. Replaced with a test that checks both fields are present, plus the new kappa-refusal test.

Open limitation OPPOSED now has one instance (wp1_fin_opp_053). Thin coverage of the relation hardest to tell from refinement, which is the confusion cell the paper reports. Worth authoring a replacement before the probe runs; it can be drafted and verified the same way these were.

Next up

Remaining human tasks are unchanged: second annotator for the kappa pilot, supervisor sign-off on the Tier 1/Tier 2 split, SG-DT sections 3-4, and the closed-venue literature sweep.

2026-09-24 (later still) - Member B built: B1, B2, B3 and the preservation metrics. Four scoring bugs found on the way.

The pipeline now runs end to end. Building B exercised the shared scoring code against real structures for the first time, and it broke in four places -- every one of them silently, and three of them in the direction of a better-looking number.

Done

extraction/probing.py (B1) - Contrastive Scope Probing plus the grounding gate. Contrastive questions rather than open ones, because a question with a presupposition is harder to answer vacuously. The gate scores passage |= (condition AND outcome) locally and its rejection rate is reported: a gate that rejects nothing is not a gate. Includes the cue-matching fallback as the baseline the probe must beat.
composition/operator.py (B2) - the composition operator. Routes on A4's relation, merges restatements rather than branching them, and flags nested and crossed separately instead of composing second-order structure.
generation/scoped_answer.py (B3) - scoped answers with per-branch attribution. Default renderer uses no model at all; check_faithfulness() verifies afterwards that every branch reached the answer, that no figure was invented, and that the generator did not rank sources.
metrics/preservation.py - PR / SR / HCR / SCR implemented against the frozen signatures, broken out by tier. The LLM-judge path raises rather than silently working, because proposal 6.1 requires validation against human labels first.
36 tests added (423 -> 459). Order-invariance gate still PASS at 1.0000.

Four bugs in the shared scoring code

Negation was a stopword. "no" and "not" were in _STOP, so "no fee applies" and "a fee applies" scored a Jaccard of 1.00 -- exact opposites judged identical. A system emitting the reverse of a gold branch was scored as preserving it. This sits in metrics/branch_match.py, which the decisive experiment scores on. Fixed by removing them from the stoplist and adding polarity_conflict() as a hard veto beside numbers_conflict().
Two BranchJudgement enums. align() returned branch_match's and the scorer compared against preservation's, so every is check was False and a perfectly preserved run scored PR = 0.0 with all branches counted distorted. Now one enum, imported.
Passage ids read as invented figures. (per p0) matched the number regex, so every correctly-attributed answer was flagged for inventing a figure -- including the template renderer's, which cannot invent anything. That the deterministic path failed its own check is what exposed it.
Decimals split as clause boundaries. [;.] tore "a 2.75% fee applies" into "a 2" and "75% fee applies", so any answer quoting a decimal was reported as dropping its own branches.

Plus a false positive from the fix to (1): "no more than 20 hours" is a quantifier bound, not a negation, and it made a permission read as a prohibition. Comparative quantifier phrases are now stripped before polarity is counted.

Decisions

PASS_THROUGH and the flagged path keep only the branches the answer actually states. Both previously carried every branch through while emitting an answer that stated one or none. On PASS_THROUGH that mattered most: a missed conditional conflict is exactly the suppression the project measures, and crediting it with full preservation hid the detector's own failures behind the scorer. End-to-end PR on mock fell from an inflated 0.97 to 0.58 against 0.73 with oracle routing -- the 0.15 gap is the detector's real contribution.
Generation defaults to the template renderer, no model. It cannot drop a branch or invent a figure. The model earns its place only where fluency is being measured, and the two are reported as separate systems.
The LLM judge is unreachable by default. Passing a client raises. An unvalidated judge that quietly works would get used.
validate_judge reports rank correlation as well as agreement. A judge can agree on 85% of branches and still mis-rank two systems if its errors concentrate in one -- and the ranking is what a conclusion rests on.

Decisive experiment re-run after the matcher fix Baseline PR 58.3% -> 62.5%, difference -8.3% -> -12.5% [-37.5%, +12.5%], McNemar p = 0.51, 9 discordant branches. Conclusion unchanged: the interval contains zero and the test has almost no power at this size. Both directions of the negation fix moved numbers -- negation-to-negation now matches better, negation-to-affirmation is now vetoed.

Verified

A -> B end to end on mock: A1-A4 output feeds B2-B3 without adaptation, 25/25 answers faithful, PR/SR separate composition (0.78) from pure selection (0.49) as the design requires.

Next up

Everything remaining is data, not code: the 54 probe cases need human verification (3 flagged in docs/wp1_probe_verification.md), WP0's closed-venue sweep, WP2 annotation, and the kappa pilot, which needs a second person.

2026-09-24 (later) - Annotation tooling, probe verification, WP0 first pass, and a schema bug that inverted four test cases

Cleared the remaining Member A work and the manual tasks. The verification pass found a defect that had been silently inverting the four cases most important to the scope-relation claim.

Done

benchmark/annotation/ — the labelling tool. Was empty; it blocked annotation, which is the project's critical path.
- store.py — per-annotator files, blind by construction. There is deliberately no "load everything" method.
- agreement.py — Cohen's kappa per axis, with bootstrap CI and the disagreement cells.
- cli.py — annotate (blind pass), adjudicate (review model proposals), agreement, status.
docs/wp1_probe_verification.md — adversarial pass over all 54 probe cases.
docs/wp0_positioning_memo.md — completed from skeleton to a first-pass deliverable with a verdict.
23 tests added (400 -> 423). Order-invariance gate still PASS at 1.0000.

Found: four probe cases contradicted their own labels wp1_fin_dis_051, wp1_imm_dis_052, wp1_fin_opp_053, wp1_imm_opp_054 each declared disjoint or opposed while carrying a branch structure that says refinement. Mechanically checkable: compare(exception, default) = subset in all four.

Root cause was in the schema. GoldInstance required exactly one default branch, whose applicability is by definition the whole case space — which makes every other branch a subset of it, and nested-with-differing-outcomes is refinement. The rule encoded the refinement shape and imposed it on every relation. These four cases exist specifically to test telling refinement from the rest, so they were testing the opposite of what they were for.

The same bug was in contract/mock.py, and worse: _build_opposed used scopes first-year holders and first-year holders at designated institutions — genuinely nested, so the template was a refinement mislabelled as opposed, training the detector to call refinements opposed. Now uses crossing scopes (first-year x part-time).

Decisions

Contract v1.0.0 -> v1.1.0: "exactly one default branch" relaxed to "at most one". A default is still required for refinement and redundant, which genuinely are the default-plus-carve-out shape. Backwards-compatible (existing instances still validate) and the wire format is unchanged, so the bump is minor.
Kappa is refused, in code, when it could not measure inter-annotator agreement — the same person twice, or anything model-generated. It renders a plausible number either way, and a fabricated kappa is unrecoverable once submitted.
A degenerate batch reports "undefined", not 1.0. Both annotators using one category gives 0/0; reporting 1.0 would present a batch that tested nothing as flawless annotation.
Adjudicated labels carry basis="model". Reviewing a proposal is ~4x faster than labelling cold and carries automation bias. Real human judgement, but not the same evidence, and never pooled silently.
Explicitness docstring corrected. It said "condition stated or inferred"; the operational test is absence of a strong exception cue. Those differ — in "Where the statement balance is settled in full...", the condition is stated and only the override is not. Write-up must say "recovery of unmarked exceptions", never "implicit condition recovery".

WP0 verdict: bridging claim survives, narrowed. Confidence moderate.

All 9 citations verified against arXiv / ACL Anthology. None fabricated.
SG-DT takes an already-identified provision and never models competition between separately retrieved sources, so the retrieval-stage distinction holds. This rests on an abstract plus a partial PDF read, and one automated read of that PDF returned wrong answers about the paper's own terminology. A human must confirm against SG-DT §3-§4.
Four citation discrepancies to fix: the "37%" NormBench figure is unverified; TCR is context-memory not inter-source; ConflictRAG's "Entropy-TOPSIS" is unconfirmed; SG-DT's second pathology ("Auditability Trap") was missing.
Five works found that the memo lacked. CARE-RAG (arXiv:2507.01281, "conflict-driven summarization") is the nearest neighbour and must be cited before a reviewer raises it.
Closed-venue sweep (KR, TPLP, AAAI, AIJ, JAR) is NOT discharged and is where the residual risk sits: classical KR has reasoned about specificity-based override since Reiter 1980.

Not done, and cannot be

The kappa pilot. It measures whether two independent humans converge. Both label sets from one source measures self-consistency instead. Needs a second person.
Bulk annotation as gold. Labels generated by the same model as the detector make the evaluation circular. adjudicate exists to cut the human cost from ~20 hours to ~3, not to remove the human.

Next up

Member B's modules (extraction/, composition/, generation/) are still 8-line stubs and are the largest remaining build.

2026-09-24 - Cost/latency instrumentation, error analysis, and a key that should not have been committed

Closes the last two measurement gaps against the proposal, and catches a credential sitting in a tracked file on the way to publishing the repo.

Done

experiments/run_cost_latency.py — per-query cost and latency broken down by stage, per proposal §6.1. --compare-stage2 prices the escalation itself by running the same records with stage 2 disabled and differencing.
experiments/error_analysis.py — the qualitative pass §5.2 promises on the confused instances. Errors are grouped by the feature pattern that drove them, so a systematic weakness shows up as a cluster rather than as a list of anecdotes. Emits an annotatable markdown sheet for the manual read.
tests/test_cost_latency.py (10) and tests/test_error_analysis.py (14). Suite now 400 passing; order-invariance gate still PASS at 1.0000.

Found

.env.example contained the real Groq key, not a placeholder — tracked, and present in commit ca0ec7d. The key had already been rotated, so the exposure is historical rather than live, but it would have been published to a public remote. Replaced with gsk_your_key_here and scrubbed from history in the same rewrite that removed the commit trailers. .env itself was never tracked (.gitignore:2), which is why this survived earlier checks — the check confirmed .env was ignored and never looked at .env.example.

Decisions

Cost/latency reports median and p95, not just the mean. The distribution is bimodal — most queries resolve in stage 1, a few escalate to an API call — so the mean sits between the two modes and describes no query that actually ran.
The bimodal warning needs an absolute floor, not just a ratio. Offline, both modes are sub-millisecond and the p95/median ratio between two samples of timer noise runs past 50x. The first version duly announced a bimodal distribution on a run where nothing took any time. BIMODAL_FLOOR_S = 0.010.
A fully cached run is reported as such. Warm-path cost is zero, which is true and useless for planning; without the caveat it reads as "the system is free to run" when what happened is that somebody ran it before.
Errors are grouped by feature signature rather than by surface text. An error grouped by what drove it points at a fix; grouped by its wording it points at an anecdote.

Verified

On 12 mock records the error analysis found a genuine cluster: 57% of the conditional -> no_conflict misses share the signature no-cue / no-numeric-clash / nli-neutral / scope-asymmetry — cueless exceptions that the heuristic NLI reads as unrelated. That is the harness working, not a result: mock data with heuristic NLI, so the 83% error rate is a fixture property and is reported as one.
One test in the first draft (test_records_absent_from_gold_...) passed for the wrong reason — mock ids derive from the template name, so two seeds produce identical ids and nothing was ever skipped. Rewritten to rename the ids explicitly and assert the denominator excludes them.

Blocked / open

Unchanged and all on the data side: the 54 probe cases need human verification, WP0 literature review, WP2 annotation (250–350 instances), and the κ pilot with two annotators. No code blocks on any of these.

Next up

Published to the remote. Next substantive step is WP2 annotation — the infrastructure is ahead of the data, and every remaining headline number is waiting on labelled instances rather than on more code.

2026-09-24 (final) - The decisive experiment exists, and its first result was fake

Proposal 6.3's structured long-context baseline now runs end to end. The important part of this session is that the first number it produced was wrong in the flattering direction, and both causes are now tested against.

Done

baselines/structured_long_context.py - a long-context model given every passage at once, asked for the gold branch schema. Two variants: structured (6.3's strong form) and freetext, the latter to separate the pipeline's architectural benefit from the benefit of merely requiring structured output.
metrics/branch_match.py - deterministic gold-to-predicted branch alignment, so the experiment is scoreable without Member B's LLM judge.
experiments/run_decisive.py - head-to-head with Wilson intervals, paired bootstrap difference, and McNemar.
376 tests pass (up from 354), 22 of them on this code specifically.

The first result, and why it was fake

pipeline 100.0%  vs  baseline 33.3%   difference +66.7%   p = 3e-05

Spectacular, and wrong at both ends.

The pipeline was scored with GOLD labels. to_query_record( include_gold_pairs=True) hands routing the correct conflict type, so it routes perfectly by construction. That is an ORACLE pipeline compared against a baseline doing its own detection - it flatters the pipeline by exactly the detector's error rate.
The matcher was too strict. Gold outcomes are terse annotations; model outputs are verbose prose. Symmetric Jaccard scored gold "a 2.50 per-transaction charge applies" against "You will be charged a 2.50 fee per cash withdrawal..." at 0.19 and called two identical outcomes a distortion.

After fixing both

pipeline 50.0%  vs  baseline 58.3%   difference -8.3% [-29.2%, +12.5%]   p = 0.73

The baseline is nominally AHEAD and the interval contains zero. With only 8 discordant branches the test has almost no power, so this settles nothing yet - but it is the honest reading, and it is the opposite of what the broken version said.

Fixes

score_pipeline_routing(oracle=...) defaults to running the REAL detector. Oracle mode still exists, is labelled "oracle routing" in every output, and its docstring says plainly that comparing it to a self-detecting baseline is the easiest way to manufacture a result here.
Matching now combines three signals: numeric disagreement as a hard veto, symmetric Jaccard OR asymmetric containment, and optional NLI entailment. Containment handles the verbose-prediction case; NLI is required for true paraphrase, and a test records that containment alone does not rescue the original failing example.

What the renderer now does

Prints the falsifying outcome as prominently as a positive one, per 6.5; warns when fewer than 10 branches are discordant; and states the open-weights asymmetry inline - a pipeline win is weak evidence, a baseline win is decisive.

Still not journal-ready

Zero annotated instances, no kappa, WP0 not started. Every number above is on 12 hand-built unverified instances with the offline detector. The machinery is right; the data is not there.

2026-09-24 (final) - Statistical rigour layer, and a broken invariance guarantee

Answering "is this journal-ready?" honestly: no, and the largest closable gap was that every number in the repo was N=1 with no interval. That is now fixed. The data gap remains and is not closable by code.

A real defect found while auditing

interaction_features claimed order-symmetry by sorting the two embeddings into a canonical position on their element sum. Two different embeddings can share a sum, and on that tie the sort is arbitrary and the vector genuinely differs under swap. The bug survived its own unit test because the test used embeddings with unequal sums.

It matters more than a normal defect: order invariance is a CLAIMED DESIGN PROPERTY of this pipeline, and one that holds "almost always" is not a property. Now [e_i + e_j; |e_i - e_j|; e_i * e_j] - every term symmetric, so invariance holds by construction. Added a tie-case test and a randomised property test across 3/64/384 dimensions.

New: metrics/stats.py

Wilson intervals for proportions. Chosen over the normal approximation because this project's rates sit at the boundaries, where the normal interval misbehaves: 72/72 returns [1.0, 1.0] and claims certainty from 72 observations.
Bootstrap intervals for F1, macro-F1 and the leak rates, resampled over instances, not pairs - pairs within a record share passages, and resampling them independently understates every interval.
bootstrap_difference resamples both systems together, preserving the pairing. Checking whether two separate intervals overlap is a weaker and different test, and a common mistake.
McNemar with the exact binomial below 25 discordant items, because on a few hundred instances the disagreement count is often single digits and chi-square is unreliable there.
Holm correction, because three variants means three comparisons and reporting the one that cleared 0.05 is how a null result becomes a finding.

New: reproducibility.py

seed_everything() covers all four random sources (random, numpy, torch CPU and CUDA, PYTHONHASHSEED) plus cuDNN determinism. run_manifest() records git commit and dirty state, contract version, hardware profile, tier map and whether the threshold band was ever tuned - a seed alone does not reproduce a number if the checkpoint moved underneath it.

What the intervals immediately revealed

On the three-variant comparison:

cond->fact  0.0% [0.0%, 16.8%]
accuracy  100.0% [89.3%, 100.0%]

A perfect zero leak rate - consistent with up to 16.8% leakage, because there were only 19 conditional instances. That is the entire argument for this layer: the point estimate said "flawless", the interval says "we have not measured enough to tell".

Also: lexical_nli vs structured_entailment, McNemar p = 1.5e-05, Holm-adjusted p = 1.5e-05 - that ranking is real, and now demonstrably so rather than asserted from a leaderboard.

Wired in

compare.py records per-item correctness and prints Holm-corrected pairwise McNemar, flagging underpowered comparisons explicitly.
run_pipeline.py --evaluate prints the provenance block and bootstrap intervals for every headline metric.
354 tests pass (up from 319), 35 of them on the statistics itself.

One note on method

My first hand-computed Wilson reference was wrong, and the test caught the docstring rather than the code. Corrected, with the derivation written out so the reference is checkable rather than asserted.

Still not journal-ready, and not closable by code

Zero annotated instances. No kappa. WP0 not started. The structured long-context baseline still does not exist. Every number above is on mock or hand-built data.

2026-09-24 (final) - WP1 batch 2: the estimate was optimistic, and that matters

Done

Fetched and audited 10 more banking pairings: Santander, Nationwide, Starling, Monzo. Three Nationwide pairings were unfetchable - their robots.txt disallows /current-accounts/ wholesale.
Ran both miners on the 7 usable pairings.
Consolidated docs/wp1_tier1_findings.md across all 38 pairings.
316 tests pass.

The finding that changed the plan

batch	pairings	distinct docs	lexical	LLM	plausible on inspection
gov.uk	17	5 (29%)	4	not run	unverified
banking 1	11	10 (91%)	1	4	3
banking 2	10	7 (70%)	0	2	0
total	38	22 (58%)	5	6	3

Batch 2 produced two LLM proposals and both fail on reading: Santander pairs two dated fee changes (temporal, not conditional), Monzo pairs two statements about the same scope with inconsistent outcomes (closer to a contradiction).

Batch 1 was the lucky batch. An estimate built on it alone - roughly 0.4 plausible pairs per pairing, "~375 curated pairs for the floor" - was optimistic by about five times. Across all 38 pairings the real rate is about 0.08 per pairing, which puts the 150-instance floor at ~1,900 pairings.

Revised recommendation: Tier 1 cannot carry the corpus. Set a Tier-1 target the evidence supports (20-40 instances, reported as the naturally occurring subset) and let Tier 2 carry the volume. A small, honestly labelled Tier-1 set is worth more than a large blended count.

Why the pairs are scarce even when documents are separate

Banking scores 81% on document separation and still yields almost nothing. The cross-document content is a product page saying what the account does and a charges page saying what it costs - complementary information, the DRAGged into Conflicts category section 4 already distinguishes this project from. The genuine conditional structure sits inside one document, because an author writing one clause states the carve-out next to the rule. Splitting a rule from its exception across documents is simply not how organisations write.

That is a finding about the phenomenon, not only about the mining method, and it belongs in the paper: it explains why the Tier 1 / Tier 2 construction is necessary rather than merely convenient.

Note to self on estimating

I reported "~375 curated pairs" after one batch. One more batch moved it to ~1,900. Single-batch yield estimates on small samples are worth very little, and the honest move would have been to label the first figure provisional.

2026-09-24 (later still) - WP1 domain 2: banking, and the proposer swap

Two findings that change how WP2 should run. Full write-up in docs/wp1_tier1_findings.md.

Done

benchmark/mining/fetch_web.py - general fetcher for HTML and PDF, with per-host robots.txt checked before every request and content-hash identity (commercial sites expose no content_id).
benchmark/mining/llm_proposer.py - the proposer the proposal actually specifies, compared head-to-head against the lexical miner.
11 banking pairings fetched across NatWest, HSBC, Barclays, Lloyds. One URL was an origin-server hostname that robots.txt disallows; skipped and reported rather than fetched.
316 tests pass. Whole WP1 exercise cost 0.00 USD and 157K tokens.

Finding 1 - banking is the far better Tier-1 domain

	gov.uk	banking
distinct-document pairings	5 / 17 (29%)	10 / 11 (91%)

Banks publish the rule and the charge in separate documents. gov.uk publishes one guide with parts. Make financial terms the primary Tier-1 source.

Finding 2 - the lexical miner was under-implementing the proposal

Section 6.2 says a model proposes candidate pairs. The miner was cue words plus word overlap. On identical documents:

method	cross-document pairs
lexical	1
LLM proposer	4

Three of four look genuinely right, including an implicit one: "you won't pay any non-sterling transaction fees when paying with your card" against "Not valid for ATM withdrawals." A product page and a charges schedule share few content words, so a bag-of-words bridge fails even where the pairing is obvious to a reader.

A tuning result worth keeping

Shrinking the proposer's input window from 9,000 to 5,500 chars per document fixed a rate-limit problem and made the output WORSE - it stopped finding fee-waiver pairs and started returning October price changes, because truncation had cut the section where the rules live. Pacing the calls is the right lever; truncation is not. PACE_S = 22 now carries the TPM limit and the window stays wide.

Two more bugs, both caught by running it

The console crashed on a non-breaking hyphen (cp1252) after the work was done but before the JSON was written - the worst place to fail. Output is now sanitised for display and the JSON is written first.
Four of ten pairings had failed on TPM exhaustion because the client's exponential backoff tops out well short of the 60-second rate-limit window. Backoff cannot clear a per-minute cap; pacing can.

Net WP1 position

Tier-1 pairs are scarce in both domains - the section 8 risk is confirmed. Commit to the Tier 1 / Tier 2 split. But banking plus an LLM proposer makes the 150-instance floor plausible at roughly 375 curated document pairs, against ~1,200 documents for gov.uk. That should be planned, not assumed.

2026-09-24 (later) — WP1 Tier-1 gate ANSWERED on real gov.uk data

The gating question is closed: Tier-1 pairs are scarce. Commit to the Tier 1 / Tier 2 split. Full write-up in docs/wp1_tier1_findings.md.

Done

benchmark/mining/fetch_govuk.py — fetches through the gov.uk Content API rather than scraping. The API returns a content_id, and that field turned out to decide the whole question.
Fetched 34 URLs / 17 candidate pairings / 32 documents, audited each pair, ran the miner, swept the threshold.
docs/wp1_tier1_findings.md — the gating deliverable.
316 tests pass (up from 303).

Three findings

Most "pairs" are one document wearing two URLs. Only 5 of 17 pairings are genuinely distinct content items. Ten are two parts of one gov.uk guide — /student-visa and /student-visa/money share a content_id. URL-based mining would have counted those as Tier 1 and inflated the naturally-occurring claim about 3x.
There is a third category the Tier 1 / Tier 2 binary does not cover. "Two parts of one guide" is not Tier 1 (one document, one author) and not Tier 2 (nothing was artificially split — a RAG chunker really does index them apart). Needs its own reporting; do not silently absorb it into either.
Yield: 4 Tier-1 candidates from 32 documents (0.12/doc). The 150-floor would need ~1,200 documents. Across a threshold sweep the count moves 2→21, but the Tier-1 share stays 5–18% — same-document pairs outnumber Tier-1 pairs by 5x to 50x at every setting. The share is the robust finding; the count is tunable.

Four harness bugs, found because the first run's answer was obviously wrong

The first run reported 939 Tier-1 candidates, 29.34 per document, "proceed with Tier 1 as the primary corpus". It had passed its synthetic demo cleanly.

Cross product — every rule sentence paired with every exception-looking sentence in the provider. Fixed with a topic-overlap requirement and a cap per rule.
Sentence splitting — gov.uk headings have no terminal punctuation, so they glued to the next sentence. Now splits on newlines, with a length floor and a navigation filter.
Weak cues treated as exception markers — subject to, where the, if the appear constantly in this register. Strong cues only now.
Five regex word boundaries were literal backspace bytes (�), written by a shell heredoc that interpreted the escape. Those patterns silently never matched. Repaired; a repo-wide scan found no other affected file.

None of the four changed the conclusion, but the first run would have produced exactly the inflated number the honesty guardrail exists to prevent. Worth remembering that a harness passing a synthetic demo says little about how it behaves on real prose.

The caveat that could still move the answer

All 34 URLs are gov.uk immigration. The proposal specifies two domains and financial terms are untested. Banks publish separate product pages, standalone fee schedules and dated amendment notices — a more Tier-1-friendly structure than a government guide with parts. Testing that domain is the cheapest remaining action that could change the corpus plan.

2026-09-24 — Ablation runner, zeroth-review deck, probe triage

Everything remaining that did not require Member A, Member B, or real data.

Done

experiments/run_ablations.py — the two named ablations that had no runner. The headline one (route conditional → factual) is Member A's alone: it measures routing, so it needs no LLM judge and no generation step. Relabel every conditional pair as factual — what a four-class detector emits — and those pairs route to selection, which keeps one passage. Result on the probe set: 95.0% of exception branches lost at the routing step. That is the argument for the conditional class existing, and it holds without the metric suite. Plus Tier-1/Tier-2 and explicit/implicit breakdowns.
Zeroth-review deck, slides 4–8 — docs/zeroth_review_member_a.html, published as an artifact. Each card is one slide, sized to screenshot into PowerPoint or print to PDF one page per slide. Every figure traces to a command in the repo. The honesty guardrail is on the slide, not buried: the deck says outright which numbers are structural results and which are fixture artifacts awaiting the corpus.
Probe review triage — --review now surfaces 14 genuine judgement calls ahead of the other 40 cases, each with the specific boundary it sits on and an accept/relabel line. Verification attention is finite; a reviewer who runs out of patience should run out of it on the easy cases.
6 more tests for the ablation runner. Suite: 297 → 303.

Note on the 95% figure

Measured on the 54-case probe set, which is conditional-heavy by construction. The denominator is exception branches only, so the rate is not inflated by the class balance — but it is still hand-built data, and the deck labels it as such. Re-run on the WP2 corpus before it appears in the paper.

Now genuinely blocked on Member A / Member B / real data

#	Item	What unblocks it
1	Verify 54 probe cases	`--review`, start with the 14 flagged
2	Contract sign-off	send `contract/schema/query_record-v1.0.0.json` to B
3	Rotate the Groq key	console → revoke → reissue → update `.env`
4	WP0 literature review	reading; the three anchors are in `docs/wp0_positioning_memo.md`
5	Tier-1 mining	~20 real provider URLs — then the harness runs
6	WP2 annotation	two annotators
7	kappa pilot	two annotators
8	Deck slides 1–3	co-authored with Member B

2026-09-22 (later still) — Closed the debt from today's own work

An audit of what I had built during the day, not of the original plan. Four issues, all self-inflicted, all now fixed.

Done

Tests for the six untested modules. tests/test_tooling.py, 51 tests covering threshold.py, train.py, run_baselines.py, run_retrieval_eval.py, wp1_probe_set.py and budget_estimate.py. Everything written that day had shipped with zero coverage while the core pipeline had 246 tests. Suite: 246 -> 297.
Stale trained heads are now rejected instead of silently used. models/stage1_head.pkl had been fitted before the cue-list split changed what exception_cues_* counts. The vector shape was unchanged, so it loaded fine and scored nonsense — the worst kind of failure, because nothing surfaces it. PairHead now records a feature_signature() hashing the feature names together with every cue vocabulary; load() refuses a mismatch with a retrain command, and Stage1Filter warns loudly rather than falling back in silence. The stale file was deleted.
--write now exists. threshold.py documented a flag that was never implemented, so tuning produced numbers with no path back into config. It now writes the chosen band to config/hardware.yaml, records tuned_on, and refuses to write a band tuned on a saturated curve. Stage1Filter reads its band from config, so a tuned band is actually used.
README documents the tooling. Eight of nine new tools were missing from the plan of record — the file Member B reads. Added, grouped by purpose, with the correct running order (train the head before tuning the band; the sweep is meaningless against the near-chance rule fallback). Status legend rewritten: several rows are DONE for the harness while the run itself is PENDING real data, and saying so is the point.

Bugs found while doing it

train.py was auto-loading an existing head while fitting a new one, so the "rule-based baseline" it reported was not the rule-based baseline.
My first --write patch silently failed to apply: the argparse flag landed but the handler did not, reproducing the exact documented-but-unimplemented bug I was fixing. Caught by running it rather than trusting the patch.
Running --write on mock data marked the committed config tuned: true. Reverted. test_shipped_thresholds_are_marked_untuned now guards it — a band fitted to templated data is worse than no band, because it looks legitimate.

State

Check	Result
Test suite	297 passed
Order-invariance gate	PASS, 1.0000
WP1 probe set	54/54 valid
Shipped escalation band	`tuned: false` — placeholder, as it should be

Still open — all non-code, all Member A's

Verify the 54 probe cases → unblocks Member B
Contract sign-off from Member B → blocks everything
Rotate the Groq key
WP0 literature review — gating
Tier-1 mining on ~20 real documents — gating, harness ready, no documents
WP2 annotation, 250–350 instances — largest time cost
kappa pilot with Member B
Zeroth review deck, slides 4–8

Two tools produce fixture artifacts until real data exists, and say so themselves: the threshold curve saturates on templates, and exception recall is floored by near-duplicate mock passages.

2026-09-22 (later) — WP1 probe set drafted (54 cases, awaiting Member A verification)

Done

benchmark/wp1_probe_set.py — 54 hand-built cases for Member B's WP1 feasibility probe, emitted as validated GoldInstance JSONL at benchmark/data/wp1_probe.jsonl.

Group	n	Purpose
Implicit conditions	28	the probe's core — what Contrastive Scope Probing must recover
Explicit conditions	8	control arm; explicit/implicit are reported separately
Distractors	14	factual, temporal, opinion, no-conflict, redundant — the SCR denominator
Disjoint / opposed	4	so the set covers all four scope relations

Every instance carries annotator_a="model-proposed", annotator_b=None. This is not gold data until Member A signs off. That is the workflow the proposal specifies (model proposes, human verifies), and reviewing is far faster than authoring.
python -m benchmark.wp1_probe_set --review prints a verification sheet with a four-point checklist per case and a why note saying exactly what the reader must infer.

Design notes

The 28 implicit cases each carry the narrowing by a different device — product name, scheme membership, tier, amount threshold, jurisdiction, time window, customer status, sponsor attribute, recharacterisation, negated sub-condition. A method that handles only one device is exposed rather than flattered.
Deliberate boundary cases for Member A to adjudicate: imp_008 (origination date — conditional, NOT temporal, since both texts remain in force); imp_019 and imp_020 (surface contradictions that are conditional, not factual); fin_red_049 (nested scope, agreeing outcome — the single most important distractor, since proposing a condition there fabricates a branch).
Prose is written for this benchmark, not copied from any provider: realistic in shape, fictional in content, so the set carries no licensing question. Tagged construction="split" so it can never be counted toward Tier 1.

Two problems the tooling caught in the draft

Four "implicit" cases contained exception cues. One (imp_009, "exempt from") was genuinely mislabelled and has been reworded. The other three were false positives from an overly broad cue list — see below. check_implicitness() now runs inside build_all(), so a mislabelled case cannot be emitted.
contract.validate reported no disjoint or opposed instances, with the note that a fixture missing a relation cannot catch confusions involving it. Four cases added; all four relations are now present.

A real defect in the detector, found while checking the probe set

EXCEPTION_CUES lumped strong exception markers together with generic conditional prose — "where the", "if the", "subject to". Those fire on almost every sentence in this domain, so exception_cues_max, one of the strongest signals for the conditional class, was partly measuring "this text describes a rule" rather than "this text states an exception". The noise landed directly on the factual/conditional cell the detector is judged on. Now split into STRONG_EXCEPTION_CUES and WEAK_CONDITION_CUES; the feature counts strong cues only.

Measured

Check	Result
Probe set validation	54/54 valid, all 4 scope relations
Implicit/explicit split	0 mislabelled (enforced at build time)
Test suite	246 passed
Baselines on probe prose	nli_filter 26.8% branch loss (vs 15.8% on templates)

That last row is worth noting: realistic varied phrasing is harder than the templates, which is exactly why the probe set exists.

Next for Member A

python -m benchmark.wp1_probe_set --review and verify all 54. Correct the Python source, not the JSONL, then regenerate.
Hand the verified JSONL to Member B to start the probe.
Remaining non-code items unchanged: contract sign-off, rotate the key, WP0, Tier-1 mining on real documents, WP2 annotation, kappa pilot, review deck.

2026-09-22 — Closed Member A's six code gaps

All six items from the completeness audit. Four of them surfaced real bugs.

Done

detection/threshold.py (gap 1) — sweeps the stage-1/stage-2 escalation band and reports the cost/accuracy curve with a Pareto front. Selection requires an explicit constraint (--max-escalation or --min-accuracy), because without one the answer is always "escalate everything", which is what the two-stage design exists to avoid. Detects and refuses to tune on a saturated curve.
detection/train.py (gap 3) — trains and persists the stage-1 head, compares it against the rule-based fallback on a held-out split, and refuses to save a head that does not beat the fallback. Stage1Filter auto-loads it.
experiments/run_baselines.py (gap 2) — scores the three retrieval-side baselines on branch loss, not answer correctness: did the exception passage survive to generation? Prints worked examples.
Variant (ii) now trains (gap 4) — two blockers fixed, see below. The three-architecture comparison is finally three-way.
experiments/run_retrieval_eval.py (gap 5) — recall@k and, more importantly, exception recall@k, comparing BM25 against hybrid.
scripts/eval_descriptors.py (gap 6) — measures the typed-attribute rate directly.

Bugs found by running these

Stage 1 was at chance. The threshold sweep showed stage-1 accuracy of 0.500 at zero escalation with the rule-based fallback. The earlier "70% of pairs resolved locally at zero API cost" was therefore measuring escalation rate, not correctness -- it was resolving them wrongly. After training the head: 0.460 -> 1.000 held-out. (1.000 is itself a fixture artifact; the mock templates are regular enough to memorise.)
fp16 master weights broke fine-tuning. Variant (ii) died with "Attempting to unscale FP16 gradients": some checkpoints carry a half-precision dtype in their config which from_pretrained honours, so parameters arrived as fp16 while GradScaler expects fp32 masters with fp16 activations. Fixed with an explicit .float(), which is version-agnostic where the dtype kwarg name is not. Also needed tiktoken + sentencepiece for the DeBERTa-v3 tokenizer.
A contradictory ConflictPair could be constructed. When stage 1 said "conflict" but the A3 classifier returned no_conflict, the pipeline built is_conflict=True, type=no_conflict -- which the contract rejects. Latent until the trained head changed stage-1's verdicts and exposed it. The classifier now wins: it answers how two passages disagree, and "not actually" is a legitimate answer.
compare() deferred a decidable case. When two applicability sets constrain different dimensions (account_kind vs age) it returned UNKNOWN and fell through to entailment. That case is provably OVERLAPPING: they necessarily intersect, and neither contains the other because each leaves the other's dimension unconstrained. Since UNKNOWN falls back to opposed, this was also costing composability on merely cross-cutting pairs. Live effect: exact attribute decisions 0% -> 44%, and compose fired 4x instead of 1x on the same input.

Measured

Check	Result
Test suite	246 passed
Order-invariance gate	PASS, 1.0000, 65/71 non-positional
Typed-attribute rate (live)	82% on representative passages
Baseline branch loss	rerank_top1 55.4%, nli_filter 15.8%, standard_rag 0%
Three-variant comparison	lexical_nli 0.988 acc, combined leak 0.010; other two flagged DEGENERATE

The suppression demo now runs, and it is the slide:

query:      Do I pay a fee on cross-border transfers?
nli_filter: dropped p1 as inconsistent (contradiction score 1.000)
should say: ...the fee is waived for premium-tier cardholders (per p1).
will say:   Cross-border transfers incur a 3.5% fee.

Two numbers that are fixture artifacts, not results

Exception recall reads 28.6%, but 65% of mock passages are near-duplicates of each other, so pooled retrieval is being asked an unanswerable question. The tool now detects this and says so before the number can be misread.
Stage-1 held-out accuracy of 1.000, and the saturated threshold curve, are both the templates being memorised.

Both need re-running on the WP2 corpus. Neither is quotable now.

Still open for Member A — none of it code:

Contract sign-off from Member B (blocks everything)
Rotate the Groq key
WP0 literature review, A's slice — gating
~50 labelled cases for B's probe — gating
Tier-1 mining on ~20 real documents — harness ready, zero documents — gating
WP2 annotation of 250-350 instances — the largest single time cost
kappa pilot with B on refinement-vs-opposed
Zeroth review deck, slides 4-8

2026-09-21 (later) — Groq wired up live; four provider findings

Key added, all four tiers verified against the real API, and the rate limits turned into a planning artifact rather than a surprise.

Done

.env created with the Groq key (gitignored; verified untracked).
Tier map set from the real free-tier limits (config/models.yaml), with those limits stored as data so tooling can reason about them.
scripts/smoke_test.py — live check: every tier reachable and non-empty, structured JSON parses, truncation fails loudly, cache serves the second call free. All green.
scripts/budget_estimate.py — multiplies pipeline call volume against the daily caps and reports wall-clock days per model.
api_budget hardened for reasoning models: reasoning_effort passthrough, TruncatedResponseError, per-model capability guard.
Live end-to-end run (5 instances, real DeBERTa on GPU + real Groq calls): detection resolved 80% of pairs locally at zero API cost, stage 2 escalated one, A4 ran live, routing produced one COMPOSE.
245 tests pass.

Findings — all four came from running the thing, not from reading docs

Groq rate limits are PER MODEL, not per account. Putting every tier on gpt-oss-120b caps the project at 200K tokens/day. Spreading BULK, JUDGE and SECOND_BACKBONE across three models gives 1.1M/day — 5.5x. This is the single reason the tier map looks the way it does; it is not a quality compromise.
gpt-oss is a reasoning model, and its reasoning tokens consume max_tokens before any answer appears. At max_tokens=20 it returned finish_reason='length' with empty content. The old backend would have returned '', every downstream JSON parse would have quietly fallen back, and a whole corpus run would have produced plausible-looking empty extractions. Now raises TruncatedResponseError on any length finish — partial output too, since JSON cut mid-string is garbage rather than an obvious error. reasoning_effort is set per tier (low for bulk, medium for load-bearing).
allam-2-7b rejected as the overflow model, despite having 2.5x the daily token budget of anything else. Two disqualifiers, both measured: it returns HTTP 400 on reasoning_effort, and it answers English prompts in Arabic by default. Mixed-language extractions on financial and immigration prose would be worse than a rate-limit stall and far harder to notice. Overflow is now qwen, which has spare quota.
groq/compound has no daily token cap but is unusable as the decisive baseline. It is an agentic system with built-in web search — it could look up the real policy instead of parsing the passages it was given, which would silently invalidate the one comparison the central claim rests on. The unlimited quota is a trap here. LONG_CONTEXT stays on gpt-oss-120b.

Separately: llama-prompt-guard-2-22m/86m have the highest request limits on the account and are not chat models — they are prompt-injection classifiers. canopylabs/orpheus-* are text-to-speech. All excluded.

The schedule problem, and the fix

python scripts/budget_estimate.py on the 300-instance target, 12 configs:

Plan	Wall clock
Naive (per-branch judging, all ablations on full corpus)	115 days
`--judge-batched --ablation-instances 100`	41 days

WP4 is roughly 42 days. The fitted plan fits with essentially zero slack. The metric judge is 65% of the total, so the two levers that matter are:

batch the judge per instance, not per branch — the passages and rubric are re-sent for every branch otherwise, roughly doubling the cost for identical judgements;
run non-headline ablations on a stratified subsample, headline configurations on the full corpus. Ordinary practice; report it as such.

Re-run the estimator before committing to any full-corpus pass. The cache makes re-runs free, so the number above is for a cold run only.

Decisions

BULK → openai/gpt-oss-20b, not allam. allam has 7x the request budget, but BULK feeds the factual/conditional decision — the paper's headline number — and buying throughput with quality there is a bad trade.
JUDGE / LONG_CONTEXT → openai/gpt-oss-120b (they share one 200K/day cap, so their days add; the estimator now says so explicitly).
SECOND_BACKBONE → qwen/qwen3.8-27b. Different family from gpt-oss, so the RQ3 robustness ablation is meaningful, and it carries its own 200K/day.
Rotate the API key. It was pasted into a chat transcript, so treat it as public regardless of what the transcript is used for.

Next up

Contract sign-off from Member B — still the gating item.
Rotate the Groq key; update .env.
Tell Member B to batch the metric judge per instance from the start. It is a 2x budget difference and much cheaper to design in than to retrofit.
A4 currently decides 100% of relations by entailment rather than typed attributes on mock data — the LLM extractor is not emitting typed attributes for these passages. Worth tuning the descriptor prompt on real text, since attribute decisions are exact and entailment ones are not.
Start WP0 (gating, reading not code).
Collect ~20 real documents for the Tier-1 yield trial.

2026-09-21 — Repository bootstrap: contract, API layer, and all of Member A

Done

Scaffold. Created conflict-rag/ following §7 of the implementation plan. Python 3.11 venv (not the system 3.13 — parts of the ML stack lag behind it), .gitignore covering .env / venv / model caches / cache DB / working data, .env.example, requirements.txt, pyproject.toml with pytest config and slow / network markers.
README.md — the plan of record: problem statement, A/B ownership split, the frozen contract, hardware and provider decisions, module build order with status, WP0–WP5 timeline, setup instructions, named risks.
CONVENTIONS.md — repo conventions: the session-log rule, the contract freeze rule, hardware rules, API rules, and the research-integrity rules carried over from §12 of the implementation plan.
contract/ (Week-1 joint deliverable, A leads) — Pydantic v2 wire schema (models.py), gold annotation schema (gold.py), the routing table as executable code (routing.py), a seeded deterministic mock generator (mock.py), JSON Schema export, and a validator CLI. CONTRACT_VERSION = 1.0.0, stamped on every record.
api_budget/ (Week-1 joint deliverable) — SQLite request cache keyed on everything that can change a response, provider-agnostic client over the openai SDK with configurable base_url (Groq + Ollama + any future provider on one code path), tiered model routing, JSONL cost log with a report CLI, and a deterministic offline mock backend.
config/ — hardware.yaml with three profiles (6 GB floor, 8 GB opt-in, CPU-only for CI) and models.yaml mapping logical tiers to models. scripts/discover_models.py populates the tier map from Groq's live model list rather than hardcoding IDs.
A1 retrieval/ — BM25 (rank_bm25) + dense (Contriever, fp16, local) + reciprocal-rank fusion at K = 5, over a passage store carrying source type, date and document_id.
A2 detection/ — two-stage detector. Stage 1: local DeBERTa-v3 NLI cross-encoder + hand-authored lexical features + a learned head over embedding interaction features. Stage 2: selective cached LLM escalation for the uncertain band only.
A3 detection/variants/ — all three architectures: lexical+NLI (with a coefficient-table explain()), fine-tuned DeBERTa-v3-base (multi-task: five classes + four-way relation), and structured entailment (the cross-encoder called twice and combined by the A4 rule). Plus compare.py, which reports the factual/conditional confusion cell as the headline.
A4 scope/ — the four-way relation analyser. Typed attribute algebra (attributes.py) for exact subset/disjoint decisions over tiers, ranges and categories; LLM + rule-based descriptor extraction; and relation.py, which decides applicability and outcome agreement separately and combines them.
scope/order_invariance.py — the permutation harness, wired to exit non-zero so it can gate CI.
baselines/retrieval_side.py — standard RAG, rerank-top-1, and the NLI-filter (the demonstrative one: on a conditional conflict it deletes exactly the exception branch).
metrics/ — A's scorers (binary detection, five-class with the named confusion cells, four-way scope with the four named cells, tier splitting) plus typed stubs for B's PR/SR/HCR/SCR with the agreed signatures.
Member B landing zone — extraction/, composition/, generation/ each hold an ownership README listing what goes there, what already exists to build against, which metrics B owns, and B's WP1 go/no-go.
WP support — benchmark/mining/tier1_pilot.py (the ~20-document Tier-1 yield trial, with a recommendation that changes based on measured yield), docs/wp0_positioning_memo.md (template with the three confirmed prior-art anchors and the venue checklist), benchmark/manual/annotation_manual.md (draft, including the refinement-vs-opposed decision procedure).
Tests — 242 passing, offline, zero API spend, ~3 seconds.

Verified by running

Check	Result
`pytest -m "not slow"`	242 passed
Mock determinism (same seed, two runs)	byte-identical SHA-256
Mock coverage at n=30	all 5 conflict types, all 4 scope relations, nested + crossed, explicit + implicit
Contract validation of generated records	60/60 valid
`scope.order_invariance --sample 128`	PASS, 1.0000, 71 permutations, 65/71 non-positional roles
`detection.variants.compare`	runs, emits the three-variant table
`experiments.run_pipeline --evaluate`	runs A1→A4, emits a valid record and all scorers
`benchmark.mining.tier1_pilot --demo`	runs, produces a yield-based recommendation
torch CUDA	2.6.0+cu124, RTX 4060 Laptop, 8.59 GB, `cuda.is_available() == True`
`pytest -m slow` (dense retrieval on GPU)	passed — Contriever downloads, encodes, and fuses with BM25
Full suite incl. slow	244 passed, 1 skipped

The one skip is test_second_backbone_is_a_different_family_from_judge, which correctly skips until SECOND_BACKBONE is assigned.

Decisions

Provider: Groq, zero frontier-API spend, local Ollama/Qwen 2.5 7B Q4 as the rate-limit fallback. Why: the user's constraint. The consequence is written into README §5 as a named risk rather than buried — see below.
The open-weights asymmetry. At K = 5 the structured long-context baseline's input is only ~2–5K tokens, so context length was never the binding constraint; capability is. That makes the decisive experiment interpretable in only one direction: if the open-weights baseline matches or beats the pipeline, that is valid falsification and still publishable; if it loses, a reviewer attributes the gap to model tier. The experiment is still worth running — it can falsify the claim but cannot confirm it. This must go in the paper. Model IDs are config, not code, so one line swaps in a frontier model if credits appear.
Not using DeepSeek-R1-8B for bulk work. It is a reasoning distill that emits long <think> traces — slow and awkward to parse across thousands of structured-extraction calls. Qwen 2.5 7B Instruct is the better local bulk model. R1-distill stays available as an extraction-model-size sweep row.
6 GB VRAM floor, 8 GB opt-in. This machine has 8 GB; Member B's is unconfirmed. Both profiles train at the same effective batch size (32), so the two machines produce the same model from one config — otherwise the comparison between them would be meaningless.
Local GPU never runs a generative model in the reported pipeline. It runs cross-encoders and embeddings. Putting a Q4 7B resident on the same 8 GB card would contend with the DeBERTa fine-tune and serialise the whole dev loop.
set_relation is stored canonically (as the exception relative to the default), not as compare()'s raw argument-relative output. Why: found by a failing test — the raw value mirrors SUBSET↔SUPERSET under argument swap, which is semantically correct but would have made the order-invariance harness report a false violation on every single refinement pair.
A redundant gold instance resolves to exactly one merged branch, not two. Why: found by the schema validator rejecting the mock generator's output. Recording a second branch would assert an exception that does not exist — the fabricated-branch failure that separating redundant from refinement exists to prevent.
Stage 1 alone never emits a conditional label. It establishes only that two passages disagree, not how. An unearned conditional label routes to composition and can fabricate a branch, so the honest default is factual.
An undecidable scope relation falls back to opposed (→ selection) rather than optimistically composing. Composing on unestablished scope invents a branch; selection is what prior work does and is the safe default.
Model: openai/gpt-oss-120b (chosen by Member A, set late in the session) for JUDGE and LONG_CONTEXT — the two load-bearing tiers. It is among the strongest open-weights models Groq serves, which is the right call for the metric judge and the decisive baseline.
- BULK is currently also pointed at it only so nothing is unresolved. This is the one tier that should move to a smaller model: bulk steps are thousands of calls and will exhaust a free-tier daily token cap long before the load-bearing steps do. Run scripts/discover_models.py and reassign.
- SECOND_BACKBONE deliberately left unset. It must be a different family from JUDGE — another gpt-oss model would make the robustness ablation (RQ3, "architectural rather than model-specific") vacuous. A test enforces this and skips until it is assigned.

Blocked / open

Contract sign-off from Member B. contract/schema/query_record-v1.0.0.json is ready to send. Nothing should be built against a contract B has not agreed to.
No Groq key on disk yet — THE remaining blocker for live runs. Everything above ran offline. .env does not exist; only .env.example does, and GROQ_API_KEY is not in the process or user environment either. Create .env with the key, then run python scripts/discover_models.py. Until then every live call raises a provider-not-configured error.
Member B's GPU size is unknown. The 6 GB floor is assumed. If their card is 4 GB, config/hardware.yaml needs a fourth profile.
FineTunedDetector has never been trained. The code is written and fits the memory budget on paper; it has not been run. It is the one A3 variant with no rule-based fallback.
SECOND_BACKBONE unassigned. Needs a non-gpt-oss model of comparable size, chosen after seeing Groq's real catalogue.
Supervisor should hear about the zero-spend consequence before WP4, with the asymmetry argument. It is a framing decision, not only a budget one.

Next up

Send contract/schema/query_record-v1.0.0.json to Member B and get the contract frozen. Everything downstream depends on it.
Create .env with GROQ_API_KEY, run python scripts/discover_models.py, then make one live BULK call and confirm the second identical call is served from cache at zero tokens. Reassign BULK to a smaller model and pick a non-gpt-oss SECOND_BACKBONE from the discovered catalogue.
Train FineTunedDetector on mock data — not for accuracy, but to confirm it fits in 8 GB at the stated batch size and that the training loop runs.
Start WP0. It is gating and it is reading, not code. Begin with the three confirmed anchors in docs/wp0_positioning_memo.md.
Collect ~20 real documents and run benchmark.mining.tier1_pilot for the actual Tier-1 yield. Due end of week 4.

Navigation Menu

Repository navigation

PROGRESS.md

PROGRESS.md

Progress log

Sessions

2026-10-31 (late night) - the human loop, K = 6 contexts, first live G1, and a rehearsal of everything after annotation

2026-10-31 (night) - scoped precision, the V1 and V2 drivers, the release bundle, the generated datasheet

2026-10-31 (later) - v7/v8 build: systems, detection, gates, analysis and the claims ledger

2026-10-31 - v7/v8 build, day 1: contract v1.3.0, CCA, the validity battery, Propositions 2-3, signatures

2026-10-30 - The pipeline runs end to end, live, as a product: the Studio and the engine server

2026-10-22 - The site rebuilt as a 3D scroll story; the engines untouched and now guarded four ways

2026-10-02 - Everything buildable is built. 501 tests.

2026-09-30 - Corpus at 128 of 300; Tier 1 done; kappa settled at p_e 0.54

2026-09-25 (evening) - Kappa pilot 1 failed on the type axis, and diagnosed cleanly

2026-09-25 (end of day) - WP0 closed, replacement opposed case, second annotator secured

2026-09-25 (later still) - SG-DT verified against the source. WP0's biggest risk closed, and an overclaim corrected.

2026-09-25 (later) - Corpus composition settled. Contract v1.2.0.

2026-09-25 - WP1 probe set human-verified. Gating item closed.

2026-09-24 (later still) - Member B built: B1, B2, B3 and the preservation metrics. Four scoring bugs found on the way.

2026-09-24 (later) - Annotation tooling, probe verification, WP0 first pass, and a schema bug that inverted four test cases

2026-09-24 - Cost/latency instrumentation, error analysis, and a key that should not have been committed

2026-09-24 (final) - The decisive experiment exists, and its first result was fake

2026-09-24 (final) - Statistical rigour layer, and a broken invariance guarantee

2026-09-24 (final) - WP1 batch 2: the estimate was optimistic, and that matters

2026-09-24 (later still) - WP1 domain 2: banking, and the proposer swap

2026-09-24 (later) — WP1 Tier-1 gate ANSWERED on real gov.uk data

2026-09-24 — Ablation runner, zeroth-review deck, probe triage

2026-09-22 (later still) — Closed the debt from today's own work

2026-09-22 (later) — WP1 probe set drafted (54 cases, awaiting Member A verification)

2026-09-22 — Closed Member A's six code gaps

2026-09-21 (later) — Groq wired up live; four provider findings

2026-09-21 — Repository bootstrap: contract, API layer, and all of Member A

FilesExpand file tree

PROGRESS.md

Latest commit

History

PROGRESS.md

File metadata and controls

Progress log

Sessions

2026-10-31 (late night) - the human loop, K = 6 contexts, first live G1, and a rehearsal of everything after annotation

2026-10-31 (night) - scoped precision, the V1 and V2 drivers, the release bundle, the generated datasheet

2026-10-31 (later) - v7/v8 build: systems, detection, gates, analysis and the claims ledger

2026-10-31 - v7/v8 build, day 1: contract v1.3.0, CCA, the validity battery, Propositions 2-3, signatures

2026-10-30 - The pipeline runs end to end, live, as a product: the Studio and the engine server

2026-10-22 - The site rebuilt as a 3D scroll story; the engines untouched and now guarded four ways

2026-10-02 - Everything buildable is built. 501 tests.

2026-09-30 - Corpus at 128 of 300; Tier 1 done; kappa settled at p_e 0.54

2026-09-25 (evening) - Kappa pilot 1 failed on the type axis, and diagnosed cleanly

2026-09-25 (end of day) - WP0 closed, replacement opposed case, second annotator secured

2026-09-25 (later still) - SG-DT verified against the source. WP0's biggest risk closed, and an overclaim corrected.

2026-09-25 (later) - Corpus composition settled. Contract v1.2.0.

2026-09-25 - WP1 probe set human-verified. Gating item closed.

2026-09-24 (later still) - Member B built: B1, B2, B3 and the preservation metrics. Four scoring bugs found on the way.

2026-09-24 (later) - Annotation tooling, probe verification, WP0 first pass, and a schema bug that inverted four test cases

2026-09-24 - Cost/latency instrumentation, error analysis, and a key that should not have been committed

2026-09-24 (final) - The decisive experiment exists, and its first result was fake

2026-09-24 (final) - Statistical rigour layer, and a broken invariance guarantee

2026-09-24 (final) - WP1 batch 2: the estimate was optimistic, and that matters

2026-09-24 (later still) - WP1 domain 2: banking, and the proposer swap

2026-09-24 (later) — WP1 Tier-1 gate ANSWERED on real gov.uk data

2026-09-24 — Ablation runner, zeroth-review deck, probe triage

2026-09-22 (later still) — Closed the debt from today's own work

2026-09-22 (later) — WP1 probe set drafted (54 cases, awaiting Member A verification)

2026-09-22 — Closed Member A's six code gaps

2026-09-21 (later) — Groq wired up live; four provider findings

2026-09-21 — Repository bootstrap: contract, API layer, and all of Member A