Skip to content

A held-out, repeatable score for the explanation layer (#346) - #385

Merged
shubhamkataria2005 merged 12 commits into
mainfrom
feature/346-explanation-quality-evaluation
Nov 2, 2026
Merged

shubhamkataria2005 merged 12 commits into
mainfrom
feature/346-explanation-quality-evaluation

Conversation

@patelankeet2

@patelankeet2 patelankeet2 commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #346.

What #346 actually needs

Three acceptance criteria: a held-out set of findings with human-rated reference explanations, a repeatable score run in CI with the number published, and a documented threshold below which the deterministic fallback is preferred.

The core constraint, stated up front because it shapes every decision below: CI runs with no Ollama and no Batfish (.github/workflows/tests.yml), and LLM output is not bit-for-bit repeatable anyway. So "repeatable" cannot mean calling a live model in CI -- it means freezing a held-out set (finding + captured explanation + human rating) as a committed fixture, and running pure arithmetic over it. That is the same tradeoff docs/traceability.md and docs/evaluation.md already make peace with: a measured snapshot, dated, re-run by hand when something that could change it changes (CONTRIBUTING.md 5c).

Alternatives considered for what goes in the held-out set

  1. Reuse docs/evaluation.md Part 3's exact 5 findings and their existing 4-rater scores as-is. Fast, already multi-rater -- but that text predates Generated explanations can invert which side (policy vs config) requires what #145's fix, so it would pin a known-fixed bug's output forever and call it current behaviour.
  2. Regenerate fresh explanations for the same 5 findings post-Generated explanations can invert which side (policy vs config) requires what #145, rate them myself as a v1 single-rater set, flagged honestly as a first round (the same way Part 3 itself started with one rater before the other three joined).
  3. Regenerate and get all four of us to re-rate before closing this issue. Most complete, but turns a size-M solo issue into a team-blocking one.

Went with 2. It measures what the system actually does today rather than a stale snapshot, and it matches how this project already bootstrapped Part 3's own rating exercise.

What changed

tests/fixtures/explanation_quality/holdout.json -- the same 5 findings Part 3 used (RT-050, AC-001, PC-001, AC-002, PC-005), regenerated live against the real pipeline and the real model on 10 October, rated by hand on Part 3's own 1-5 accuracy/usefulness scale. Held out in the literal sense -- none of these appear among ai/Modelfile's own worked examples.

tools/explanation_quality.py -- load_holdout() / score_holdout(), pure arithmetic, no Ollama or Batfish call. Reports accuracy and usefulness as two separate numbers (never blended -- same reasoning this project already gives for F-4 and the controls table), each as a mean and a minimum, because the mean alone hides exactly the case that matters (see below). Also defines EXPLANATION_QUALITY_THRESHOLD (mean accuracy 4.0, mean usefulness 3.0). Runs both documented tools/ invocation forms; imports neither analysis, ai, nor web, so it needs no sys.path guard (same exemption pfsense_shape.py already has -- confirmed by tests/test_tools_invocation.py itself, which now parametrizes over this file too).

tests/test_explanation_quality.py -- synthetic cases for multi-rater/multi-finding averaging (the real fixture currently has one rater per finding, which alone couldn't exercise that path), plus tests pinning the real committed holdout's published numbers against the documented threshold, and the F-4-style guard that scoring an empty set raises rather than returning a fake zero.

docs/evaluation.md Part 4 -- the full writeup, including a defect this PR does not fix (see below).

A real finding, from doing this properly rather than assumed away

Regenerating live (rather than reusing Part 3's stale text) paid off immediately: both PC-001 and PC-005 -- the two findings #145 was filed against -- regenerate correctly now. Confirmed, not assumed.

It also surfaced a different, new defect. AC-002's regenerated text describes the dead line's own action backwards (calls a PERMIT line one that "is supposed to block traffic"), even though the final traffic-outcome claim is correct -- a different fact than the one #52's existing guard protects. Checked for reproducibility before concluding anything: six more immediate live regenerations of the same finding, zero repeated it. Filed as #384 with the full repro log and a suggested (not applied) fix, and kept in the held-out set rated honestly low rather than re-rolled for a cleaner sample or fixed inline -- the same discipline #145 itself established: fixing the generator mid-measurement means rating text that no longer exists.

Verified, not assumed

python -m tools.explanation_quality
  5 findings, 5 ratings
  accuracy:   mean 4.40  min 2
  usefulness: mean 3.40  min 2

python -m pytest tests/test_explanation_quality.py -v
  8 passed

python -m pytest tests/ -q
  1836 passed, 15 skipped (main alone: 1824 passed, 15 skipped -- the
  +12 is this PR's 8 new tests plus 4 house-wide meta-tests that
  automatically parametrize over every new test/tool file)

Mutation-tested both safety properties that matter: temporarily made score_holdout() return a fake zero score for an empty set instead of raising -- test_scoring_an_empty_set_raises_rather_than_returning_a_fake_zero failed immediately. Temporarily raised EXPLANATION_QUALITY_THRESHOLD["mean_accuracy"] above the real measured 4.4 -- test_current_mean_quality_meets_the_documented_threshold failed immediately, for the right reason. Both reverted after.

Acceptance criteria (#346)

  • A held-out set of findings with human-rated reference explanations -- tests/fixtures/explanation_quality/holdout.json, 5 findings, rated with reasoning recorded per finding.
  • A repeatable score, run in CI, with the number published -- tools/explanation_quality.py + tests/test_explanation_quality.py, needs neither Batfish nor Ollama; published in docs/evaluation.md Part 4.
  • A documented threshold below which the deterministic fallback is preferred -- EXPLANATION_QUALITY_THRESHOLD, deliberately based on the mean rather than the minimum (reasoning in both the code comment and the doc), and deliberately a documented/human-reviewed decision rather than a runtime auto-disable, since there is no honest way to score one live generation's quality without another model judging it.

What this deliberately does not do

Does not touch ai/Modelfile, the three existing deterministic outcome guards, or anything on the live explanation-generation path -- this is a measurement added alongside, not a change to how explanations are generated. Does not fix #384 -- filed separately, on purpose. Does not require team sign-off before merging -- unlike #191, nothing here is a shared-contract change.

Left open, explicitly

A second independent rater on this held-out set (@ARSH871-bot @shubhamkataria2005 @SamikaPerera) would turn this from a first round into the same strength of evidence Part 3 reached with four raters. Not required to close this issue -- noted in docs/evaluation.md Part 4 as a next step, not a blocker.

Five real findings from the real pipeline (rtr-us5-insecure,
rtr-us5-messy), regenerated live against the real model after #145's
fix, rated by hand on the same accuracy/usefulness scale
docs/evaluation.md Part 3 already uses. Held out in the literal sense:
none of these findings or their text appear among ai/Modelfile's own
worked examples.

PC-001 and PC-005 are the two findings #145 was filed against -- both
regenerate correctly now, confirming the fix in practice rather than
by inspection.

AC-002's captured explanation has a real, different defect: it
describes the unreachable line's own action backwards (calls a PERMIT
line a rule that 'is supposed to block'), even though the final
traffic-outcome claim is correct. Checked for reproducibility (0 of 6
immediate re-runs repeated it) and filed as its own issue (#384)
rather than quietly swapped for a cleaner sample or fixed inline --
fixing it here would mean rating text that no longer exists.
tools/explanation_quality.py -- pure arithmetic over the frozen
held-out set, mean and minimum of each rating dimension separately
(not blended into one number, same reasoning docs/evaluation.md
already gives for the controls table and F-4 generally). Runs both
documented ways; imports neither analysis, ai, nor web, so it needs no
sys.path guard, same exemption pfsense_shape.py already has.
EXPLANATION_QUALITY_THRESHOLD (mean_accuracy 4.0, mean_usefulness 3.0),
deliberately based on the mean rather than the minimum -- a single
low-scoring finding is a correctness bug to fix for that shape (same
as #52, #145, #384), not a reason to disable the model for every
finding. No runtime auto-check exists: crossing this is detected by a
human re-running the held-out measurement, since there is no honest
way to score one live generation without another model judging it.

tests/test_explanation_quality.py: synthetic cases for the
multi-rater/multi-finding averaging the real fixture's single-rater
data can't exercise, plus tests pinning the real committed holdout's
published numbers against the documented threshold.

python -m pytest tests/test_explanation_quality.py -v -> 8 passed.
docs/evaluation.md Part 4 -- the full writeup: why this round
regenerated rather than reused Part 3's pre-#145 text, confirmation
that both PC-001/PC-005 inversions #145 was filed against are gone on
live regeneration, the new AC-002 defect found and filed as #384
(kept in the held-out set rather than re-rolled for a cleaner sample),
the published score, and why EXPLANATION_QUALITY_THRESHOLD is based on
the mean rather than the minimum and is a documented decision point
rather than a runtime check.

Explicitly left open for the team: a second independent rater on this
set, the same way Part 3 grew from one rater to four.
@patelankeet2 patelankeet2 added size: M About a week. phase: 3 Market readiness -- what stands between us and a product someone could buy. labels Oct 10, 2026
@patelankeet2 patelankeet2 self-assigned this Oct 10, 2026

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@patelankeet2, this is careful work, and keeping AC-002's bad capture in the set rather than re-rolling it is exactly right; #384 is the honest consequence. I am requesting changes for one thing about what the score measures, because this PR exists to produce that number.

What I checked that holds

python -m tools.explanation_quality         accuracy mean 4.40 min 2 · usefulness mean 3.40 min 2   (matches the PR)
python tools/explanation_quality.py         runs (script form)
tests/test_explanation_quality.py           8 passed
empty-set raise replaced with a fake zero   KILLED  (test_scoring_an_empty_set_raises...)
full suite on the head (contains main)      1794 passed, 42 skipped   ruff clean

The four model-written captures pass the live grounding guard (ai/explain.py's _is_unacceptable), so they are the kind of text the product can actually show. RT-050's captured text is byte-for-byte what _fallback_error_explanation() produces today. These tests need no Batfish, so the mutant result is the same either way.

Blocking: the score that decides "model or fallback" counts fallback text

RT-050's capture is captured_explanation_source: "fallback", rated 5/4 with the note "Deterministic fallback, so accuracy cannot be wrong by construction", and it is in the mean. Two consequences, both measured:

published (5 entries, 1 fallback)      mean accuracy 4.40   mean usefulness 3.40
model-written only (4 entries)         mean accuracy 4.25   mean usefulness 3.25
five fallback entries, no model text   mean accuracy 5.00   mean usefulness 4.00   <- passes the threshold

Today's conclusion does not change: 4.25/3.25 still clears 4.0/3.0. But the third row is the problem. EXPLANATION_QUALITY_THRESHOLD exists to say when the model is worse than the fallback, and as written it can be met by a set with no model output in it. That is a weaker claim standing in for a stronger one, and it cuts against the PR's own rule that accuracy and usefulness are "never blended": the same reasoning applies to blending the two sources.

RT-050 has a second issue. It is status="error", and web/main.py attaches explanations only to status="found" findings, so no user ever sees an explanation for it.

Suggested fix: score captured_explanation_source == "model" entries only, and raise if there are none, mirroring your empty-set guard. Then either drop RT-050 or report the fallback entries as their own line beside the model's. Update the published figures to the model-only numbers. A test that would have caught it: a held-out set of only fallback entries must not produce a passing model score.

Not blocking, but I would like it before this is relied on

Nothing makes the score stale when the model changes. I appended a comment line to ai/Modelfile and the suite stayed green, 8 of 8. So "CI green" on test_current_mean_quality_meets_the_documented_threshold will keep reading as "the explanation layer meets its threshold" after a prompt or base-model change that this set never measured. The PR is honest that this is a dated snapshot, but the test name says "current".

A cheap guard: record a sha256 of ai/Modelfile (and the base model tag) in holdout.json when capturing, and have one test fail when it no longer matches, with a message saying to re-capture and re-rate. It goes red exactly when the number has genuinely gone stale, unlike the #267 traceability case, where any new test turned it red.

Happy to be the second rater on the held-out set once the scoring is model-only. Re-request me when it is pushed.

Found by @ARSH871-bot: RT-050's fallback-sourced capture sat in the
same aggregate as model-written text, so the published score could
be met by a held-out set with no model output at all (five fallback
copies alone would score 5.00/4.00, comfortably above the threshold).
EXPLANATION_QUALITY_THRESHOLD exists to judge the model, and a
fallback explanation cannot be wrong by construction, so it says
nothing about that question. RT-050 is also status="error", which
web/main.py never attaches an explanation to, so no user sees it
regardless of source.

score_holdout() now filters to captured_explanation_source == "model"
for the headline numbers, raises ValueError if none remain (a
fallback-only set included), and reports any fallback-sourced entries
under their own "fallback" key rather than dropping them.

Published numbers move from 5 findings/4.40/3.40 to 4 model-written
findings/4.25/3.25 -- the conclusion (clears the 4.0/3.0 threshold)
does not change, but the number it is measured from now actually
measures what the threshold exists to judge.

python -m pytest tests/test_explanation_quality.py -v -> 13 passed (was 8).
EXPLANATION_QUALITY_THRESHOLD's own comment still cited the old
blended 4.4/3.4 measurement. docs/evaluation.md Part 4 now shows the
corrected model-only score (4.25/3.25), reports RT-050 on its own
fallback line instead of folded into the table, and carries a dated
correction note explaining what changed and why, per the project's
own convention of recording a correction rather than silently editing
a past measurement.
Non-blocking note from the review (@ARSH871-bot): nothing made the
published score go stale when the model that produced it changes --
appending one comment line to ai/Modelfile left the whole suite
green, because every test replays frozen captured text rather than
calling the model again.

holdout.json now records what it was captured against
(captured_against.modelfile_sha256, base_model). A new test compares
that recorded hash to ai/Modelfile's live one and fails loudly,
pointing at re-capturing and re-rating rather than at silently
updating the recorded hash.

python -m pytest tests/test_explanation_quality.py -v -> 14 passed (was 13).
@patelankeet2

Copy link
Copy Markdown
Collaborator Author

Both the blocking and the non-blocking points fixed.

Blocking: fallback-sourced captures no longer count toward the model score

Reproduced your numbers exactly before touching anything: all-5 gives 4.40/3.40, model-only gives 4.25/3.25, and five copies of RT-050's rating alone gives 5.00/4.00 -- confirmed that third case really does clear the threshold with zero model output behind it.

score_holdout() now filters to captured_explanation_source == "model" for the headline numbers and raises ValueError if none remain (a fallback-only set, mirroring the existing empty-set guard). Fallback-sourced entries are reported under their own "fallback" key, not dropped -- RT-050 stays in the committed set, just on its own line.

Added the test you named directly: a held-out set of only fallback entries now raises rather than scoring. Also added tests for the separation itself (mixed set keeps the fallback entry out of the headline numbers), a set with no fallback entries (no "fallback" key at all), and a defensive case for a record missing captured_explanation_source entirely (treated as not-model, same as any other non-"model" value).

Published figures in docs/evaluation.md Part 4 updated to 4.25/3.25, with a dated correction note (not silently edited) explaining what changed and why, including the second, independent reason RT-050 belongs out of the model score: it's status="error", and confirmed directly in web/main.py's attachment loop that an explanation is never attached to anything but status="found", so no user sees it regardless of source.

Non-blocking: staleness guard

Implemented your suggested fix directly. holdout.json now records captured_against.modelfile_sha256 (and the base model tag). A new test compares that against ai/Modelfile's live hash and fails with a message pointing at re-capturing and re-rating, not at updating the recorded hash. Mutation-tested by actually appending a line to ai/Modelfile -- the test failed immediately with the exact before/after hashes in the message, reverted after.

python -m pytest tests/test_explanation_quality.py -v -> 14 passed (was 8)
python -m pytest tests/ -q -> 1864 passed, 15 skipped, 0 failed

Re-requesting you -- and the offer to be second rater stands, now that the scoring is model-only.

@patelankeet2

Copy link
Copy Markdown
Collaborator Author

@ARSH871-bot -- your CHANGES_REQUESTED review is on 4be3c266, and two commits landed after it (1ea50c07, e308c3af, 3c110ecb actually three -- the fallback-exclusion fix plus the threshold/doc update plus the staleness-hash addition), including the exact fix for what you flagged. Void under M-2, same shape as #390 this week. Could you take a fresh look when you have a moment? #394 (US-61) needs this merged before it can use the real per-finding-type accuracy figures, so it's the one thing actually blocking a second specialist pick right now.

Both strings in _format_report() have no placeholders; ruff flagged them
as the two failures blocking CI on this PR.
@patelankeet2

Copy link
Copy Markdown
Collaborator Author

@ARSH871-bot the only thing still blocking this was CI, not the review itself.

Pushed a fix: tools/explanation_quality.py:251-252 had two plain strings marked as f-strings with no placeholders (ruff F541). Both commits since your review already address the substance of your feedback -- confirmed, not assumed:

python -m tools.explanation_quality
  4 model-written findings, 4 ratings
  accuracy:   mean 4.25  min 2
  usefulness: mean 3.25  min 2
  fallback-sourced (not scored against the threshold -- cannot be
  wrong by construction): RT-050

That matches the model-only numbers you asked for, and RT-050 is reported separately rather than blended into the mean, same as your "not blocking" staleness suggestion -- there's now a sha256 check against ai/Modelfile that fails the test when the held-out set goes stale.

pytest tests/test_explanation_quality.py -q    14 passed
pytest tests/ -q                                1864 passed, 15 skipped
ruff check --select E4,E7,E9,F ...              clean
CI on 59d368c                                   pytest (3.12) and (3.13): success

Re-requesting you now that it's actually green.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 59d368c, merged onto current main (b69d591). Both of Arsh's points from 11 October are fixed, and I checked each one. I'm requesting one more change, because the staleness guard covers the Modelfile but not the prompt, and your own #401 changes the prompt for exactly the finding this set rates lowest.

Arsh's two points:

  • Model-only score. python -m tools.explanation_quality now reports 4 model-written findings, accuracy 4.25 / usefulness 3.25, with RT-050 on its own fallback line. Mutants: putting the fallback entries back into the mean is killed, and removing the "no model-written entries" raise is killed.
  • Staleness. Appending a comment to ai/Modelfile fails test_the_holdout_set_is_captured_against_the_current_modelfile (1 failed, 13 passed). That's what Arsh asked for.
  • Merged: 1915 passed, 15 skipped, ruff clean. The two extra skips are test_tools_invocation's by-design skips for a tool that imports nothing from the package. Both invocation forms run.

Change requested: the prompt can change without the set noticing

What the model sees is ai/Modelfile plus _build_prompt(finding) in ai/explain.py, which includes the computed facts. The hash covers only the first. Measured: with #401 merged on top of this PR, all 14 tests still pass. But #401 rewrites the computed fact for AC-002, the entry this set rates 2 / 2. It's the #384 capture, and #401 exists to fix it. So the moment #401 lands, the published 4.25 still counts AC-002's old 2, from a prompt the product no longer sends, and CI stays green. #399 does the same for routing findings, although none are in this set.

Suggested fix, in the same spirit as the Modelfile hash: when capturing, record a sha256 of _build_prompt(finding) for each entry, and have one test rebuild the prompt and compare. Then the set goes stale exactly when what the model receives changes, per finding and no more often, so it won't turn into the always-red #267 case. On merge order: whichever of this and #401 lands second needs AC-002 re-captured and re-rated. With the per-entry hash, the test makes that impossible to miss.

Nits:

  • Hard-coding current_modelfile_sha256() to return the recorded value survives the tests, because no test checks it actually hashes the file. A two-line test on a temp file would close it.
  • Every entry still has one rating. Arsh offered to be second rater, and since this number decides "model or fallback", two raters would make it much stronger. That's not a merge blocker, as long as the doc says it's one person's rating.

The Modelfile hash cannot see a change to _build_prompt() itself.
Measured: with #401 merged on top of this held-out set, every test still
passed, because #401 rewrites the computed fact for AC-002 (this set's
own #384 capture, rated 2/2) without touching ai/Modelfile at all. The
published score would stay green while describing a prompt the product
no longer sends.

Added current_prompt_sha256(finding), recorded one hash per model-written
entry in holdout.json, and a test that rebuilds each prompt from the
stored finding dict and compares. One hash per finding rather than one
for the whole set, so a change to one computed-fact function marks only
the findings whose own prompt actually moved.

Also closes the nit that current_modelfile_sha256() was never shown to
hash file content rather than return a constant, and adds the
__package__ guard the new ai.explain import requires for the script
invocation form (tests/test_tools_invocation.py caught this immediately).

Addresses the change requested on PR #385 (review by Shubham).
@patelankeet2

Copy link
Copy Markdown
Collaborator Author

Fixed, and reproduced your exact scenario to confirm it.

Added current_prompt_sha256(finding), which hashes _build_prompt(finding) the same way current_modelfile_sha256() hashes the Modelfile. Recorded one captured_prompt_sha256 per model-written entry in holdout.json (computed now, against this branch's own ai/explain.py, before #401's change -- so it matches exactly what actually generated the 10 October captures), and added a test that rebuilds each entry's prompt from its own stored finding dict and compares.

One hash per finding rather than one for the whole set, so a change to one computed-fact function (the routing one, say) marks only the findings whose own prompt actually moved -- not the whole document, which is the #267 trap you named.

Reproduced your exact scenario before trusting the fix. Temporarily changed one word in _compute_dead_rule_outcome()'s return text (the same shape of change #401 makes to AC-002), and ran both staleness tests:

test_the_holdout_set_is_captured_against_the_current_modelfile   PASSED  (Modelfile untouched)
test_each_model_written_entry_is_captured_against_the_current_prompt
  FAILED: these held-out entries were captured against a prompt
  _build_prompt() no longer builds: ['AC-002']

Exactly the gap you found -- the old guard stays green, the new one catches it and names the right finding. Reverted the simulated change afterward.

Also closed both nits:

  • test_current_modelfile_sha256_actually_hashes_the_file_not_a_constant: writes two different temp files and confirms the hash changes between them, so a hard-coded return value can no longer pass silently.
  • Adding from ai.explain import _build_prompt gave this file its first cross-package import, which tests/test_tools_invocation.py caught immediately -- it needs the same __package__ guard preflight.py has for Fix preflight's Batfish misattribution and complete the setup instructions (#16) #172, now added. Confirmed both documented invocation forms work for real: python -m tools.explanation_quality and python tools/explanation_quality.py.
pytest tests/test_explanation_quality.py -q       16 passed (14 existing + 2 new)
pytest tests/ -q  (merged onto current main)      1919 passed, 13 skipped
ruff check --select E4,E7,E9,F ...                clean

Pushed as 2f84b0b. Re-requesting your review.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 2f84b0b. The per-finding prompt hash does exactly what I asked, and I tested it against both of the PRs it has to cope with.

Verified, on current main (d4535ef):

  • Merged: 2022 passed, 13 skipped, ruff clean on CI's selection. python -m tools.explanation_quality still gives 4.25 / 3.25 model-only, and python tools/explanation_quality.py works with the new __package__ guard.
  • With #401 merged on top, the new test fails and names the right entry:
    these held-out entries were captured against a prompt _build_prompt() no longer builds: ['AC-002']
    
    Yesterday the same combination was 14 of 14 green.
  • With #399 merged on top, all 16 still pass. That's correct: the set holds no model-written routing finding, so a routing-only prompt change shouldn't mark anything stale. That's the "per finding, not per document" property, and it's what keeps this out of #267's always-red trap.
  • Your claim that the hashes describe the 10 October prompts holds: git log --since=2026-10-09 origin/main -- ai/explain.py is empty, so _build_prompt() hasn't changed since the captures, and hashing it on 1 November gives the same prompt.

One thing to agree before merging, not a code change: merge order with #401. Whichever of the two lands second turns main's CI red, because this test fails as soon as both are in. That's this guard doing its job, but someone has to re-capture AC-002 under #401's prompt and re-rate it by hand before then. Two clean options:

  1. Merge #385 first, then add AC-002's re-capture and new rating to #401 before it merges, so #401's own CI shows the guard satisfied. I'd go with this one: the re-rating then sits in the PR that changed the prompt.
  2. Merge #401 first, and re-capture AC-002 in this PR before it merges.

Either way, the new AC-002 rating is the one to watch. It was 2/2 because of the exact inversion #401 fixes, so the published mean should go up, and it should be measured, not assumed.

@patelankeet2

Copy link
Copy Markdown
Collaborator Author

Approved on the current head and everything's verified. Please merge when you have a moment.

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator

Merging. My approval is of 2f84b0b, still the head. M-2 rule 2: tested merged onto current main (79f921b) together with #385, #399 and #407: 2240 passed, 13 skipped, ruff clean, CI green. Merged first on purpose: #401 now needs AC-002 re-captured under its new prompt and re-rated by hand before it can merge, because this PR's prompt-hash test will fail on it. That's the guard doing its job.

@shubhamkataria2005
shubhamkataria2005 merged commit 5db6498 into main Nov 2, 2026
2 checks passed
@shubhamkataria2005
shubhamkataria2005 deleted the feature/346-explanation-quality-evaluation branch November 2, 2026 02:49
patelankeet2 added a commit that referenced this pull request Nov 3, 2026
#385's per-finding prompt hash correctly flagged AC-002 as stale the
moment main (with #385 and #399) merged onto this branch -- the prompt
_build_prompt() builds for it moved when the previous commit fixed #384.

Re-captured one live generation under the new prompt and rated it by
hand (Ankeet): accuracy 5, usefulness 4, up from 2/2. The old rating's
defect is gone -- the dead line's own PERMIT action is now named
correctly, and the real blocked outcome is stated without inverting
which rule is which.

Updated docs/evaluation.md Part 4 with a dated note recording the
re-capture rather than silently rewriting the AC-002 row, and updated
the two tests that pin the published numbers (mean accuracy 4.25 -> 5.0,
mean usefulness 3.25 -> 3.75; the breakdown's low-scorer demonstration
now points at PC-005's usefulness, the new minimum, since AC-002 no
longer has an outlier to show).

Full suite: 2245 passed, 13 skipped, 0 failed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

phase: 3 Market readiness -- what stands between us and a product someone could buy. size: M About a week.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A dead-rule explanation can invert the unreachable line's own stated action US-55 — Show me the explanation layer is any good

3 participants