Repository navigation
A repeatable score for the question layer, run in CI (#393) - #407
Conversation
ai/query.py calls no model in either direction (CLAUDE.md 7c), so unlike #385's held-out set for explanations -- which had to freeze a human rating of model-written prose, since that varies run to run -- this one freezes the INPUT (a question plus a FakeSession configuration) and the expected outcome, then actually executes answer_question() against it every time the tool runs. There is nothing here that can go stale the way #385's captured text can, so there is no staleness-hash check. 20 entries, mirroring docs/evaluation.md Part 2's own four-outcome shape (answered-correct, refused-correctly, refused-but-should-answer, answered-wrong -- the dangerous one). 10 answered-correct, 10 refused-correctly, built from scenarios already proven individually in tests/test_query_translation.py rather than invented fresh, plus the #70 regression (a network destination resolved to a real host, not the network address) pinned by name as Q05. Verified live before trusting anything: ran both documented invocation forms (python -m tools.query_quality, python tools/query_quality.py) against the real code, 20/20 matched expectations, 0 answered_wrong. python -m tools.query_quality 20 entries, 10/10/0/0, 0 mismatches
…us the committed fixture against the CI threshold Most of the risk here is the classification arithmetic in score_holdout(), not the committed fixture, so most of these are small synthetic entries exercising one outcome at a time -- including the specific gap found while writing it: an entry with no answer_contains/ answer_starts_with set must still land in answered_wrong if the code answers when it should have refused, not fall through to answered_correct for lack of anything to text-match against. Then pins the real committed holdout.json against the documented threshold (max_answered_wrong = 0), the same published-number pattern #385's own test file already uses, plus a check that every entry's actual outcome matches what it claims to expect and a named pin for the #70 regression case. python -m pytest tests/test_query_quality.py tests/test_query_translation.py tests/test_tools_invocation.py -v 60 passed, 10 skipped
Dated update note, not a rewrite -- the original 10 August measurement stays as the historical record, same convention this file already uses elsewhere (the #70 entry a few paragraphs above does the same). Points at tools/query_quality.py and names the one real difference from #346's held-out set: no model means no staleness risk, so there is no hash check here, by design, not by oversight.
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Reviewed b294fe4, merged onto current main (b69d591). The code does what it says. I'm requesting changes because of what docs/evaluation.md would say about it.
What holds up:
- Merged: 1915 passed, 13 skipped, ruff clean.
python -m tools.query_qualitygives 10 / 10 / 0 / 0, andpython tools/query_quality.pyworks too. - Answering when a refusal was expected always counts as answered-wrong, and the mismatch test means a shift in any entry fails CI, not only an answered-wrong. Good.
- Running the real
answer_question()every time, rather than replaying output, is the right call for a module with no model in it.
Change requested: this set can't show "the same clean record"
The description says the 20 entries were "built from scenarios already individually proven in tests/test_query_translation.py rather than invented fresh". So 0 answered-wrong is guaranteed by construction. Every entry is a case a test already makes pass. That makes it a useful regression set, but it isn't held out, and it can't show what the 10 August measurement showed. To check, I gave answer_question() eight phrasings that aren't in the set, on Q01's own fake session (rtr-us5 → 218.8.104.58, ACCEPTED):
grounded question understood as / answer
False Is 218.8.104.58 reachable from rtr-us5? refused
True can rtr-us5 get to 218.8.104.58 "Yes. Traffic from rtr-us5 reaches 218.8.104.58." fine
False Does rtr-us5 have connectivity to 218.8.104.58? refused
False Can 218.8.104.58 reach rtr-us5? refused
True Can rtr-us5 NOT reach 218.8.104.58? "Yes. Traffic from rtr-us5 reaches ..." <- negation dropped
True Why can't rtr-us5 reach 218.8.104.58? "Yes. Traffic from rtr-us5 reaches ..."
True Can rtr-us5 reach 218.8.104.58 on port 443? understood "Can rtr-us5 reach 218.8.104.58?" <- port dropped
True Can rtr-us5 reach 218.8.104.58 over udp? understood "Can rtr-us5 reach 218.8.104.58?" <- protocol dropped
question_understood does show what was really asked, which is §7c's safety argument. But docs/evaluation.md's own #70 entry sets the bar: "judged by what a user meant it was wrong". By that bar, the last three are answered-wrong. The port one is also a change since 10 August: back then "is port 443 open to the internal server" was refused, and now a port question is answered with the port silently dropped.
None of this is caused by your PR. These are ai/query.py behaviours on main. But the doc update says "equivalent held-out question set ... with the same clean record: 0 answered-wrong", and a reader will take that as a re-measurement. Asked for:
- In
docs/evaluation.md, call it a regression set built from the existing tests, not an equivalent held-out set, and drop "the same clean record". That claim needs questions nobody tuned against. - Your call whether the negation/port/protocol cases go into the set now (they'd fail the threshold until
ai/query.pyrefuses them) or into a new issue first. It's your layer. Either is fine, as long as they're written down somewhere they'll be seen.
Nit: _FakeQuestions/FakeSession are a second copy of the fakes in tests/test_query_translation.py. If ai/query.py starts asking Batfish a new question, the two will drift. The mismatch test would catch it as refusals, so it's not urgent.
Every entry is a scenario tests/test_query_translation.py already makes pass, so 0 answered-wrong is guaranteed by construction, not measured. Dropped the 'same clean record' framing, which read as a genuine re-measurement against the 10 August baseline when it cannot be one. Checked with eight phrasings outside the set on Q01's own fixture, reproduced independently before writing this: a negation, a port constraint and a protocol constraint are each dropped and answered confidently rather than refused. The port case is a real regression since 10 August, when the equivalent question was refused. Tracked as #435, not fixed here -- it is a change to ai/query.py, not to this scoring tool or its own held-out set (#393's own acceptance criteria is the score, not the translation fix). Addresses the change requested on PR #407 (review by Shubham).
|
Fixed the doc, and filed #435 for the three real gaps rather than folding them in here.
Independently reproduced your eight-phrasing check before writing any of this, same fixture (Q01, All three confirmed exactly as you found them. The doc now names all three plainly, including that the port case is a real regression against the 10 August baseline (refused then, answered with the port dropped now). On your question -- filed #435 rather than adding these to the set now. #393's own acceptance criteria is the score, not a fix to Nothing in Pushed as |
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Approving 21e4c18. The doc now says what the set actually shows.
docs/evaluation.mdcalls it a regression set and explains why 0 answered-wrong is guaranteed by construction. It drops "the same clean record", names the negation, port and protocol cases, and calls out the port case as a regression since 10 August. That's the change I asked for, and the wording is accurate.- #435 is a good home for the
ai/query.pygaps. It has the live reproduction, the regression point, and a narrow-or-refuse suggestion for each case, so nothing is lost when this merges. Keeping known-failing cases out of a PR that builds the scoring is the right split. - The tool and fixture are unchanged since
b294fe4, which I'd already checked. Merged onto currentmain(d4535ef): 2018 passed, 13 skipped, ruff clean on CI's selection, andpython -m tools.query_qualityis still 10 / 10 / 0 / 0.
Small thing for #435, not this PR: "Why can't rtr-us5 reach 218.8.104.58?" was also answered "Yes. Traffic ... reaches". #435's negation bullet covers "can't reach", so it's in scope there. Just make sure the fix's tests include that phrasing as well as "NOT reach".
|
Approved on the current head and everything's verified. Please merge when you have a moment. |
Closes #393 (US-60).
What it does
A held-out question set for
ai/query.py, scored every time it runs, mirroringdocs/evaluation.mdPart 2's own four-outcome shape from the one hand measurement that has existed since 10 August (answered-correct, refused-correctly, refused-but-should-answer, and the dangerous one, answered-wrong).The one design difference from #385's held-out set, and why
#385 (
tools/explanation_quality.py) had to freeze a human rating of model-written prose, because the model's text varies run to run and judging whether it reads well needs a person.ai/query.pycalls no model in either direction (CLAUDE.md7c) -- it is pure, deterministic classification and resolution over a snapshot. There is nothing here that can go stale the way #385's captured text can, so:FakeSessionconfiguration, the exact harnesstests/test_query_translation.pyalready uses) and the expected outcome.answer_question()against every entry, every single run -- in a unit test, in CI, by hand, two years from now. Stronger evidence than replaying a snapshot of a past run, not weaker.The held-out set
20 entries (
tests/fixtures/query_quality/holdout.json), 10 answered-correct / 10 refused-correctly, built from scenarios already individually proven intests/test_query_translation.pyrather than invented fresh -- plus Q05, named explicitly as a regression pin for the #70 bug (a network destination resolved to a real host, not the network address, which the historical measurement found and fixed the same day). The exact 20 questions measured on 10 August were never preserved verbatim indocs/evaluation.md, so this is a rebuilt, equivalent set rather than a literal replay -- said plainly in the doc update.A real gap I found in my own first version
While writing the classification logic, caught a case where an entry with no
answer_contains/answer_starts_withto check against would silently fall through toanswered_correctif the code answered when it should have refused -- nothing to text-match against means the check just... passed. Fixed before it ever shipped:should_groundis now compared explicitly against the entry's ownoutcome_expected, so "answered when a refusal was expected" is alwaysanswered_wrong, regardless of what the answer text happens to say. Pinned with its own test (test_answering_when_a_refusal_was_expected_is_answered_wrong_regardless_of_text) and mutation-tested by reverting to the original logic -- failed for exactly the right reason, reverted.Verified
Mutation-tested twice: flipped the committed fixture's own expectation on one entry (caught by both the threshold test and the mismatch test, reverted), and reverted the classification-logic fix described above (caught by its own dedicated test, reverted).
Acceptance criteria
ai/query.py, scored with the same method as A held-out, repeatable score for the explanation layer (#346) #385 (the pattern -- A held-out, repeatable score for the explanation layer (#346) #385 itself isn't merged yet; see note below)FakeSessionhere needs neitherdocs/evaluation.mdPart 2 doesOne thing worth knowing before merge
#393 says "depends on #385: its method is reused, so it merges first." #385 isn't merged -- its last
CHANGES_REQUESTEDreview is stale under M-2 (flagged separately, on #385 itself). This PR only reuses #385's pattern (the held-out-set-plus-documented-threshold shape), not its code or data, so it does not need #385 to land first -- #394 (US-61) is the one that actually needs #385's real numbers. Flagging this so nobody assumes the dependency note above means this PR is blocked when it isn't.