Skip to content

A repeatable score for the question layer, run in CI (#393) - #407

Merged
shubhamkataria2005 merged 5 commits into
mainfrom
feature/393-query-quality-score
Nov 2, 2026
Merged

shubhamkataria2005 merged 5 commits into
mainfrom
feature/393-query-quality-score

Conversation

@patelankeet2

Copy link
Copy Markdown
Collaborator

Closes #393 (US-60).

What it does

A held-out question set for ai/query.py, scored every time it runs, mirroring docs/evaluation.md Part 2's own four-outcome shape from the one hand measurement that has existed since 10 August (answered-correct, refused-correctly, refused-but-should-answer, and the dangerous one, answered-wrong).

The one design difference from #385's held-out set, and why

#385 (tools/explanation_quality.py) had to freeze a human rating of model-written prose, because the model's text varies run to run and judging whether it reads well needs a person. ai/query.py calls no model in either direction (CLAUDE.md 7c) -- it is pure, deterministic classification and resolution over a snapshot. There is nothing here that can go stale the way #385's captured text can, so:

  • No frozen output, no staleness-hash check.
  • The held-out set freezes the input (a question plus a FakeSession configuration, the exact harness tests/test_query_translation.py already uses) and the expected outcome.
  • The tool actually executes answer_question() against every entry, every single run -- in a unit test, in CI, by hand, two years from now. Stronger evidence than replaying a snapshot of a past run, not weaker.

The held-out set

20 entries (tests/fixtures/query_quality/holdout.json), 10 answered-correct / 10 refused-correctly, built from scenarios already individually proven in tests/test_query_translation.py rather than invented fresh -- plus Q05, named explicitly as a regression pin for the #70 bug (a network destination resolved to a real host, not the network address, which the historical measurement found and fixed the same day). The exact 20 questions measured on 10 August were never preserved verbatim in docs/evaluation.md, so this is a rebuilt, equivalent set rather than a literal replay -- said plainly in the doc update.

A real gap I found in my own first version

While writing the classification logic, caught a case where an entry with no answer_contains/answer_starts_with to check against would silently fall through to answered_correct if the code answered when it should have refused -- nothing to text-match against means the check just... passed. Fixed before it ever shipped: should_ground is now compared explicitly against the entry's own outcome_expected, so "answered when a refusal was expected" is always answered_wrong, regardless of what the answer text happens to say. Pinned with its own test (test_answering_when_a_refusal_was_expected_is_answered_wrong_regardless_of_text) and mutation-tested by reverting to the original logic -- failed for exactly the right reason, reverted.

Verified

python -m tools.query_quality          20 entries, 10/10/0/0, 0 mismatches
python tools/query_quality.py          same -- both documented invocation forms

python -m pytest tests/test_query_quality.py tests/test_query_translation.py tests/test_tools_invocation.py -v
  60 passed, 10 skipped

python -m pytest tests/ -q
  1901 passed, 13 skipped   (main: 1885 passed -- +16 accounted for by the new
                             test file plus tools/test_tools_invocation.py's
                             auto-discovery of the new tool)

Mutation-tested twice: flipped the committed fixture's own expectation on one entry (caught by both the threshold test and the mismatch test, reverted), and reverted the classification-logic fix described above (caught by its own dedicated test, reverted).

Acceptance criteria

One thing worth knowing before merge

#393 says "depends on #385: its method is reused, so it merges first." #385 isn't merged -- its last CHANGES_REQUESTED review is stale under M-2 (flagged separately, on #385 itself). This PR only reuses #385's pattern (the held-out-set-plus-documented-threshold shape), not its code or data, so it does not need #385 to land first -- #394 (US-61) is the one that actually needs #385's real numbers. Flagging this so nobody assumes the dependency note above means this PR is blocked when it isn't.

ai/query.py calls no model in either direction (CLAUDE.md 7c), so unlike
#385's held-out set for explanations -- which had to freeze a human
rating of model-written prose, since that varies run to run -- this one
freezes the INPUT (a question plus a FakeSession configuration) and the
expected outcome, then actually executes answer_question() against it
every time the tool runs. There is nothing here that can go stale the
way #385's captured text can, so there is no staleness-hash check.

20 entries, mirroring docs/evaluation.md Part 2's own four-outcome shape
(answered-correct, refused-correctly, refused-but-should-answer,
answered-wrong -- the dangerous one). 10 answered-correct, 10
refused-correctly, built from scenarios already proven individually in
tests/test_query_translation.py rather than invented fresh, plus the #70
regression (a network destination resolved to a real host, not the
network address) pinned by name as Q05.

Verified live before trusting anything: ran both documented invocation
forms (python -m tools.query_quality, python tools/query_quality.py)
against the real code, 20/20 matched expectations, 0 answered_wrong.

python -m tools.query_quality
  20 entries, 10/10/0/0, 0 mismatches
…us the committed fixture against the CI threshold

Most of the risk here is the classification arithmetic in
score_holdout(), not the committed fixture, so most of these are small
synthetic entries exercising one outcome at a time -- including the
specific gap found while writing it: an entry with no answer_contains/
answer_starts_with set must still land in answered_wrong if the code
answers when it should have refused, not fall through to
answered_correct for lack of anything to text-match against.

Then pins the real committed holdout.json against the documented
threshold (max_answered_wrong = 0), the same published-number pattern
#385's own test file already uses, plus a check that every entry's
actual outcome matches what it claims to expect and a named pin for
the #70 regression case.

python -m pytest tests/test_query_quality.py tests/test_query_translation.py tests/test_tools_invocation.py -v
  60 passed, 10 skipped
Dated update note, not a rewrite -- the original 10 August measurement
stays as the historical record, same convention this file already uses
elsewhere (the #70 entry a few paragraphs above does the same). Points
at tools/query_quality.py and names the one real difference from #346's
held-out set: no model means no staleness risk, so there is no hash
check here, by design, not by oversight.
@patelankeet2 patelankeet2 added enhancement New feature or request layer: ai Layer 2 — local LLM explanation and Q&A phase: 3 Market readiness -- what stands between us and a product someone could buy. labels Oct 20, 2026
@patelankeet2 patelankeet2 self-assigned this Oct 20, 2026

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed b294fe4, merged onto current main (b69d591). The code does what it says. I'm requesting changes because of what docs/evaluation.md would say about it.

What holds up:

  • Merged: 1915 passed, 13 skipped, ruff clean. python -m tools.query_quality gives 10 / 10 / 0 / 0, and python tools/query_quality.py works too.
  • Answering when a refusal was expected always counts as answered-wrong, and the mismatch test means a shift in any entry fails CI, not only an answered-wrong. Good.
  • Running the real answer_question() every time, rather than replaying output, is the right call for a module with no model in it.

Change requested: this set can't show "the same clean record"

The description says the 20 entries were "built from scenarios already individually proven in tests/test_query_translation.py rather than invented fresh". So 0 answered-wrong is guaranteed by construction. Every entry is a case a test already makes pass. That makes it a useful regression set, but it isn't held out, and it can't show what the 10 August measurement showed. To check, I gave answer_question() eight phrasings that aren't in the set, on Q01's own fake session (rtr-us5 → 218.8.104.58, ACCEPTED):

grounded  question                                          understood as / answer
False     Is 218.8.104.58 reachable from rtr-us5?           refused
True      can rtr-us5 get to 218.8.104.58                   "Yes. Traffic from rtr-us5 reaches 218.8.104.58."   fine
False     Does rtr-us5 have connectivity to 218.8.104.58?   refused
False     Can 218.8.104.58 reach rtr-us5?                   refused
True      Can rtr-us5 NOT reach 218.8.104.58?               "Yes. Traffic from rtr-us5 reaches ..."   <- negation dropped
True      Why can't rtr-us5 reach 218.8.104.58?             "Yes. Traffic from rtr-us5 reaches ..."
True      Can rtr-us5 reach 218.8.104.58 on port 443?       understood "Can rtr-us5 reach 218.8.104.58?"   <- port dropped
True      Can rtr-us5 reach 218.8.104.58 over udp?          understood "Can rtr-us5 reach 218.8.104.58?"   <- protocol dropped

question_understood does show what was really asked, which is §7c's safety argument. But docs/evaluation.md's own #70 entry sets the bar: "judged by what a user meant it was wrong". By that bar, the last three are answered-wrong. The port one is also a change since 10 August: back then "is port 443 open to the internal server" was refused, and now a port question is answered with the port silently dropped.

None of this is caused by your PR. These are ai/query.py behaviours on main. But the doc update says "equivalent held-out question set ... with the same clean record: 0 answered-wrong", and a reader will take that as a re-measurement. Asked for:

  1. In docs/evaluation.md, call it a regression set built from the existing tests, not an equivalent held-out set, and drop "the same clean record". That claim needs questions nobody tuned against.
  2. Your call whether the negation/port/protocol cases go into the set now (they'd fail the threshold until ai/query.py refuses them) or into a new issue first. It's your layer. Either is fine, as long as they're written down somewhere they'll be seen.

Nit: _FakeQuestions/FakeSession are a second copy of the fakes in tests/test_query_translation.py. If ai/query.py starts asking Batfish a new question, the two will drift. The mismatch test would catch it as refusals, so it's not urgent.

Every entry is a scenario tests/test_query_translation.py already makes
pass, so 0 answered-wrong is guaranteed by construction, not measured.
Dropped the 'same clean record' framing, which read as a genuine
re-measurement against the 10 August baseline when it cannot be one.

Checked with eight phrasings outside the set on Q01's own fixture,
reproduced independently before writing this: a negation, a port
constraint and a protocol constraint are each dropped and answered
confidently rather than refused. The port case is a real regression
since 10 August, when the equivalent question was refused. Tracked as
#435, not fixed here -- it is a change to ai/query.py, not to this
scoring tool or its own held-out set (#393's own acceptance criteria is
the score, not the translation fix).

Addresses the change requested on PR #407 (review by Shubham).
@patelankeet2

Copy link
Copy Markdown
Collaborator Author

Fixed the doc, and filed #435 for the three real gaps rather than folding them in here.

docs/evaluation.md now calls this a regression set, drops "the same clean record", and explains why: every entry is a scenario tests/test_query_translation.py already makes pass, so 0 answered-wrong is guaranteed by construction, not measured.

Independently reproduced your eight-phrasing check before writing any of this, same fixture (Q01, rtr-us5 to 218.8.104.58, ACCEPTED):

Can rtr-us5 NOT reach 218.8.104.58?            grounded=True  "Yes. Traffic from rtr-us5 reaches..."
Can rtr-us5 reach ... on port 443?             grounded=True  understood without the port
Can rtr-us5 reach ... over udp?                grounded=True  understood without the protocol

All three confirmed exactly as you found them. The doc now names all three plainly, including that the port case is a real regression against the 10 August baseline (refused then, answered with the port dropped now).

On your question -- filed #435 rather than adding these to the set now. #393's own acceptance criteria is the score, not a fix to ai/query.py's translation, and bundling a known-failing case into the PR that builds the scoring infrastructure would make this PR's own CI red for a reason unrelated to what it closes. #435 has the live reproduction, why the port case is a regression and not just a gap, and a suggested next step for each of the three (narrow the translation or refuse, never silently drop the constraint -- same rule policy_conflicts.py already applies to an unmodelled query field).

pytest tests/test_query_quality.py tests/test_query_translation.py -q   43 passed
pytest tests/ -q  (merged onto current main)                           1915 passed, 13 skipped
ruff check --select E4,E7,E9,F ...                                     clean

Nothing in tools/query_quality.py or the held-out fixture itself changed -- this PR's own work was correct, the gap was entirely in what the doc claimed about it.

Pushed as 21e4c18. Re-requesting your review.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 21e4c18. The doc now says what the set actually shows.

  • docs/evaluation.md calls it a regression set and explains why 0 answered-wrong is guaranteed by construction. It drops "the same clean record", names the negation, port and protocol cases, and calls out the port case as a regression since 10 August. That's the change I asked for, and the wording is accurate.
  • #435 is a good home for the ai/query.py gaps. It has the live reproduction, the regression point, and a narrow-or-refuse suggestion for each case, so nothing is lost when this merges. Keeping known-failing cases out of a PR that builds the scoring is the right split.
  • The tool and fixture are unchanged since b294fe4, which I'd already checked. Merged onto current main (d4535ef): 2018 passed, 13 skipped, ruff clean on CI's selection, and python -m tools.query_quality is still 10 / 10 / 0 / 0.

Small thing for #435, not this PR: "Why can't rtr-us5 reach 218.8.104.58?" was also answered "Yes. Traffic ... reaches". #435's negation bullet covers "can't reach", so it's in scope there. Just make sure the fix's tests include that phrasing as well as "NOT reach".

@patelankeet2

Copy link
Copy Markdown
Collaborator Author

Approved on the current head and everything's verified. Please merge when you have a moment.

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator

Merging. My approval is of 21e4c18, still the head. M-2 rule 2: tested merged onto current main (79f921b) together with #385, #399 and #407: 2240 passed, 13 skipped, ruff clean, CI green. #435 tracks the ai/query.py gaps the regression set doesn't cover yet.

@shubhamkataria2005
shubhamkataria2005 merged commit db59b00 into main Nov 2, 2026
2 checks passed
@shubhamkataria2005
shubhamkataria2005 deleted the feature/393-query-quality-score branch November 2, 2026 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request layer: ai Layer 2 — local LLM explanation and Q&A phase: 3 Market readiness -- what stands between us and a product someone could buy.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

US-60 — A repeatable score for ai/query.py, run in CI

2 participants