Repository navigation
Description
Found while reviewing #407 (closes #393, US-60) -- @shubhamkataria2005 checked ai/query.py against eight phrasings outside the new held-out set, on Q01's own fake session (rtr-us5 to 218.8.104.58, ACCEPTED), and found three real gaps.
The bug, measured live
question understood as / answer
Can rtr-us5 NOT reach 218.8.104.58? "Yes. Traffic from rtr-us5 reaches ..." <- negation dropped
Can rtr-us5 reach 218.8.104.58 on port 443? understood "Can rtr-us5 reach 218.8.104.58?" <- port dropped
Can rtr-us5 reach 218.8.104.58 over udp? understood "Can rtr-us5 reach 218.8.104.58?" <- protocol dropped
All three come back grounded: True with a confident answer. By docs/evaluation.md's own bar -- "judged by what a user meant it was wrong" -- each is answered-wrong: the negation is dropped entirely rather than negating the result, and the port/protocol constraints are silently dropped rather than carried into the traceroute query or refused.
Why the port case matters specifically: a regression, not a gap that was always there
Checked against the 10 August measurement in docs/evaluation.md Part 2: "is port 443 open to the internal server" was refused then. The same shape of question is now answered, with the port silently dropped from what actually gets checked. Something between 10 August and now made this case more confident while becoming less correct.
What this is not
Not a defect in #407's own held-out set or tools/query_quality.py -- that tool correctly measures the 20 entries it has; these three phrasings simply are not among them. See #407's discussion for why they were not folded into that set: they would fail the CI threshold today, and bundling a known-failing case into a scoring-infrastructure PR is not the same change as fixing ai/query.py's question-understanding.
Suggested next step, not applied here
Three separate, narrow fixes, matching the project's one-shape-at-a-time discipline already used for #52/#145/#384's inversions:
- Negation.
ai/query.py's reachability path needs to detect a negated question ("NOT reach", "can't reach") and either invert the expected disposition or refuse rather than answering the un-negated question with high confidence. - Port / protocol constraints. Either thread the constraint into the
HeaderConstraintsthe traceroute query actually uses, or refuse when a constraint is present that the translation does not model -- the same "narrow or refuse, never silently drop" ruleanalysis/policy_conflicts.pyalready applies to an unmodelled query field.
Why this is filed separately rather than fixed under #393/#407
#393's own acceptance criteria is a held-out score, not a fix to ai/query.py's translation. Folding a fix for three real behaviour gaps into the PR that builds the scoring tool would also make that PR's own CI red until the fix landed, which is a bigger and different change than the issue it closes. Filed here so the finding is not lost in a PR comment thread.
Activity
docs/evaluation.md: call US-60's set a regression set, not held-out