Repository navigation
Reject a Modelfile worked example copied into a real explanation (#455) - #458
patelankeet2 wants to merge 1 commit into
Conversation
Found by Arsh while measuring PR #401: the model occasionally abandons the real finding and returns ai/Modelfile's own Example 1 or Example 2 almost word for word, unrelated to the finding it was supposedly explaining -- measured 2/120 across two independent live runs on two real findings. _is_unacceptable() did not catch this. Covers Examples 1 and 2 only, not Example 3. Both describe one fixed, never-otherwise-correct fictional scenario, so any match is unambiguously a copy. Example 3's wording is generic all-clear language a genuine status=none finding is supposed to produce, so treating it as forbidden would reject correct answers, not just copied ones. Checked for every status, not just error, since the fictional content is equally wrong glued onto any real finding. Dropped one candidate phrase while writing the tests: 'allows all traffic through with no restriction' is in Example 1's own text, but two existing tests in this file independently use that exact wording as an ordinary stub for an unrelated, genuinely wide-open ACL finding -- it is a plausible real description, not specific to the fictional scenario. The other two Example-1 phrases are specific enough to still catch the measured copy without it. Verified: 98 tests in the safety-net file (9 new), full suite 2268 passed / 13 skipped / 0 failed, mutation tested (removing the check's one call site makes 5 of the 6 phrase tests fail for the right reason; the 6th is independently caught by the existing error-only result-claim check), and 30 fresh live generations each on AC-002 and a redundant dead-rule finding through the real model: 0/30 copies slipped through, 0/30 fallbacks for any other reason. Not done: #410 (a broader, statistical drift-detection check) remains open and unaffected -- this is the small, exact check, not a substitute.
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Approving 9b09766, with one correction to make before merge: a comment that isn't true.
What holds:
- Merged onto current
main(2833a1e, with #442 and #446): 2318 passed, 13 skipped, ruff clean on CI's selection. - Removing the new check fails 6 in
test_explain_safety_net.py. - It catches the exact copy I saw on #401 last week, word for word: "The internal server can be reached over unencrypted web traffic that should have been blocked. … which is what let this through."
- Dropping "allows all traffic through with no restriction" was the right call, for the reason you give.
The correction. The comment says Examples 1 and 2 each describe a "fixed, never-otherwise-correct scenario". Example 1 is a real finding. Its summary and detail ("Unencrypted web traffic reaches the internal server", "Expected DENY but got PERMIT, decided by: permit ip any any") are word for word AC-001 on rtr-us5-insecure, the router in every demo. So on that finding, "…unencrypted web traffic that should have been blocked" is a correct sentence, and the guard would reject it as a copy.
I measured whether that happens: 20 live generations of the real AC-001 through _is_unacceptable. 0 of 20 rejected on this branch. The model paraphrases rather than reusing the exact phrases (main rejected 1 of 20, for an unrelated reason). So it isn't biting today, and when it does, the cost is the deterministic fallback text, which is the safe direction. That's why this is an approval. But the comment states the opposite of the truth, and the next reader will trust it. Either:
- reword it ("Example 1 is modelled on AC-001 itself, so a correct explanation of AC-001 can match; that costs a fallback, which is safe, and measured 0/20"), or
- skip the Example 1 phrases when the finding is Example 1's (same summary and detail).
Either is fine. I'd take the first: it's honest, and it keeps the guard simple.
ARSH871-bot
left a comment
There was a problem hiding this comment.
Reviewed 9b09766. The check is right, but this branch now conflicts with main in ai/explain.py, so I'm approving the content and asking for the merge before it lands.
- On its own head: CI-style 2227 passed, 54 skipped, ruff clean. The new
_MODELFILE_EXAMPLE_PATTERNcheck is unconditional, which is right: a copied example is wrong for every status. - Dropping "allows all traffic through with no restriction" was the right call; that phrase is a fair description of a real wide-open rule.
- Conflict:
ai/explain.py, from #456 landing since (8c22f2e). Please mergemainin and re-request. If the merge is mechanical I'll re-approve the same day; I'll re-run your phrase tests and the holdout set on the merged tree before I do.
Closes #455.
What's wrong
Found by @ARSH871-bot while measuring PR #401: the model occasionally abandons the real finding and returns
ai/Modelfile's own Example 1 or Example 2 almost word for word, unrelated to the finding it was supposedly explaining._is_unacceptable()did not catch this, so it reached the dashboard labelled as genuine model output. Measured: 2/120 across two independent live runs on two real findings (a dead DNS permit, a dead SSH deny) -- a steady low-rate failure, not a one-off.Same family as #456 (an all-clear passing the output check) and #457 (the specific phrase that motivated #456 still getting through): the output check is a safety net with known holes, and this is one more named hole.
What changed
ai/explain.py: one new_MODELFILE_EXAMPLE_PATTERNand one new unconditional check inside_is_unacceptable()(not gated onis_error, since the fictional content is equally wrong for any status).Scope: Examples 1 and 2 only, not Example 3. Examples 1 and 2 each describe one fixed, never-otherwise-correct scenario (an unencrypted-web-traffic server; a crashed HQ-to-branch check), so any match is unambiguously a copy. Example 3's wording is generic "everything checked out" language that a genuine
status="none"finding is supposed to produce -- treating it as forbidden would reject correct answers, not just copied ones.One candidate phrase dropped while testing. "allows all traffic through with no restriction" is in Example 1's own text, but it turned out to be a plausible, legitimate description of any genuinely wide-open ACL rule, not specific to the fictional scenario -- two existing tests in this file independently use that exact wording as an ordinary stub for an unrelated finding, and broke when I first included it. The other two Example-1 phrases are specific enough to still catch the measured copy without it.
Not done: #410 (a broader, statistical drift-detection check) remains open and unaffected. This is the small, exact, no-model check #455 itself asked for; it is not a substitute for #410.
Testing
tests/test_explain_safety_net.py: 98 passed (9 new) -- both measured real sentences pinned word for word, each of the remaining phrases individually, a faithful control for bothfoundanderrorstatus (theerrorcontrol needed its own wording since a legitimate error explanation can never share result-claiming words with afoundone), and one end-to-end test throughexplain_with_source().is_error=True, which is correct, not a gap). Restored, re-confirmed clean.main(d424e05).@ARSH871-bot, requesting your review since you found and filed this.