Skip to content

Reject a Modelfile worked example copied into a real explanation (#455) - #458

Open
patelankeet2 wants to merge 1 commit into
mainfrom
fix/455-reject-modelfile-example-copy
Open

patelankeet2 wants to merge 1 commit into
mainfrom
fix/455-reject-modelfile-example-copy

Conversation

@patelankeet2

Copy link
Copy Markdown
Collaborator

Closes #455.

What's wrong

Found by @ARSH871-bot while measuring PR #401: the model occasionally abandons the real finding and returns ai/Modelfile's own Example 1 or Example 2 almost word for word, unrelated to the finding it was supposedly explaining. _is_unacceptable() did not catch this, so it reached the dashboard labelled as genuine model output. Measured: 2/120 across two independent live runs on two real findings (a dead DNS permit, a dead SSH deny) -- a steady low-rate failure, not a one-off.

Same family as #456 (an all-clear passing the output check) and #457 (the specific phrase that motivated #456 still getting through): the output check is a safety net with known holes, and this is one more named hole.

What changed

ai/explain.py: one new _MODELFILE_EXAMPLE_PATTERN and one new unconditional check inside _is_unacceptable() (not gated on is_error, since the fictional content is equally wrong for any status).

Scope: Examples 1 and 2 only, not Example 3. Examples 1 and 2 each describe one fixed, never-otherwise-correct scenario (an unencrypted-web-traffic server; a crashed HQ-to-branch check), so any match is unambiguously a copy. Example 3's wording is generic "everything checked out" language that a genuine status="none" finding is supposed to produce -- treating it as forbidden would reject correct answers, not just copied ones.

One candidate phrase dropped while testing. "allows all traffic through with no restriction" is in Example 1's own text, but it turned out to be a plausible, legitimate description of any genuinely wide-open ACL rule, not specific to the fictional scenario -- two existing tests in this file independently use that exact wording as an ordinary stub for an unrelated finding, and broke when I first included it. The other two Example-1 phrases are specific enough to still catch the measured copy without it.

Not done: #410 (a broader, statistical drift-detection check) remains open and unaffected. This is the small, exact, no-model check #455 itself asked for; it is not a substitute for #410.

Testing

  • tests/test_explain_safety_net.py: 98 passed (9 new) -- both measured real sentences pinned word for word, each of the remaining phrases individually, a faithful control for both found and error status (the error control needed its own wording since a legitimate error explanation can never share result-claiming words with a found one), and one end-to-end test through explain_with_source().
  • Mutation tested: commented out the new check's one call site -- 5 of 6 phrase tests failed for the right reason (the 6th, the Example-2 copy, is independently caught by the existing error-only result-claim check for is_error=True, which is correct, not a gap). Restored, re-confirmed clean.
  • Full suite: 2268 passed, 13 skipped, 0 failed, merged onto current main (d424e05).
  • Live, against the real model (Ollama + Batfish): 30 fresh generations each on AC-002 and a redundant dead-rule finding -- 0/30 copies slipped through, 0/30 fallbacks for any other reason. Small sample against a ~2/120 measured base rate, so this does not prove the guard will always catch one; the unit + mutation tests are what prove the guard fires when a copy does occur.
  • Byte-level em-dash check: 0 in both changed files.

@ARSH871-bot, requesting your review since you found and filed this.

Found by Arsh while measuring PR #401: the model occasionally abandons
the real finding and returns ai/Modelfile's own Example 1 or Example 2
almost word for word, unrelated to the finding it was supposedly
explaining -- measured 2/120 across two independent live runs on two real
findings. _is_unacceptable() did not catch this.

Covers Examples 1 and 2 only, not Example 3. Both describe one fixed,
never-otherwise-correct fictional scenario, so any match is unambiguously
a copy. Example 3's wording is generic all-clear language a genuine
status=none finding is supposed to produce, so treating it as forbidden
would reject correct answers, not just copied ones.

Checked for every status, not just error, since the fictional content is
equally wrong glued onto any real finding.

Dropped one candidate phrase while writing the tests: 'allows all traffic
through with no restriction' is in Example 1's own text, but two existing
tests in this file independently use that exact wording as an ordinary
stub for an unrelated, genuinely wide-open ACL finding -- it is a
plausible real description, not specific to the fictional scenario. The
other two Example-1 phrases are specific enough to still catch the
measured copy without it.

Verified: 98 tests in the safety-net file (9 new), full suite 2268 passed
/ 13 skipped / 0 failed, mutation tested (removing the check's one call
site makes 5 of the 6 phrase tests fail for the right reason; the 6th is
independently caught by the existing error-only result-claim check), and
30 fresh live generations each on AC-002 and a redundant dead-rule finding
through the real model: 0/30 copies slipped through, 0/30 fallbacks for
any other reason.

Not done: #410 (a broader, statistical drift-detection check) remains
open and unaffected -- this is the small, exact check, not a substitute.
@patelankeet2 patelankeet2 added bug Something isn't working layer: ai Layer 2 — local LLM explanation and Q&A safety Touches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care labels Nov 8, 2026
@patelankeet2 patelankeet2 self-assigned this Nov 8, 2026

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 9b09766, with one correction to make before merge: a comment that isn't true.

What holds:

  • Merged onto current main (2833a1e, with #442 and #446): 2318 passed, 13 skipped, ruff clean on CI's selection.
  • Removing the new check fails 6 in test_explain_safety_net.py.
  • It catches the exact copy I saw on #401 last week, word for word: "The internal server can be reached over unencrypted web traffic that should have been blocked. … which is what let this through."
  • Dropping "allows all traffic through with no restriction" was the right call, for the reason you give.

The correction. The comment says Examples 1 and 2 each describe a "fixed, never-otherwise-correct scenario". Example 1 is a real finding. Its summary and detail ("Unencrypted web traffic reaches the internal server", "Expected DENY but got PERMIT, decided by: permit ip any any") are word for word AC-001 on rtr-us5-insecure, the router in every demo. So on that finding, "…unencrypted web traffic that should have been blocked" is a correct sentence, and the guard would reject it as a copy.

I measured whether that happens: 20 live generations of the real AC-001 through _is_unacceptable. 0 of 20 rejected on this branch. The model paraphrases rather than reusing the exact phrases (main rejected 1 of 20, for an unrelated reason). So it isn't biting today, and when it does, the cost is the deterministic fallback text, which is the safe direction. That's why this is an approval. But the comment states the opposite of the truth, and the next reader will trust it. Either:

  • reword it ("Example 1 is modelled on AC-001 itself, so a correct explanation of AC-001 can match; that costs a fallback, which is safe, and measured 0/20"), or
  • skip the Example 1 phrases when the finding is Example 1's (same summary and detail).

Either is fine. I'd take the first: it's honest, and it keeps the guard simple.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 9b09766. The check is right, but this branch now conflicts with main in ai/explain.py, so I'm approving the content and asking for the merge before it lands.

  • On its own head: CI-style 2227 passed, 54 skipped, ruff clean. The new _MODELFILE_EXAMPLE_PATTERN check is unconditional, which is right: a copied example is wrong for every status.
  • Dropping "allows all traffic through with no restriction" was the right call; that phrase is a fair description of a real wide-open rule.
  • Conflict: ai/explain.py, from #456 landing since (8c22f2e). Please merge main in and re-request. If the merge is mechanical I'll re-approve the same day; I'll re-run your phrase tests and the holdout set on the merged tree before I do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working layer: ai Layer 2 — local LLM explanation and Q&A safety Touches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The model can copy Modelfile Example 1 word for word, and the check passes it

3 participants