Skip to content

The model can copy Modelfile Example 1 word for word, and the check passes it #455

Open

Description

@ARSH871-bot

What's wrong

The explanation for a finding can come back as ai/Modelfile's Example 1, unrelated to the finding, and _is_unacceptable() passes it, so it is shown labelled as AI.

Measured

finding:  a dead DNS permit (AC-002), and a dead SSH deny
text:     "The internal server can be reached over unencrypted web traffic that should have
           been blocked. The device's rule set currently allows all traffic through with no
           restriction, which is what let this through."
source:   model

Across AC-002 and a redundant DENY, 30 live runs each: 2/60 on #401 and 0/60 on main. Shubham's run on #401 had 0/60 and 2/60, so about 2 in 120 on either. The 9/60 rate under #401's earlier wording is gone, but this remains.

Suggested fix

Reject any text containing a sentence from the Modelfile's examples, so it falls back to the fixed text. It needs no model. #410 is a broader, statistical check on drift, and this is a much smaller exact one.

Activity

  1. added
    bugSomething isn't working
    layer: aiLayer 2 — local LLM explanation and Q&A
    safetyTouches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care
    on Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinglayer: aiLayer 2 — local LLM explanation and Q&AsafetyTouches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions