Skip to content

_ALL_CLEAR_PATTERN still misses the exact phrase that started this (#456 follow-up) #457

Open

Description

@patelankeet2

Follow-up from reviewing #456 (#456 (review)).

#456 added _ALL_CLEAR_PATTERN to ai/explain.py to reject all-clear wording for found and error findings. It works for the phrases it lists, but the one phrase that actually motivated this whole line of research still gets through, because an inserted word breaks the literal match.

Measured live, on 8c22f2e:

'There are no problems with this rule.'                    MATCH
'There are no security problems with this configuration.'   NO MATCH
'The scan found no real issues of concern.'                  NO MATCH

'no security problems' is the exact sentence from the original prompt-injection research (Samika's test, reported this week): the model said 'no security problems' about a confirmed finding, and the old error-only word list let it through because 'problems' alone was not on that list. #456 closes that for the literal phrase 'no problems', but a model that inserts one adjective still escapes the same way.

Not asking for open-ended paraphrase detection here, that's a much bigger problem than one PR should solve. Just the one-word-inserted case, since it's the one already measured happening for real:

_ALL_CLEAR_PATTERN = re.compile(
    r"\b(no (\w+\s+)?problems|no (\w+\s+)?issues|nothing needs fixing|looks clean|"
    r"check passed|is fine|is safe)\b",
    re.IGNORECASE,
)

Needs live re-testing against the same false-positive set #456 already built (tests/fixtures/explanation_quality/holdout.json plus the seven-fixture fresh-model run) before landing, since a wider pattern risks rejecting more honest explanations, not just catching more bad ones.

Assignee: patelankeet2. ai/ is mine.

Activity

  1. added
    bugSomething isn't working
    layer: aiLayer 2 — local LLM explanation and Q&A
    safetyTouches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care
    on Nov 8, 2026
  2. self-assigned this
    on Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinglayer: aiLayer 2 — local LLM explanation and Q&AsafetyTouches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions