Skip to content

CI lint: enforce the plan-template Verify deny-list (falsifiable-verify rules are advisory without it) #52

Description

@lantisprime

Summary

PR #51 added a Falsifiable Verify rule + deny-list to agents/PLAN_TEMPLATE.md so lower-capability executors can't ship hollow guards (the P6-0b run-p6-smoke.sh / --append-system-prompt /dev/null failure). But the template only describes the deny-list — nothing enforces it. This repo has no active hook/CI that checks plan tables, so the rules are advisory until a lint exists.

Per the repo principle "enforce rules in CI, not just docs," add a lint.

Root cause it closes

A low-capability executor produces the weakest artifact its verify command accepts. The deny-list only bites if a machine rejects a non-falsifiable Verify cell at authoring/PR time.

Proposed lint

A check (script + CI step) over Appendix B mechanical-spec tables in agents/docs/*PLAN*.md (and any plan using the template's step tables) that fails when a Verify cell:

  • tests only an exit code with a tolerant comparison — test $? -ne 1, || true, ; true, && echo ok;
  • contains no capture-and-compare (no named observed value: stdout/exit/file/return vs an expected concrete value) — e.g. bare "exits 0" / "script runs";
  • (heuristic) greps for a string literal that also appears in an echo/comment the same step writes (self-fulfilling);
  • belongs to a MUST requirement or Safety/Security mitigation row and is a command -v …-skippable smoke without an UNGUARDED-IN-CI tag.

Plus: every UNGUARDED-IN-CI row MUST name a covering manual step (the lint asserts the Notes column is non-empty).

Acceptance

  • Lint script under scripts/ (or agents/), runnable locally, exits non-zero on a violating Verify cell with a clear message + file:line.
  • Wired into .github/workflows/ci.yml.
  • A regression fixture: a deliberately-hollow Verify cell (test $? -ne 1) that the lint catches, and a falsifiable one it passes. (Per repo rule: regression test in the same commit as the lint.)
  • Run against existing P6 plan tables; fix or UNGUARDED-IN-CI-tag any current violators (0b.17's row is the known one).

Context

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions