Skip to content

fix(framework): audit-prompt hardening — enforce independence + surface output contract (#261)#272

Merged
montfort merged 2 commits into
mainfrom
fix/audit-prompt-hardening-261
Jun 16, 2026
Merged

fix(framework): audit-prompt hardening — enforce independence + surface output contract (#261)#272
montfort merged 2 commits into
mainfrom
fix/audit-prompt-hardening-261

Conversation

@montfort

Copy link
Copy Markdown
Contributor

Closes #261.

Two prompt-design hardening fixes surfaced by a real 5-auditor cross-family cycle. Neither is a CLI mechanics bug — both are template gaps that let auditors drift from the intended discipline. Folded into the shipped template (and the review skill) so they persist across --prepare regenerations.

Problem A — auditor independence was not enforced

Sequential audits share .straymark/audits/<charter>/, so a later auditor can read the earlier report-*.md. In the observed cycle, the last auditor produced a meta-consolidation of the four prior reports ("I independently verified all N findings from the prior audit" + a comparison table) instead of an independent audit. Its "convergence" was copied, not independent — which silently inflates both its rating and the consolidated review's confidence.

  • audit-prompts/audit-prompt.md (EN + ES): the ABSOLUTE RULE now forbids reading/grepping/referencing any other auditor's report-*.md (this Charter or any other). "Your role" and the contradictory "DO NOT ignore prior-audits folders" bullet were reworded — a sibling report may already be on disk; you must not open it.
  • straymark-audit-review (all 4 runtime copies): a contamination guard in the read-reports step detects reports that reference siblings (another report-*.md, a cross-auditor comparison table, "I verified all N findings from the prior audit") and excludes them from the convergence/dedup math and the auditor rating. A prompt rule is weak — verify at review time too.

Problem B — the output contract was buried

The required frontmatter sits in the last ~170 lines of a ~3,900-line resolved prompt; by then the auditor has patterned its mental model on the embedded AILOGs (a different frontmatter), and drifts off-schema.

  • New "Output contract (read this first)" block immediately after the ABSOLUTE RULE: the exact required frontmatter, the four finding categories, and an explicit "DELIBERATELY DIFFERENT from the AILOG/AIDEC frontmatter" disambiguation — with a pointer to the full §Output format at the end. A "Frontmatter note" was also added beside the embedded AILOGs.
  • Note: the finding categories were already defined before the output format in this template, so the reorder the issue contemplated wasn't needed — the burial (not the order) was the real problem.

Verification

EN + ES template edits mirrored. All audit tests green: skill 12, template 9, charter_audit 21, audit 9. No CLI mechanics changed — placeholders intact, schema validation untouched.

🤖 Generated with Claude Code

montfort and others added 2 commits June 16, 2026 12:04
…ce output contract (#261)

A real 5-auditor cross-family cycle exposed two prompt-design gaps that let
auditors drift from the intended discipline (neither is a CLI mechanics bug).

Problem A — auditor independence was not enforced. When audits run
sequentially, each report lands in the shared .straymark/audits/<charter>/
dir, and a later auditor can read the earlier ones. One real auditor produced
a meta-consolidation of the four prior reports instead of an independent
audit — its "convergence" was copied, not independent, which silently inflates
both its rating and the consolidated review's confidence.

- audit-prompt.md (EN + ES): ABSOLUTE RULE now forbids reading/grepping/
  referencing any other report-*.md; "Your role" + "What you must NOT do"
  reworded (a sibling report may already be on disk — do not open it).
- straymark-audit-review (4 runtime copies): contamination guard that detects
  reports referencing siblings and excludes them from convergence/dedup +
  rating. A prompt rule is weak; verify at review time too.

Problem B — the output contract was buried at the end of a ~3,900-line
resolved prompt, and the auditor patterned its frontmatter after the embedded
AILOGs (different schema), producing off-contract reports.

- New "Output contract (read this first)" block right after the ABSOLUTE RULE:
  required frontmatter + the four finding categories + an explicit
  "DELIBERATELY DIFFERENT from the AILOG/AIDEC frontmatter" disambiguation,
  plus a Frontmatter note beside the embedded AILOGs. (Categories were already
  defined before the output format, so no reorder was needed.)

All audit tests green (skill 12, template 9, charter_audit 21, audit 9).

Closes #261.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@montfort montfort merged commit 817efa9 into main Jun 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Audit-prompt hardening: enforce auditor independence + surface the output contract early

1 participant