fix(framework): audit-prompt hardening — enforce independence + surface output contract (#261)#272
Merged
Merged
Conversation
…ce output contract (#261) A real 5-auditor cross-family cycle exposed two prompt-design gaps that let auditors drift from the intended discipline (neither is a CLI mechanics bug). Problem A — auditor independence was not enforced. When audits run sequentially, each report lands in the shared .straymark/audits/<charter>/ dir, and a later auditor can read the earlier ones. One real auditor produced a meta-consolidation of the four prior reports instead of an independent audit — its "convergence" was copied, not independent, which silently inflates both its rating and the consolidated review's confidence. - audit-prompt.md (EN + ES): ABSOLUTE RULE now forbids reading/grepping/ referencing any other report-*.md; "Your role" + "What you must NOT do" reworded (a sibling report may already be on disk — do not open it). - straymark-audit-review (4 runtime copies): contamination guard that detects reports referencing siblings and excludes them from convergence/dedup + rating. A prompt rule is weak; verify at review time too. Problem B — the output contract was buried at the end of a ~3,900-line resolved prompt, and the auditor patterned its frontmatter after the embedded AILOGs (different schema), producing off-contract reports. - New "Output contract (read this first)" block right after the ABSOLUTE RULE: required frontmatter + the four finding categories + an explicit "DELIBERATELY DIFFERENT from the AILOG/AIDEC frontmatter" disambiguation, plus a Frontmatter note beside the embedded AILOGs. (Categories were already defined before the output format, so no reorder was needed.) All audit tests green (skill 12, template 9, charter_audit 21, audit 9). Closes #261. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ening-261 # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #261.
Two prompt-design hardening fixes surfaced by a real 5-auditor cross-family cycle. Neither is a CLI mechanics bug — both are template gaps that let auditors drift from the intended discipline. Folded into the shipped template (and the review skill) so they persist across
--prepareregenerations.Problem A — auditor independence was not enforced
Sequential audits share
.straymark/audits/<charter>/, so a later auditor can read the earlierreport-*.md. In the observed cycle, the last auditor produced a meta-consolidation of the four prior reports ("I independently verified all N findings from the prior audit" + a comparison table) instead of an independent audit. Its "convergence" was copied, not independent — which silently inflates both its rating and the consolidated review's confidence.audit-prompts/audit-prompt.md(EN + ES): the ABSOLUTE RULE now forbids reading/grepping/referencing any other auditor'sreport-*.md(this Charter or any other). "Your role" and the contradictory "DO NOT ignore prior-audits folders" bullet were reworded — a sibling report may already be on disk; you must not open it.straymark-audit-review(all 4 runtime copies): a contamination guard in the read-reports step detects reports that reference siblings (anotherreport-*.md, a cross-auditor comparison table, "I verified all N findings from the prior audit") and excludes them from the convergence/dedup math and the auditor rating. A prompt rule is weak — verify at review time too.Problem B — the output contract was buried
The required frontmatter sits in the last ~170 lines of a ~3,900-line resolved prompt; by then the auditor has patterned its mental model on the embedded AILOGs (a different frontmatter), and drifts off-schema.
Verification
EN + ES template edits mirrored. All audit tests green: skill 12, template 9, charter_audit 21, audit 9. No CLI mechanics changed — placeholders intact, schema validation untouched.
🤖 Generated with Claude Code