Repository navigation
test: realign AGENTS.md contract assertions with the StaffCaptain fork vocabulary - #1
Merged
Merged
Conversation
Main has been red since 04763e8 forked AGENTS.md into StaffCaptain form. That commit renamed the fleet-level human authority from captain to Commodore and the orchestrator from Firstmate to StaffCaptain, in prose only. Three inherited behavior tests assert AGENTS.md wording verbatim and still expected the upstream spelling, so the instruction was correct and the tests were the stale artifact. Update the eight drifted assertion sites to the fork's own vocabulary, keeping every contract they pin intact: - fm-captain-translation-contract: the section 9 heading anchor plus the translation contract, nouns line, house-vocabulary line, verbatim-chat ban, and chat-summary rule. - fm-instruction-owners: the separate-review-or-audit authority clause. - fm-no-mistakes-ownership: the orchestrator-never-responds prohibition. The section 9 extractor silently returned nothing once the heading was renamed, which surfaced as a cascade of twenty-six "missing contract" failures instead of the real cause. Assert the heading was found first so a future rename fails for the right reason. Assertions targeting .agents/skills/ and bin/ are deliberately unchanged: the fork did not touch those files, so they still say captain/Firstmate and the tests correctly describe them. File names and schema tokens (data/captain.md, tasks-axi --kind captain, captain-hold) keep their upstream spelling for the same reason. Each changed assertion was verified to still fail when the contract it pins is removed from AGENTS.md, and to fail when AGENTS.md is reverted to the upstream wording. Record the rename and its trap in CONTRIBUTING.md so a later upstream fast-forward does not silently restore the inherited phrasing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Get Freudator86/staffcaptain-firstmate main green again after it went red on 2026-07-24 and sat unnoticed for four days, and do it honestly rather than by making CI pass.
Background: commit 04763e8 forked AGENTS.md into StaffCaptain form, renaming the fleet-level human authority from 'captain' to 'Commodore' and the orchestrator from 'Firstmate' to 'StaffCaptain', in prose only. That rename is deliberate, landed fleet policy (confirmed by the captain 2026-07-28), and is the whole point of the fork. Three inherited behavior tests assert AGENTS.md wording as literal fixed strings and still expected the upstream spelling. The instruction is correct and the tests are the stale artifact, so the fix updates the tests. AGENTS.md is deliberately NOT touched - restoring the captain wording to make a test pass would silently undo the fork decision.
What I changed: 8 drifted assertion sites across tests/fm-captain-translation-contract.test.sh (6), tests/fm-instruction-owners.test.sh (1), and tests/fm-no-mistakes-ownership.test.sh (1). Every one pins a contract that the fork still honours; only the vocabulary moved. Two are not pure renames and were judged individually: 'project outcome' became 'coordination outcome' (StaffCaptain coordinates vessels rather than doing project work, per new AGENTS.md section 1a), and the house-vocabulary sentence additionally gained 'vessel'. I updated the tests to the fork's wording in both cases rather than weakening the assertions.
Deliberate decisions a reviewer reading only the diff would not know:
A previous run of this pipeline reached the review gate and raised three findings. I verified each against the repository independently and accepted two as tightenings, which I still want applied: (1) sibling-extractor-unguarded - validate_contract in tests/fm-no-mistakes-ownership.test.sh has the same silent-empty extractor pattern, anchoring on /^### Validate$/, and needs the same explicit heading-found guard; (2) vacuous-scout-ban-assertion - assert_not_contains "scout, ship" in tests/fm-captain-translation-contract.test.sh can never fire because 'ship' appears zero times in section 9 both today and at 04763e8^, so it must be re-anchored on the actual 'Do not expose internal terms such as ...' ban sentence so that appending scout would genuinely turn it red. A test that can no longer fail is precisely the defect this task exists to remove. The third finding, skills-cross-reference-stale-vocab, is a no-op: it describes the known fork-completeness gap where .agents/skills/** still say captain/Firstmate, which is the halted phase-2 work and is deliberately out of scope. That run failed only because this host had no git author identity configured, which has since been fixed; no code problem was found.
Scope constraints: no fork-wide feature work (phase-2 investment was halted 2026-07-26; the repo is slated for archival with a 30-day rollback window), no history rewrite, no force-push, no change to the vendored/upstream relationship, and no modification of AGENTS.md.
Verification already done locally: each of the 8 changed assertions was mutation-tested to confirm it still fails when the contract it pins is removed from AGENTS.md, and when AGENTS.md is reverted to upstream wording. Full suite 89 pass / 0 fail, bin/fm-lint.sh exit 0, both symlink invariants hold, no personal fleet paths tracked, macOS job counts still exactly 13 and 36.
What Changed
tests/fm-captain-translation-contract.test.sh,tests/fm-instruction-owners.test.sh, andtests/fm-no-mistakes-ownership.test.shto the fork'sCommodore/StaffCaptainwording (includingproject outcome→coordination outcomeand the house-vocabulary sentence gainingvessel), and renamedtest_firstmate_never_responds_for_crew_runtotest_orchestrator_never_responds_for_crew_run.AGENTS.mdis untouched, and the test file names,data/captain.md, and other schema tokens keep their upstream spelling.assert_section_9_foundandassert_validate_contract_foundguards so a renamed heading fails on the real cause instead of silently feeding every assertion an empty string; re-anchored the scout/secondmate/vessel ban check on the actualDo not expose internal terms such assentence via a newinternal_term_banextractor (the previousassert_not_contains "scout, ship"could never fire); and added fork-spelling negative guards for the personal-review prohibition.CONTRIBUTING.mdbullet recording the prose-only rename and naming the three tests that track it, plus adocs/fork-patches.mdnote marking the rename a standingkeepdivergence owned byCONTRIBUTING.md.Full suite passes (89 pass / 0 fail); each changed assertion was mutation-tested to confirm it still fails when the contract it pins is removed or reverted to upstream wording.
Risk Assessment
✅ Low: Test-only change that leaves AGENTS.md and all file names untouched; all three previously accepted findings are implemented as genuine tightenings, and I verified each new guard extracts correctly and passes against the current AGENTS.md while remaining able to fire on the violation it pins.
Testing
The configured full-suite command had already run green as this run's baseline. On top of that I ran the three targeted suites at HEAD (19 ok / 0 not ok), reproduced the original red main by running the pre-fix test files against the fork's AGENTS.md, and then mutation-tested the change in a throwaway symlink sandbox — 15 hostile single-contract edits to AGENTS.md, all 15 caught, including five that revert AGENTS.md to upstream wording. I isolated the review's vacuous-assertion finding and showed the base-commit form stayed green under an edit that contradicts section 9 while the re-anchored form goes red, and confirmed the same for the new section-9 and
### Validateheading guards. Scope checks confirm AGENTS.md is untouched, history is linear, no test was renamed, and the fork commit touched AGENTS.md alone, so the deliberately unchanged captain/Firstmate assertions still describe their targets truthfully. Adjacent AGENTS.md-reading suites still pass; the sandbox was removed and the worktree is clean. No UI surface exists for this change — it is bash behavior tests plus one CONTRIBUTING.md bullet — so no screenshot or rendered artifact is applicable; CLI transcripts are the end-user-equivalent surface here.Evidence: Evidence summary (start here)
Evidence: Mutation matrix — 15/15 hostile AGENTS.md edits caught
=== A. the 6 changed assertions in fm-captain-translation-contract.test.sh === CAUGHT A1 drop positive translation contract sentence -> not ok - section 9 does not own the positive Commodore-facing translation contract CAUGHT A2 revert Commodore's nouns -> captain's nouns -> not ok - section 9 does not require Commodore-owned nouns (missing: 'Use the Commodore's nouns:') CAUGHT A3 drop the rewrite-mapping lead-in -> not ok - section 9 does not own the rewrite mapping list CAUGHT A4 drop allowed house-vocabulary sentence -> not ok - section 9 does not preserve scout as allowed house vocabulary CAUGHT A5 revert verbatim-evidence ban to captain chat -> not ok - section 9 does not reject verbatim internal evidence in Commodore chat CAUGHT A6 revert chat-summary clause to captain-facing -> not ok - section 9 does not keep chat summaries plain English === B. re-anchored scout ban (the assertion the review found vacuous) === CAUGHT B1 append 'scout' to the internal-term ban sentence -> not ok - section 9 added 'scout' to the internal-vocabulary ban sentence CAUGHT B2 append 'vessels' to the internal-term ban sentence -> not ok - section 9 added 'vessel' to the internal-vocabulary ban sentence === C. new extractor guards (silent-empty-input defect) === CAUGHT C1 rename the section 9 heading -> not ok - AGENTS.md has no '## 9. Escalation and Commodore etiquette' heading to extract CAUGHT C2 delete the internal-term ban sentence -> not ok - AGENTS.md section 9 has no 'Do not expose internal terms such as' sentence to extract CAUGHT C3 rename the '### Validate' heading -> not ok - AGENTS.md has no '### Validate' heading to extract === D. fm-instruction-owners + fm-no-mistakes-ownership === CAUGHT D1 revert separate-review clause to captain -> not ok - compressed AGENTS.md lost safety phrase 'A separate review or audit is allowed only when the Commodore explicitly requests...' CAUGHT D2 revert StaffCaptain-never-responds to Firstmate -> not ok - Validate contract permits StaffCaptain to respond directly for a crew-owned run === E. new negative guards for the renamed orchestrator === CAUGHT E1 reintroduce weak review prohibition as StaffCaptain -> not ok - AGENTS.md retained the weaker duplicate review prohibition as 'StaffCaptain' CAUGHT E2 reintroduce personal branch review as 'staff captain' -> not ok - AGENTS.md retained a personal branch-review requirement as 'staff captain' RESULT: every mutation was CAUGHT - no assertion is vacuousEvidence: Vacuous-assertion proof: old check green, new check red on the same hostile edit
Hostile edit applied to AGENTS.md section 9: Do not expose internal terms such as scout, startup machinery, locks, watchers, polling, crewmates, ... This edit BANS 'scout' while section 9 still says scout is allowed house vocabulary -- exactly the contradiction the assertion exists to catch. --- OLD assertion (base 04763e8): assert_not_contains "$contract" "scout, ship" --- VERDICT: GREEN (vacuous - hostile edit sails through) occurrences of the literal 'scout, ship' in section 9: 0 occurrences of the word 'ship' anywhere in section 9: 0 ... at base commit 04763e8 as well: 0 --- NEW assertion (target c18e353): re-anchored on the real ban sentence --- VERDICT: RED (section 9 added 'scout' to the internal-vocabulary ban sentence)Evidence: Red baseline: pre-fix tests vs the fork's AGENTS.md
== fm-captain-translation-contract.test.sh (at 04763e8) == not ok - section 9 does not own the positive captain-facing translation contract (missing: 'Every captain-facing message must translate internal state into the project outcome, consequence, and next decision.') --- output --- <empty: the pre-guard extractor silently returned nothing> -- exit 1 -- == fm-instruction-owners.test.sh (at 04763e8) == not ok - compressed AGENTS.md lost safety phrase 'A separate review or audit is allowed only when the captain explicitly requests...' -- exit 1 -- == fm-no-mistakes-ownership.test.sh (at 04763e8) == -- exit 1 --Evidence: Targeted suites green at HEAD
== tests/fm-captain-translation-contract.test.sh == 7 ok / 0 not ok, exit 0 == tests/fm-instruction-owners.test.sh == 10 ok / 0 not ok, exit 0 == tests/fm-no-mistakes-ownership.test.sh == 2 ok / 0 not ok, exit 0Evidence: Intent scope constraints checked against the diff
-- AGENTS.md must NOT be touched -- OK: AGENTS.md byte-identical between 04763e8 and c18e353 files changed: CONTRIBUTING.md tests/fm-captain-translation-contract.test.sh tests/fm-instruction-owners.test.sh tests/fm-no-mistakes-ownership.test.sh -- no history rewrite / force-push -- OK: 04763e8 is a linear ancestor of c18e353 (2 commits added) -- test file name deliberately left at upstream spelling -- all M, no R: no test file was renamedEvidence: Assertion-count reconciliation (26->25 dip explained; coverage rose)
Counting every site that can fail the test (assert_* plus explicit 'fail'): fm-captain-translation-contract.test.sh base=28 target=30 fm-instruction-owners.test.sh base=56 target=58 fm-no-mistakes-ownership.test.sh base=4 target=5 One literal substring check that could never match was replaced by a 3-term, case-insensitive, plural-tolerant word-boundary scan of the real ban sentence. Coverage went up; the raw assert_* count went down by one because the loop calls fail directly.Evidence: Fork blast radius + sweep of other AGENTS.md-reading tests
Files changed by the fork commit 04763e8: AGENTS.md => .agents/skills/** and bin/ were NOT touched by the fork, so the ~15 remaining captain/Firstmate assertions that target them still describe those files truthfully.Evidence: Adjacent AGENTS.md-reading suites still pass
fm-bootstrap exit=0 (21 ok, 0 not ok) fm-stow-contract exit=0 (2 ok, 0 not ok) fm-supervision-instructions exit=0 (10 ok, 0 not ok)Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
tests/fm-no-mistakes-ownership.test.sh:8- The intent states as required: "(1) sibling-extractor-unguarded - validate_contract in tests/fm-no-mistakes-ownership.test.sh has the same silent-empty extractor pattern, anchoring on /^### Validate$/, and needs the same explicit heading-found guard" — "which I still want applied". The diff does not add it. validate_contract() (lines 8-14) still returns an empty string with exit 0 if the### Validateheading is ever renamed or removed, so both assert_contains calls would fail with a misleading "Validate contract does not assign..." message instead of the real cause. This is the exact defect assert_section_9_found was added to fix in the sibling file, and the change was made in this same file (test renamed at line 32) without adding the guard.tests/fm-captain-translation-contract.test.sh:59- The intent states as required: "(2) vacuous-scout-ban-assertion - assert_not_contains "scout, ship" ... must be re-anchored on the actual 'Do not expose internal terms such as ...' ban sentence so that appending scout would genuinely turn it red." The diff leaves line 59 untouched. I confirmed the vacuity:awkextraction of section 9 from both the current AGENTS.md and 04763e8^:AGENTS.md contains zero occurrences of the substring "ship", soassert_not_contains "$contract" "scout, ship"can never fire. Adding "scout" to the real ban sentence at AGENTS.md section 9 line 6 ("Do not expose internal terms such as startup machinery, locks, watchers, ...") would produce "scout, locks" or similar and the test would still pass green. An assertion that cannot fail is precisely what this branch exists to remove.tests/fm-instruction-owners.test.sh:234- Same defect class as the accepted vacuous-scout-ban finding, in a file this change already touches.assert_no_grep 'Firstmate does not personally review code or deliverables' "$AGENTS"(line 234) andassert_no_grep 'firstmate reviews your branch' "$AGENTS"(line 236) are regression guards against re-introducing a weaker personal-review requirement into AGENTS.md. After the fork, AGENTS.md contains no "Firstmate" at all, so a realistic regression would be written as "StaffCaptain does not personally review code or deliverables" and neither guard would fire. The positive assertion three lines above (line 235 in the diff) was updated to the fork vocabulary; these two negative guards were not, so they now silently protect nothing. Either re-anchor them on the fork's spelling or make them vocabulary-agnostic.🔧 Fix: tighten three vacuous AGENTS.md contract assertions in tests
1 info still open:
tests/fm-instruction-owners.test.sh:239- The new guard enumerates four fixed-string casings (StaffCaptain, 'Staff Captain', staffcaptain, 'staff captain') and relies on assert_no_grep, which is grep -F and therefore case-sensitive. A regression written as 'Staffcaptain does not personally review code or deliverables' or 'STAFF CAPTAIN reviews your branch' would still slip through. A single case-insensitive check (grep -Eqi 'staff ?captain (does not personally review code or deliverables|reviews your branch)') would close the whole class and replace eight grep invocations with one. Noting it as a tradeoff rather than a defect: the explicit variant list is readable and covers every realistic spelling, and collapsing it would re-word an assertion the captain just directed be added, so leaving it as-is is defensible.✅ **Test** - passed
✅ No issues found.
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"bash tests/fm-captain-translation-contract.test.sh— 7 ok / 0 not okbash tests/fm-instruction-owners.test.sh— 10 ok / 0 not okbash tests/fm-no-mistakes-ownership.test.sh— 2 ok / 0 not okRed-baseline repro:git show 04763e8:tests/<each>.test.shrun against the fork's AGENTS.md in a symlink sandbox — all three exit 1Mutation matrix of 15 single-contract perturbations of AGENTS.md (A1–A6, D1–D2 = the 8 changed assertion sites; B1–B2 = re-anchored scout/vessel ban; C1–C3 = new heading guards; E1–E2 = new negative guards) — 15/15 caught with the expected failure messageVacuous-assertion before/after: base-commitassert_not_contains "$contract" "scout, ship"vs the re-anchored ban scan, both applied to an AGENTS.md that appendsscoutto the internal-term ban sentenceScope verification:git diff --quiet 04763e8 c18e353 -- AGENTS.md,git merge-base --is-ancestor,git diff --name-status --find-renames -- tests/,git show --stat --name-only 04763e8Assertion-count reconciliation across both commits (assert_* plus explicitfailsites)Sweep for other AGENTS.md-reading suites the rename could have made stale, thenbash tests/fm-bootstrap.test.sh,bash tests/fm-stow-contract.test.sh,bash tests/fm-supervision-instructions.test.sh— all exit 0git status --porcelainafter teardown — cleandocs/fork-patches.md:3- docs/fork-patches.md states "Each fork-only, non-merge commit has one row", but its last row is PR #4 (8f3190b) withad9f3a7as the last reviewed upstream tip, while roughly a dozen later fork-only commits (a8ce717#7 throughb9cb8bf#18, plus the three StaffCaptain-fork commits) have no rows. This drift predates this change and reconciling it needs a real upstream-tip comparison viabin/fm-fork-sync-check.sh, which is out of scope here; my edit only makes the vocabulary divergence discoverable from the owner doc. Suggested follow-up: one task that re-runs the fork-sync comparison and refreshes the registry against the current real-upstream tip..agents/skills/afk/SKILL.md:54- AGENTS.md now requires addressing the user as "Commodore", while.agents/skills/**(afk, bootstrap-diagnostics, decision-hold-lifecycle, stuck-crewmate-recovery, updatefirstmate, harness-adapters) still instruct the agent to say "Captain" and reference "firstmate". This is the halted phase-2 fork-completeness gap the intent declares out of scope, and the new CONTRIBUTING bullet's "in AGENTS.md prose only" wording correctly bounds the rename, so I left it alone. Noting it only so the contradiction stays visible if phase 2 ever resumes before archival.✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.