Repository navigation
feat(planning): redesign the interview flow, layout and numbering - #17965
Draft
kyle-sexton wants to merge 37 commits into
Draft
kyle-sexton wants to merge 37 commits into
kyle-sexton wants to merge 37 commits into
Conversation
The interview no longer dispatches a sub-agent before each round to check the draft against earlier answers. The end sweep keeps its Contradictions category and reconcile rule, and the answer-time conflict rule still catches a reply that conflicts with an earlier answer. Brief AC10: "the per-round contradiction pass is gone". - SKILL.md: drop the Step 2 paragraph and the pacing clause that named the pass; the sweep paragraph's pointer names the sweep-round label, whose rule lives in context/assumption-sweep.md. - context/assumption-sweep.md: drop "Pre-round contradiction pass". - context/product.md: the pacing note names only the sweep. - evals.json: remove case 36 pre-round-contradiction-pass, which graded the removed pass. Its replacement evidence is the new eval case interview-sweep-reports-contradiction (added later on this branch). - docs/contracts/verification-level: the interview's exemption names the sweep and the quote check only. Re-pins (interview-defenses.test.sh), old -> new, and why each defense still holds: - Stance: 53ff69a1cc4af1af5f7df38a03b272977431a6e8374c52df8f9ca044e70e7ef6 -> e059b8a9bdadf3a44593745098d9365d46b80fe2e2e9bdb3d449ab124d0481b0. Only the pacing clause naming the pass changed; the no-silent-resolve rule is byte-identical. - Step 2: 91e71ea48b46fdefdb4fa77f32cd6d48b47245f04c0c1bbdcf4884427211fff4 -> f332b13eda79cde79825d912827e70b18971870c60accde42f0ff5c79ca3bcea. The pass's paragraph is gone and the sweep paragraph points at its spoke; the constraint ledger, candidates, currency, tripwire and sweep rules are unchanged, and no row is resolved for the user. - context/assumption-sweep.md: 2054938d89851cd80602ba2d80aed3bd49722420cd836155c9ccdfe8e0f47e2a -> 4d1a6e8c8789645fb58bbae26822fc34d2fdcefda674e0d7210247dcccc8f0f3. Only the per-round section is deleted; scope, item shape and every disposition, USER-RESERVED included, are unchanged. - Case roster: 9f9747c65eb92352e3b940f53808bac632a4f2691a95dfd08ac74a152818d713 -> 7bd78ae7a7bc7a32988b8a7b526a189373ac3fa961f79f7aa56c826a280d588b. Only case 36 left the roster; cases 15 and 16 and every digested case are unchanged, and no case added contradicts them. The three test comments that described the pass are deleted; the reasons above replace per-pin comments, which the comment lint refuses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
Question numbers are now handed out and kept, never reused or renumbered (issue #7541, owner ruling "Keep ids (do not refuse)"). - round.sh next-id --ledger <ledger> [--dir <data dir>] [--count <n>] prints the next n ids: the highest number the ledger's register rows and `- gap |` lines hold (and, with --dir, every number the merged export writes and every pending proposal's Q<N>) plus 1. It never fills a gap. It refuses a file without exactly one open-question register heading (check_ledger, as sync-ledger does) and a --count below 1. - exporters.numbering(ids, held): a row the ledger already holds keeps its ledger number and its [<id>] lead whatever its page id; a Q<N> id new to the ledger keeps N unless a ledger row already has that number (then it takes the next number and a [Q<N>] lead); any other id takes the next number after the highest so far (ledger numbers and the Q<N> ids kept in this export), in natural id order. A row leads with [<id>] only when its number is not its own id. The second parameter is now the ledger's {id: number} map (ledger_numbers), since a bool cannot carry the numbers. ledger_gapped had no caller left and is removed with its test. - export-ledger --ledger X keeps each hand-written `- gap |` line of X, placed before the first row past its range, as sync-ledger keeps it. - check-open-questions.sh accepts a gap only where a hand-written `- gap | Q<a>-Q<b> | <why>` line in the register section declares it; an undeclared gap (or the undeclared part of one) still exits 2 and its message names the gap line to write; a malformed or fenced gap line declares nothing. - templates/checklist.md: the register header names the gap line. - surface/README.md: a next-id row in the command table. Intended reversals (the plan's numbering contract replaces the renumbering rule): 1. test_non_contiguous_ids_are_renumbered_with_the_original_id is rewritten as test_non_contiguous_ids_are_never_renumbered. 2. test_numbering_keeps_gapped_q_ids_only_for_a_gapped_ledger is rewritten as test_numbering_keeps_every_q_id_and_numbers_others_after_the_highest. The Brief test test_without_a_ledger_a_gap_keeps_the_register_number_and_ adds_the_id is updated to test_without_a_ledger_a_gap_keeps_every_id_with_ no_lead: it now asserts the gate refuses the undeclared gap, then passes with brief=ok once a gap line declares it. New tests: test_numbering_keeps_ledger_numbers_and_numbers_only_new_ids, test_a_renumbered_ledger_exports_twice_with_the_numbers_it_holds, test_export_ledger_keeps_each_hand_written_gap_line_in_place, test_a_q_id_new_to_the_ledger_keeps_its_number, test_a_new_q_id_whose_number_the_ledger_holds_takes_the_next_number, TestNextId (6 tests), and the gate's declared, partly declared, fenced and malformed gap-line cases plus the gap-line message check. Known limit: an export with no ledger text (export-ledger without --ledger, the report, a recap with a data dir and no ledger) numbers an imported renumbered ledger's rows from their ids, not the numbers the ledger held (rows [Q7] at Q1 and [J1] at Q2 export as Q7 and Q8). A later phase records each row's number in meta.seededFrom at import. Pre-flight: readers of register ids and of numbering() - surface/exporters.py: ROW regex in scan_register/parse_register (row number, cells); LEAD regex in ledger_register, import_ledger, ledger_row and recap_answer ([<id>] lead -> question id); REGISTER_LEAD in locate_row/split_lead (supersede and resolve find a row by Q<N> or its lead); numbering() called by register(), merged_register() and export_brief(); brief_label() shows "Q<N> [<id>]" when number != id. - surface/round.py: export-ledger, ledger-check, sync-ledger, add-round --sync-ledger and recap go through merged_register(); supersede and resolve take Q<N> or the [<id>] lead (locate_row); record_ledger_sync reads Q<N> from merge notes. - scripts/check-open-questions.sh: row_candidate_re and the id regex in the register loop (track_id: duplicate, out of order), the gap pass after the loop, deferred_ids for --brief, and the sweep's "register through Q<N>" and Q<N> disposition checks. - shared/session.py: QID regex Q[1-9][0-9]{0,5} validates shared-session question ids; it reads no ledger. - skills/audit-answers/SKILL.md: reads register rows by status (superseded-by-plan, hedged:) in prose; no id parsing. - skills/wayfind/SKILL.md: chart step 1 seeds from interview-checklist.md by row status in prose; no id parsing. - plugins/work-items/skills/decompose/SKILL.md: reads answered Q<N> rows of interview-checklist.md in prose; no id parsing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
Code-design review findings on 146fa27, all in Phase 3's files. - numbering(ids, held, gaps) treats every number inside a declared `- gap | Q<a>-Q<b> |` range as taken: a new non-Q id never lands in a gap, and a page Q<N> id inside one is numbered after the highest number with its [<id>] lead. page_numbering() builds the id list once from the page, its pending proposals, the ledger-only ids and the ledger's numbers and gaps; register() takes that numbering from its caller, so merged_register and export_brief no longer compute it twice, and export_report still gets the page's own. - highest_id (next-id) also counts each [Q<N>] lead in the register and, with --dir, each page question's own Q<N> id. - One gap-line grammar: Python's GAP_LINE/parse_gap now match the gate (no leading zero at either end, a <= b, a non-empty reason, both ends captured). - Gate: a declared range that covers a register row, or whose high end is past the highest row, exits 2 naming the gap line; the gap candidate is `gap` followed by optional spaces and `|`, so a prose bullet such as `- gap, see below` is not read as a gap line. - register_lines() places gap lines by range for both writers: export_ledger and sync_ledger now agree (sync_ledger used to leave a gap line where it stood while the rows moved above it). - Docstrings: numbering says what happens with no ledger; the module header states the gap-line order. Tests: test_a_new_id_never_lands_in_a_declared_gap, test_a_page_q_id_inside_a_declared_gap_is_numbered_after_it, test_export_ledger_keeps_a_gap_line_past_the_last_row_and_the_gate_refuses_it, test_sync_ledger_places_each_gap_line_by_its_range, test_the_gap_line_grammar_matches_the_gate (runs the gate and parse_gap on the gate test's valid and malformed gap lines), TestNextId.test_a_renumbered_row_lead_is_never_reused, TestNextId.test_a_pending_proposal_id_is_never_reused, and gate cases for a gap covering a row, a gap past the highest row and a prose `- gap,` bullet. The decision that a gapped export fails the gate until a hand-written gap line declares it is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
A register row's resolution can now end in five fixed-shape tags, and an answer a later fact contradicts reopens under its own number with every row built on it flagged (Phase 4 of the interview redesign plan). - exporters.py: one anchored pattern (TAG_TAIL, beside STALE_TAIL) peels a tail of `kind:: discovery`, `problem:: P<n>`, `depends:: Q<a>[, ...]`, `reopened:: round <N>` and `recheck:: Q<a>[, ...]` from the right of a stale-free resolution (split_tags; join_tags writes it back). The tags are not in the ordered named-field list. parse_register still returns the cell with its tail, so rows copied as they stand (ledger-only rows, a kept conflict) keep it; each parse point peels it first. import_ledger keeps each row's tail in meta.seededFrom.tags (declared in questions.schema.json); register() writes it back after the answer and before `stale::`. A merge into a ledger keeps the ledger's own tail for a row it holds, so a tag the terminal lane wrote after an import (a reopen, a --rechecked) is not undone by the next sync. - edit_row is split into locate + rewrite_row, which peels the tail before change() and re-appends it, so supersede and resolve keep tags (before, a named row with a tag was refused as an unknown field). - reopen_row / round.sh reopen --ledger L Q<N> --round <n> --fact-file P: answered row -> open with `reopened:: round <n>`; `- Q<N> | round <n> | was: <old answer> | fact: <fact>` appended to `## Reopened answers` (created right after the register when absent); `recheck:: Q<N>` merged into every row whose depends:: reaches Q<N> at any depth (visited set, so a cycle ends); prints `marked Q<M>` per row. `--rechecked Q<M>...` (clear_recheck) removes recheck:: and refuses a row with none. Refuses a row that is not answered and an empty fact. Follows the cmd_supersede -> supersede_row precedent: no data dir, backup, gate verdict. - check-open-questions.sh: a resolution ending in a tail with recheck:: (before any stale::) counts as `recheck=<n>` and blocks like stale::; a settled row whose resolution is only tags has an empty resolution under --procedure; rows_outside skips a `## Reopened answers` section, so its `- Q<N> | round <N> | ...` lines are not rows outside the register. `- P<n> |` rows in `## Problems` were never graded and still are not. - check-round-codes.sh: `- P<n> |` rows join the ledger codes. - The two scripts' header comments are not extended for the tags, the recheck count or P<n> codes: the pre-commit comment lint refuses new comment lines, so this message and surface/README.md carry the contract. - Phase 3 carry-over (a): export_ledger and sync_ledger both refuse a gap line the gate reads as malformed (gap_lines with a writer's `where`); before, export dropped it and sync kept it in place. - Phase 3 carry-over (b): surface/README.md's next-id row lists every source highest_id counts ([Q<N>] leads, page question ids); new reopen row; the import-ledger row names the tag tail. - templates/checklist.md: `## Problems` and `## Reopened answers` sections; the register header names the tags and points at loop.md. Not in this commit (reported to the orchestrator, blocked by existing tests the plan does not name as intended reversals): the exporter's `- gap | Q<a>-Q<b> | never assigned` lines (new D2) and carry-over (c), the trailing gap line the gate and numbering disagree on. Sanity check 1 as written (register byte-identical after import and export) does not hold for a plain-text answer: the exporter writes every row in its named grammar, so `free-text: staff; then contractors` comes back as `answer:: free-text: staff\; then contractors`, as it did before this phase. Every tag, every named-field row and `## Reopened answers` round-trip byte for byte, and the exported ledger is a fixed point. Pre-flight: parsers of the resolution cell and the tail - exporters.py parse_register: drops stale, keeps the tag tail in the cell (rows written back as they stand keep it). - exporters.py ledger_register / ledger_numbers / highest_id / row_id: read the [<id>] lead at the start and the row number; the tail is opaque to them. - exporters.py merged_register: compares and copies cells whole; takes each held row's tail from the ledger (split_tags) for register(). - exporters.py import_ledger: LEAD, then split_tags, then parse_named / parse_hold / split_tail / PROPOSES on the peeled answer. - exporters.py parse_named: called only on peeled text (import_ledger, recap_answer, ledger_row, displaced_answer and resolve_row through rewrite_row). - exporters.py drop_stale: unchanged; runs first, tags sit before stale. - exporters.py LEAD / REGISTER_LEAD (split_lead, locate_row): the lead is at the start; split_tags keeps it on the resolution. - exporters.py encode: writes named fields only; register() appends the tail after it (join_tags). - exporters.py recap_answer, ledger_row: peel before parse_named and readable(), so a recap or Brief line shows no tags. - check-open-questions.sh: stale_tail_re unchanged (the tail is before stale); recheck_tail_re and tags_only_re read the tail. - skills/audit-answers/SKILL.md: reads rows by status and a `hedged:` substring in prose; opaque text, unaffected. - skills/wayfind/SKILL.md: seeds from the checklist's settled answers in prose; opaque text. - plugins/work-items/skills/decompose/SKILL.md: reads answered Q<N> rows in prose; opaque text. Tests added - test_exporters.py TestRegisterTags: test_split_tags_peels_only_a_tail_of_whole_tags, test_import_keeps_each_tail_beside_the_answer, test_an_import_then_export_writes_every_tag_back_byte_for_byte, test_an_import_then_sync_leaves_the_register_and_reopened_answers_byte_identical, test_a_merge_keeps_the_ledgers_newer_tail, test_supersede_and_restore_keep_the_tail_on_a_named_row, test_the_brief_reads_a_ledger_only_row_without_its_tags, test_a_recap_reads_the_answer_without_its_tags. - test_exporters.py TestExportLedger: test_export_ledger_refuses_a_malformed_gap_line, test_sync_ledger_refuses_a_malformed_gap_line. - test_round.py TestReopen: test_a_reopen_flags_every_row_built_on_it_three_deep, test_rechecked_clears_only_the_named_rows, test_a_second_reopen_adds_to_the_reopened_answers_section, test_only_an_answered_row_reopens, test_a_reopen_lists_its_entry_under_the_sections_own_text, test_a_reopen_keeps_crlf_line_endings. - check-open-questions.test.sh: tags, Problems rows and Reopened answers rows grade clean; recheck rows counted beside stale (escaped `\;` not a mark); a recheck row blocks; cleared recheck is clean; tags-only resolution is empty under --procedure. - check-round-codes.test.sh: a bare problem id fails; a named one passes. No existing test changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
…aps reserve ids Finishes Phase 4 on the coordinator's decisions (DEVIATIONS.md "Phase 4, decisions"; owner row https://github.com/melodic-software/architecture/blob/b668e15b1993eb4df14ae8c0c60a31c43fa4adc7/docs/owner-decisions.md#L298). - D2: export-ledger and sync-ledger write `- gap | Q<a>-Q<b> | never assigned` for each run of numbers below the highest row that no row and no gap line covers and the data dir never recorded (never_assigned). "An id the ledger held is never declared" is enforced by a floor: no number at or below the highest number the ledger text holds (its rows, [Q<N>] leads and accepted gap ends) is declared, so a row dropped from the ledger still fails the gate. "No page question ever held" is a membership test over what the data dir records (page_held): every question's id and the ids it names (supersedes, followUpOf, dependsOn), every event in responses.json, each round's restate decided/open lists, the import's rows/roundCells/tags keys and meta.ledgerSync rows. It is a set, not a maximum, so a stray Q600 still leaves Q3-Q599 declared. Limit: the data dir keeps no question history, so a question removed from questions.json by hand that left no event and no restate entry is declared; test_a_question_lost_with_no_record_is_declared_the_limit shows it. A written line is a well-formed gap line the next run covers, so a second export or sync adds nothing. - N2 (carry-over c): the gate accepts a declared gap past the highest row (it reserves those ids); a range covering a row is still refused. numbering() and next-id honor exactly the lines the gate accepts (accepted_gaps), so a covering line no longer reserves ids. Basis: a deleted trailing row is undetectable with or without a gap line, and the tests this flips were added on this branch. - round.py edit_ledger runs the gate with shutil.which("bash"): on Windows a bare "bash" resolves to System32's WSL launcher, which cannot open the Windows path and exits 127, so reopen, supersede and resolve printed no verdict here (fixes the three TestSupersedeResolve host failures). - surface/README.md documents recheck=, the P<n> codes, tags-only resolutions, trailing gaps and the never-assigned lines (the scripts' header comments cannot grow under the comment lint). Intended reversals (authorized; only the gate-refusal assertions): - TestExportLedger.test_non_contiguous_ids_are_never_renumbered: rc 2 and "no row for Q1 to Q6" become the written `- gap | Q1-Q6 | never assigned` line and rc 0; the never-renumbered and lead assertions stay. - TestBriefIds.test_without_a_ledger_a_gap_keeps_every_id_with_no_lead: rc 2 and the gap-line message become the written `- gap | Q2-Q2 | never assigned` line and rc 0; the no-lead and hand-declared steps stay. - test_export_ledger_keeps_a_gap_line_past_the_last_row_and_the_gate_refuses_it is renamed ..._and_the_gate_accepts_it: rc 2 and "runs past the highest row Q2" become rc 1 and "registered=2 open=2" (two open rows). - check-open-questions.test.sh "a declared gap past the highest row -> 2" and "a gap past the highest row is named with its line" become "... -> 0" and "a gap past the highest row is not an error" (empty stderr). Tests added - TestNeverAssigned: test_without_a_ledger_ids_the_page_never_held_are_declared, test_a_second_export_and_a_second_sync_add_nothing, test_a_row_the_ledger_held_and_the_page_dropped_is_not_declared (gate exit 2), test_a_question_the_page_once_recorded_is_never_declared, test_a_question_lost_with_no_record_is_declared_the_limit. - TestRegisterTags: test_export_import_export_is_a_fixed_point, test_a_plain_answers_words_survive_the_named_form. - check-open-questions.test.sh: an escaped `\;` in the answer before `; recheck::` still blocks (exit 1, recheck=1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
Rewrites the 12 cases the Phase 1 audit marks rewrite and retires the six interview-pacing-* cases, per the approved interview-redesign plan. Rewritten (each moves its prompt past the intake round by stating the round-1 answers, adds Bash and Agent to allowed_tools, and gains one grader only the new skill passes): - intake round asks discovery questions with no recommendation (Brief AC8), so cases that graded a round-1 recommendation now grade round 2, and gain scope-check-line (the round header's Scope check: line, AC7): interview-intent-asked-recorded, interview-negative-recommendation-yes-accepts, interview-option-existing-control-drops, interview-option-known-repetition-keeps, interview-recommendation-basis-and-flip, interview-scenario-walk-two-purchase-kinds, interview-scope-cut-moscow, interview-term-clash-qualified-name, interview-context-unfamiliar-term-upstream (its Context regex also ends a block at Discovery:, Held:, Second view:, Departs from: and Scope check:) - the default standard detail level prints no Basis: or Origin: line (AC5, AC6), so "a Basis: line" becomes "a recommendation whose reason names its basis": interview-round-topic-groups (gains scope-check-line), interview-single-pacing-one-question (gains no-basis-line) - the opening lines take the owner-approved round-8 shape (AC14): interview-surface-unset-offer-one-line swaps one-offer-line for opening-lines (count 1, same strength) and accepts an open round-1 question Retired: interview-pacing-round-arm, -single-arm, -single-arm-open-b, -single-arm-open-d, -single-arm-step-2, -single-arm-step-3. The experiment concluded (run 3: ROUNDS HOLD UP) and measured a layout the owner changed (question_detail levels); its graders are frozen by its own rule. Recorded as Amendment 3 in interview-pacing.md; question-quality coverage moves to the new case interview-round-quality-standard (next commit). Legacy skills/interview/evals/evals.json: entries 1, 5, 7, 8, 20, 23, 25, 27, 28, 43, 44, 48 edited where an expectation became false (intake round, standard print, terms written after confirmation). Re-pins in tests/interview-defenses.test.sh (the read of each case's diff shows the defense intact): - case 1 still resolves codebase-answerable questions without asking: 5e38782253a410890320cddc3c5c44d71ab64666bb596bc954167f1447059bdd -> 1065f9d322d8ae3b5179ac13ec75e615044f42b7ada5455737d285048032a217; the codebase-resolution expectation is unchanged; only the recommendation expectation is scoped to decision questions. - case 8 still asks decisions and only looks up facts: 869a487d5f0abe19cdbe5693b90beabb55f56fbdacf6642bedad62494ce029f5 -> bedb9fd8e5923395d229e36cf50479530e58abcd8503a3ae989212162b789c45; the format is still looked up and the precedence still asked with a recommendation, now placed in the first decision round. - case 25 still refuses to settle a row on a superseded recommendation: 186a511bef86bbc7fe9679359c350c39111c3eab0c8761eb8f98196882f40b9f -> a3e7ffd4412ccfb51465f0be3d871221a6c288d53131c8436ef6025bc30974bd; only "recommendation with Basis" became "recommendation with its reason (its basis)"; the supersede and revalidate expectations are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
Seven new cases under plugins/planning/evals, each one prompt with a scaffolded ledger or stated round-1 answers, deterministic graders first: - interview-numbering-tool-display-order: a ledger ending at Q7; the run calls round.sh next-id, new questions start at Q8 in the order shown, none reuses Q1-Q7, and the ledger gains an open Q8 row (AC4). - interview-numbering-stray-id-not-reused: a ledger with Q1, Q2, a declared gap and a stray Q600; new ids start at Q601 and no id changes (AC4). - interview-reopen-contradicted-answer: a reported fact contradicts answered Q3, and Q5 depends on Q3; the run calls round.sh reopen, the ledger shows Q3 open with reopened::, Q5 with recheck:: Q3, and the old answer under ## Reopened answers (AC4). - interview-held-until-lookup-returns: a lookup reports back in another session tomorrow; the dependent question sits on the Held: line and is not asked (AC9). - interview-intake-discovery-only: a first request with unknowns; round 1 asks only open questions with a Discovery: line and no My recommendation: line (AC8). - interview-sweep-reports-contradiction: every row answered, two that cannot both hold; the reply asks one question naming both. The eval evidence for removing the per-round contradiction pass (AC10). - interview-round-quality-standard: the retired round-arm prompt with graders for the standard layout (coverage of the four planted decisions, Context citing a stated fact, a recommendation whose reason names its evidence or Withheld:, alternatives, a three-dash rule between questions, the Scope check: line, the closing probe). Not part of the frozen pacing experiment. Every case validates offline (validate-cases.py exit 0) and holds no transcript. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
…wer entries Code-design review of Phases 3-4 (.work/reviews/feat-interview-redesign/20261121T1100Z-code-reviewer.md). - C1: rewrite_row, on the reopen and --rechecked paths (the ones that pass retag), writes the row's `stale::` mark back after the new tag tail, and format_tags keeps every tag the path does not own. Before, reopen then --rechecked removed `stale:: C<n>` and the gate passed a row whose constraint was never rechecked. supersede and resolve (edit_row, no retag) still drop the mark, as main does. - C2: check-open-questions.sh rows_outside skips under `## Reopened answers` only a line in the entry shape `- Q<N> | round <N> | was: ... | fact: ...`; any other `- Q<N> |` line there is a row outside the register again (exit 2), as on main. - S4: export-ledger --ledger writes the ledger's `## Problems` above the register and its `## Reopened answers` last (carried_section), as sync-ledger leaves them, so the amended round trip holds through export-ledger too. - S1: add_reopened creates the section after the register's whole level-2 section (section_span), its `### Deferred questions` included, where the checklist template puts it; headings() now returns each heading's level. - S2: parse_tags names the row in its repeated-tag refusal; a parity test runs the gate on rows split_tags reads and compares the recheck count and the tags-only rows. - S3: the never_assigned docstring states its floor as the highest number the ledger's current text holds, and the limit above it; README too. - S5: meta.ledgerSync.rows holds `Q<N>` strings (round.py record_ledger_sync builds them from each drift note's leading Q<N>, or `order`); page_held reads them as such, tested through the real ledger-check writer. - The TestRegisterTags fixture puts `### Deferred questions` inside the register section and `## Reopened answers` after it, the template's order. Tests added - test_round.py TestReopen: test_reopen_and_rechecked_keep_a_rows_stale_mark (fails with the fix disabled), test_a_new_section_goes_after_the_registers_deferred_questions. - test_exporters.py TestRegisterTags: test_export_ledger_keeps_the_problems_and_reopened_answers_sections, test_the_tag_grammar_matches_the_gate, test_a_repeated_tag_is_refused_naming_its_row. - test_exporters.py TestNeverAssigned: test_an_id_only_the_last_ledger_check_recorded_is_never_declared. - check-open-questions.test.sh: a register row under Reopened answers is still outside the register -> 2; only the entry-shaped line is skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
…estion_detail setting Phase 5 of the interview redesign, without the round-file structure check (see below). - round.sh show --level full|standard|minimal <round-N.md>: prints a saved round at the plan's detail level. full is byte for byte; standard drops Origin:, Basis:, Checked against: and the Source: quote's body (its links stay) and puts each alternative on one line; minimal also drops Context: and Source: and prints `Alternatives: <n> (say "more Q<N>")`. The header, blank lines and `---` rules stay at every level; line endings are kept. - round.sh scope-check --ledger L --round N --limit M [--rounds-dir D]: `<rows> of M questions registered; round <n> added <a>, settled <s>` for the two rounds before N (settled: answered, deferred, withdrawn, blocked). With D, the two rounds are the highest below N that have a round-<n>.md. - round.sh measures --ledger L --rounds-dir D --level V: `rounds= median_lines= reopened= after_lookup=`; a fact names a lookup when it holds a /<plugin>:<skill> or the word lookup. - The shared fixture surface/tests/fixtures/round-file/round-full.md, read by test_round.py and check-round-codes.test.sh. - question_detail (full, standard, minimal; default standard) in plugin.json userConfig and planning.schema.json beside question_pacing, its row in reference/config.md "Convention-home keys" (now five interview keys), the generated README options table, and surface/README.md rows for show, scope-check and measures. Not in this commit: the two structure checks in check-round-codes.sh. Added unconditionally, they fail eight existing check-round-codes.test.sh cases whose rounds have no Scope check: header or a question with no recommendation line; the brief forbids editing those tests. Tests added - test_round.py TestRoundShow: test_full_prints_the_file_byte_for_byte, test_standard_drops_the_evidence_lines_and_the_quote_body, test_standard_keeps_the_source_link, test_minimal_keeps_the_answer_lines_and_counts_the_alternatives, test_no_level_hides_a_flips_if_or_commits_you_to_line, test_crlf_line_endings_are_kept, test_an_unknown_level_is_refused. - test_round.py TestScopeCheck: test_the_last_two_rounds_before_the_shown_one, test_round_two_has_one_earlier_round_and_round_one_none, test_the_rounds_dir_names_the_rounds_that_were_shown, test_a_round_below_one_is_refused. - test_round.py TestMeasures: test_rounds_median_lines_and_reopened_rows, test_median_lines_count_the_round_at_the_shown_level. - check-round-codes.test.sh: the shared round fixture names every code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
The rest of Phase 5: check-round-codes.sh runs two structure checks beside the bare-code check, unconditionally, as the plan specifies. - Each question block holds exactly one My recommendation:, Withheld:, Discovery: or (the product action) Strong answer: line; the lines before the first question hold a Scope check: line. Failures print `missing=<line>:recommendation|single-recommendation|scope-check`, and the summary keeps its `codes= bare= status=` prefix and appends `structure=<n>`. Exit 1 on a bare code or a structure failure. - One label grammar in both parsers: an optional marker (characters outside printable ASCII, such as an emoji), optional bold, the label and its colon. round.py show used `[^\x00-\x7f]*\s*`; both now use `[^ -~]*[ \t]*`, so a Context bullet such as `- Basis:` is never a label in either. - The eight existing check-round-codes.test.sh rounds that the checks would fail gain only a Scope check: header line and an answer line; no assertion text changes. The passing run of the walk-overlap round reads a sibling file, walk-overlap-round.md, so the failing run's `bare=3:S1` line number stays as it was. Tests added - check-round-codes.test.sh: the shared round fixture passes the structure checks; a Context bullet that starts with a label word is not a label; a decision question with no recommendation fails; a missing recommendation fails the summary; a discovery question with a Discovery: line passes; a header with no Scope check: line fails; a bold recommendation label after a marker counts; a question with two answer lines fails; a product-action round with a Strong answer: line passes. - test_round.py TestRoundShow: test_a_context_bullet_that_starts_with_a_label_word_is_not_a_label (fails with the old `[^A-Za-z0-9]*` prefix). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
…nd recommender and stable ids Rewrites the interview's flow text so the model calls the Phase 3-5 tools and follows the redesigned flow (Brief AC8-AC12; plan Phase 7). - context/lookups.md (new): the one home for the gap triage before round 1 (lookup, discovery question or decision question), the intake round (only open discovery questions; none when the triage leaves none), when a lookup runs and what its brief carries (lookups named by what they read; one running dispatch per lookup skill; an inline read when no explore or research skill is listed), held questions (a lookup the user reports running elsewhere holds too), and round.sh reopen for a fact from any source that contradicts an answered row. - agents/interview-recommender.md (new): the blind recommender; Read, Grep and Glob only; no model or effort field; one line per question, recommendation with sourced basis, flips-if and departs-from, or withheld; no reasoning written out. - SKILL.md: Stance item 4 loads the glossary at the start of every interview with one line and writes a term only after the user confirms its restatement (IMMEDIATELY removed); the pre-round-1 terms question becomes "Agree a term where it first appears"; a discovery question carries a Discovery: line and no recommendation (the idea-stage withheld case moves to it); ids come from round.sh next-id in display order (no-Bash fallback stated); held questions, the lookup rule and reopen point at lookups.md; Step 2 gains the recommender dispatch with Second view: and Departs from: lines and the no-Agent fallback; the sweep reports every finding and the disposition pass filters; quality bar asked as a discovery question. - context/loop.md: Frontier rounds item 2 states the round order, items 3, 4, 7 and 9 match the new flow; the register documents declared gap lines, the tag tail and the Problems and Reopened answers sections; Out-of-band drift checks answered rows too (outcome 4, reopen); terminology matrix rows are added at first use. - context/assumption-sweep.md: the brief asks for every finding with no cut; Disposition is named the filter pass; new rows take next-id; a contradiction question names both rows by id and plain name. - context/routing.md: Problems section (P1, P2 agreed in round 1, the problem:: tag, no fourth wayfind trigger) and the scope-check line from round.sh scope-check, with its no-Bash fallback; the round-1 wayfind question keeps its recommendation in an intake round. - context/composition.md: explore and research are lookups whose brief carries the constraint ledger (no "reads the Brief as scope"); rows for the recommender and the glossary lookup. - context/handoff.md: no after-the-Brief explore or research pass. - context/term-clash.md: an agreed qualified name is written with the other agreed terms after confirmation; the matrix keeps it otherwise. - context/gotchas.md: "gapped Q<N>" reads "an undeclared gap". - templates/checklist.md: Step 2 and the sweep exit match the new flow; Step 5 no longer routes to exploration or research; matrix and Reopened answers notes updated; the example register rows are deleted before the first next-id or scope-check call, which count them (a copied template made next-id print Q4). - audit-answers SKILL.md Steps 1 and 4 and evals.json id 1: an open kind:: discovery row is held out of the auto-accept and goes to the human. Re-pins in tests/interview-defenses.test.sh (no pin added, removed or loosened; every phrase and pin_exact pin passes unchanged): - SKILL.md Stance: e059b8a9bdadf3a44593745098d9365d46b80fe2e2e9bdb3d449ab124d0481b0 -> 7d7f2372cef7aaa76d271c3e57e6a9a6208727c94b4dce3c6d33357f2d2926e4: the edits add the discovery shape, next-id numbering, the lookups pointer and term agreement; the no-silent-resolve rule and the partial-round rules are byte-identical, and a discovery row is never resolved by an accept-shorthand. - loop.md open-question register: e44127457a9cf37557b903698e15949a28c723a1ed32049c53c5b1fa861c45bf -> 17235dec719241d850f2ed32efecce845e4d4eac28df159c58c0cead6a05ca81: adds declared gaps (the gate still refuses any other gap) and the tag tail; outcome 4 reopens a contradicted answer instead of changing it; ask-time writes, the drift check, the unattended ladder and the gate are unchanged. - SKILL.md Step 2: f332b13eda79cde79825d912827e70b18971870c60accde42f0ff5c79ca3bcea -> d9c84aac832447d47e2f50d5008a449d378bfa3282291d0334c9bd65129a8744: adds the recommender paragraph and the sweep's report-all wording; the constraint ledger, candidates, currency, tripwire and sweep rules still hold and no row is resolved for the user. - loop.md Step 2: d91d5185a66e75854f16198128e991ce047bc6129c947a435078d12dca083df0 -> cd3481b6f2d547f82f33ee14cc69e22b38a5c936f857461f9e8538e18cdfd3d1: round order, the lookup pointer, the recommender clause and term agreement; the constraint ledger rows and the process-change definition are byte-identical. - loop.md relentless mechanics: 493b9e062228d39672b85ea3256e8a3c360fce9df183d08de9ea081666e20076 -> 53ad96eee9dc6133be50f497d4228b725006506df953ad9d24bedbc087ea9c9d: next-id on the counter line and matrix rows added at first use; the late commitment row still takes the next free id. - context/assumption-sweep.md: 4d1a6e8c8789645fb58bbae26822fc34d2fdcefda674e0d7210247dcccc8f0f3 -> 19473131cf19c03e6f3f17d95c9012b143b505435d0d0eb8a21239774b4b6c0a: the sweep now reports every finding and Disposition is the filter; scope, item shape, every disposition, USER-RESERVED and the fork ban are unchanged. - audit-answers Step 1: d1141bf7988e5cca6c1f9566d2b61ee48d7d627d8ebe5fe1855799d6fb539b71 -> fcd7f1b010811f56cc798884acb234563423896cec52d507c8a6d2520686195f: adds one floor item (an open discovery row is never auto-accepted); the never-auto floor only widens. - audit-answers Step 4: 58e23f0911f4ee7a3cc0ff33cf3ccadf7a539cc2b680e1dc7f1344fe0fb40240 -> bdbde2c21e6f1c7bc05b752b25541ce15d59f703ba6d83a936d64842e0c1fd44: discovery rows join the rows routed to the human; hedged rows still always reach the human. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
scripts/validate-plugin-contracts.mjs fails any agent definition with no model line, and accepts `model: inherit` only with a trailing `# reason:` comment. The plan's Design wants the recommender to run on the session's model and name none; `inherit` does exactly that, so the Design's intent holds and the contract check passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
… for settled statuses Code-design review of Phase 5 (.work/reviews/feat-interview-redesign/20261121T1200Z-code-reviewer-phase5.md). - C1: a list item (`- Q4: ...`, `1. Q4: ...`, after any `>` quote) never opens a question, in check-round-codes.sh question_id and in round.py question_id. The shared fixture gains a `- Q4:` Context bullet under Q10. - C2: exporters.SETTLED (answered, deferred, withdrawn, blocked) beside UNSETTLED, used by the status count, the import status check and round.py scope-check. The gate restates the set in bash, so a test runs the gate on each status and compares the ones it settles. - C3: exporters.reopened_entry writes and REOPENED_ENTRY / reopened_entries read the `## Reopened answers` entry; reopen_row and round.py measures use them. TestMeasures builds its entries with `round.sh reopen`, and a test runs the gate on reopen_row's output. - I1: a line that starts with a protected label word in any form (Flips if, Commits you to, Withheld, Discovery, Second view, Departs from) starts a segment kept at every level, whatever field it follows. - I2: round.py takes check-round-codes.sh's fence rule (quote and list prefixes, the fence character, a closing run at least as long with nothing after it). The shared fixture gains a four-backtick fence holding a three-backtick line, a `Q9:` line and a `Basis:` line. - I3: LOOKUP matches lookup, lookups and looked up, and a /<plugin>:<skill> only when not inside a path. - I4: tests for the even-count median (integer and .5), zero rounds, a Source: with no link, and inline alternatives, which now stay as written at every level instead of printing `Alternatives: 0`. - S3: the changelog names every protected line. S4: the show docstring and surface README say every run of blank lines collapses. S5: the header needs exactly one Scope check: line (`missing=1:single-scope-check`). Tests added - test_round.py TestRoundShow: test_a_protected_line_under_a_dropped_field_is_kept, test_a_source_with_no_link_prints_nothing_at_standard, test_inline_alternatives_are_kept_as_written; the fixture's new Q4 bullet and fence are asserted by the existing minimal and standard literal tests. - test_round.py TestMeasures (entries now written by reopen): test_the_reopen_writer_and_the_measures_reader_agree, test_an_even_count_median_is_the_mean_of_the_middle_two; the other two expect reopened=5 after_lookup=3 and the fixture's 58 lines. - test_exporters.py TestRegisterTags: test_the_gate_settles_exactly_the_settled_statuses, test_the_gate_skips_the_entry_reopen_writes. - check-round-codes.test.sh: a list item that opens with Q<N>: is not a question line; a header with two Scope check: lines fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
…one home per rule Phase 8 of the interview redesign (Brief AC5, AC6, AC13, AC14), plus the four IMPORTANT findings and the cheap SUGGESTIONs of the Phase 7 code-design review (.work/reviews/feat-interview-redesign/ 20261121T1300Z-code-reviewer-phase7.md). - SKILL.md: the per-question template is the saved full block, with a blank line after the title and a `---` line between questions. "Keep the layout whole" owns Context first and the `---` rule (product.md is a declared exception); new "Save the full block; print it at the detail level": the round sent is `round.sh show --level <question_detail>` output, "more Q<N>" and "full round", the protected lines, and the no-Bash standard print. Origin:/Basis: bind the full block; the printed reason ends with its evidence label and source at every level. New "Open the round header with its scope check". The bare-code paragraph describes the structure checks and how to fix each missing= line (scope-check, single-scope-check, recommendation, single-recommendation). "One question at a time" is renamed "Question pacing". Round order, ids, the shared lane's cap, commitment parts per lane, scenario ids, Context terms, held questions and lookups point at their homes. New "Three glossaries, and who writes each"; the closing summary group is renamed "Project glossary changes". Step 4 gains "When the contract is persisted". The Question lanes resolver call passes --pacing and --ledger and prints `line` as the opening lines. - Review fixes: a term's restatement is registered like any question (finding 2); the glossary plugin offer is in offers/offerable.json (finding 3); a routing suggestion in an intake round is one plain line, never a `My recommendation:` block, in lookups.md and routing.md (finding 4); interview-recommender.md says it inherits the session's model (finding 1); a withheld question shows no Second view: line, a basis fact missing from the Context is added to it, the quality bar's three bars sit in its Context, discovery questions may arise in any round, the sweep's "no severity, confidence or blocking cut" matches assumption-sweep.md, the hold and reopen rules are stated once in lookups.md (SUGGESTIONs 5-10). Skipped: 11 (unattended rung for an open discovery row: loop.md Unattended path is a pinned ladder and the change is a design call), 12 (interview-person-deferred-document grader: an eval grader edit outside Phase 8's files). - lookups.md: one home for A1-A4, A6, A7: a released held question is asked in the next message that presents questions; a recommendation is never shown before its facts (held, else withheld); new "Passes around the interview" (before round 1, during the rounds, after confirmation; no context clear between passes). - loop.md: Frontier rounds item 2 is the one statement of round order (item 4 and SKILL.md point at it); the header names Scope check:; item 5 points at SKILL.md Step 4 for when the Brief is written; the counter points at question-shape.md; the shared cap is the one exception to "never capped"; the degrade table's column is headed `Q`; the glossary is read at the start of every interview; terminology restatements are registered. - question-shape.md: the `id` row (one counter per topic, assigned by next-id when first shown); the alternatives count in every lane; the Context terms rule is the one home. - lane-shared.md: "Round cap" (8), the id rule via next-id, gaps declared by a gap line, terminal-only layout rules. lane-page.md, lane-document.md and surface.md: id and gap pointers, terminal-only layout rules. - product.md, scenario-comparison.md: the declared exception; the scenario id home. templates/document.md: a blank line between label blocks. templates/checklist.md: the sweep's seven categories. - composition.md, handoff.md: pointers to lookups.md and the renamed group. - reference/config.md, schemas/planning.schema.json: question_detail names every line no level hides. - surfaces/registry.mjs: `line` is the round-8 opening, one sentence group per line: `Asking here, one round|question at a time. A local page or a shared session is also available; say which to switch.`, `Questions are saved to <ledger>.` (with --ledger), then one `Optional:` line per installed planner and per offer. Entries gain optional `where` (terminal: here) and `offerName` (page: local page). Readers of `line`: SKILL.md "Question lanes" only (the lane spokes print notices, not it). - offers/offer-source.mjs: the offer text is `<name> <helps>. Say "install <name>" to add it`; offerable.json adds domain-driven-design. - Legacy skills/interview/evals/evals.json: "next contiguous Q<N>" becomes "next id from round.sh next-id" (no digested case touched). - session-flow steps.md and pre-implementation-order.md: pointers only. Tests - surface-registry.test.mjs: the opening lines asserted exactly against the Q38 wording and the eval's first-line regex; pacing single; no ledger, no saved-to line; planner and offer lines; the offer name is an argument word. The null-line, unavailable-surface, document and kiosk assertions are kept. - offer-source.test.mjs: the owner-approved offer text; the glossary entry is listed by this marketplace at ./plugins/domain-driven-design. Re-pins in tests/interview-defenses.test.sh (sha literals only; no pin added, removed or loosened; every phrase and pin_exact pin passes): - SKILL.md Stance: 7d7f2372cef7aaa76d271c3e57e6a9a6208727c94b4dce3c6d33357f2d2926e4 -> 2f45a095d33c22488286ffd2c5b0248b1f3f9123084245b4c2fbc9916fd94406: layout, detail levels, scope-check header, pointers to the rules' homes and term restatements registered; partial rounds still never resolve an unanswered question, and only where it re-surfaces moved to loop.md item 2. - SKILL.md Step 4: 3fee6385252f2f476aef4e00797c9f416e7a7c7f89e3ea6ebdabad267177a640 -> 6e6577f57f115fa749e6fb5ec9bef7bf860cfdc07d36aa7aa064563f7f02b981: adds "When the contract is persisted"; the cross-check, the UNCONFIRMED stamp and the scope path are unchanged and nothing acts on an unconfirmed Brief. - SKILL.md Action Router: 9ec05ae465fc8e693fde59be1e81d386cd350f63021787ecc29179fc23eff758 -> 368743e50f46c68695014cc10793b89a1cf2c0b2e84258798579c4fbe0e5c36d: the resolver passes --pacing and --ledger, the opening prints as lines, and the cap names the shared lane's exception; the lock row and STOP-on-gap are byte-identical. - SKILL.md Step 3: 101586e9916d476589a8519aa93cf89121ad517d46278a480aab737027563419 -> d1aedd09a3cc54f12763212381e7ef7b0731165303cf4b18ba6e1da2c9526533: the Glossary group is renamed "Project glossary changes"; the gates and the confirmation exemption are unchanged. - SKILL.md "does NOT do": 261ea805a3e429229840ef9c86bacdae791acedfa9aedbfb2d95c5e949273d84 -> 6fbbb933036f9c1f2b060ec0e16099fa04bfb8082a8f429e396a462cd16856de: "in-flight lookup" becomes "a lookup" with its link; the fudge prohibition is byte-identical. - SKILL.md Step 2: d9c84aac832447d47e2f50d5008a449d378bfa3282291d0334c9bd65129a8744 -> 4f1230d595f67a615b884320885dcc743dd42080822d3197a2f4a823ab7cb95d: a failing consequential recommendation is held for a lookup instead of shown "not yet researched" (stricter), the recommender's withheld and missing-fact clauses, the sweep's full no-cut wording; constraint ledger, candidates, tripwire and sweep rules hold. - loop.md Step 2: cd3481b6f2d547f82f33ee14cc69e22b38a5c936f857461f9e8538e18cdfd3d1 -> 221af05eb4cb7c1f58a756f9218fc7f23ad420e5266d569ca2db6ca379ffd416: item 2 is the one round order, the header names Scope check:, item 5 points at Step 4, restatements are registered; the constraint rows and the process-change definition are byte-identical. - loop.md relentless mechanics: 53ad96eee9dc6133be50f497d4228b725006506df953ad9d24bedbc087ea9c9d -> 713bf5f475597226013d471a5cc1a41a4579e256b844c69d8d5c9b28aea15a01: counter and cap pointers, the `Q` column, the restored registration of a term restatement; the late commitment row still takes the next free id. Not changed: the Gates section, since no text may be added above the Action Router; its "Full rule: Step 3/Step 4" pointers already lead to the G1 home. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
A colon followed by a space inside an unquoted YAML scalar made claude plugin eval refuse both case files (YAML Parse error: Unexpected token); validate-cases.py did not catch it. The text now reads 'a Scope check line'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
…nded discovery rows Phase 8 code-design review (.work/reviews/feat-interview-redesign/20261121T1400Z-code-reviewer-phase8.md). - IMPORTANT 1: SKILL.md "Save the full block": under `single` pacing each message carries the round header and only that question's block from the `round.sh show` output (text only; round.py unchanged). "The full layout" becomes "the whole question layout", defined there as a block printed at the resolved level, never the `full` level (SKILL.md, loop.md "Discussing", routing.md, term-clash.md). The interview-single-pacing-one-question graders still match: they check one question with its context, reasoned recommendation and alternatives, and no Basis:/Origin: line. - IMPORTANT 2: legacy evals.json case 27 (resumed-recommendation-re-grounded-against-a-later-constraint): the "marked not yet researched" alternative becomes "held for a lookup and named on the round header's Held: line", in expected_output and expectation 2. Not digest-pinned. - IMPORTANT 3: loop.md "Frontier rounds" item 2 says how its classes sit inside topic groups; SKILL.md "Group the round by topic" and item 4 point at it. - Open question (decided by the coordinator): loop.md Unattended path gains "An open discovery question takes rung 3" beside the problem confirmation's rung; rungs 1-5 are byte-identical. - SUGGESTIONs: config.md and planning.schema.json point at SKILL.md's protected-line list instead of copying it; lookups.md says the interview routes no explore or research pass after the Brief, while design and plan run their own (session-flow steps.md 2-3 agree); SKILL.md Step 4 names round.sh export-brief as the UNCONFIRMED stamp's writer, and says the confirmation gate alone holds a Brief written by hand. Re-pins in tests/interview-defenses.test.sh (sha literals only): - SKILL.md Stance: 2f45a095d33c22488286ffd2c5b0248b1f3f9123084245b4c2fbc9916fd94406 -> 913182d8f122427687e3daa10eb5a74f25cd0af1df918c68a094d162ac4bb128: the single-pacing print, the "whole question layout" wording and the topic-group order pointer; no-silent-resolve and partial rounds are unchanged. - loop.md open-question register: 17235dec719241d850f2ed32efecce845e4d4eac28df159c58c0cead6a05ca81 -> 272e5524066f4072445dcf55f67574eb3ed49ceeee681fcbcc71f86d075df83f: "Discussing" says the whole question layout, and the Unattended path paragraph below; the gate, drift check and ladder rungs are unchanged. - loop.md Unattended path: dde4506c0ec506447f50ebe6aea64cc35a2f917a6707d924a62bd7777910df77 -> fd049b8e15a5dcc646e5a69e778d17f60b3e3df17353d0f2d5d72d6919a7f0f7: adds one paragraph putting an open discovery row on rung 3 (blocked, USER-RESERVED), which only widens what is never assumed; rungs 1-5 are byte-identical. - SKILL.md Step 4: 6e6577f57f115fa749e6fb5ec9bef7bf860cfdc07d36aa7aa064563f7f02b981 -> 68a8e1087285efdae51d408be2e5d551399de6c98cf1c4e8142a5d3633dd3317: the persistence paragraph names export-brief as the stamp's writer; the cross-check and brief=unconfirmed exit are unchanged. - loop.md Step 2: 221af05eb4cb7c1f58a756f9218fc7f23ad420e5266d569ca2db6ca379ffd416 -> f458232e95be64764519502cbb91bf973ce0cef5f8d1e7dbf96b1e4f87eebb6f: item 2 orders questions inside topic groups and item 4 points at it; the constraint rows and the process-change definition are byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
The shared round fixture holds a round exactly as the interview saves it (Context: with its bullets directly under it, a fence inside a Context bullet), which MD031 and MD032 reject; a per-directory .markdownlint-cli2.jsonc turns off those two rules for that fixture directory only, as seven other fixture and vendor directories already do. round.py: labelled -> labeled (typos check). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
run-validity.py marks a with-arm run INVALID on a sandbox network denial unless its case is tagged no-network (plugins/evals/skills/plugin-eval/ SKILL.md "a case whose prompt.md or case.yaml tags include no-network"; only run-validity reads the tag, the CLI does not). Rule, written before any counted batch: a case gets the tag when its prompt or scaffold supplies every fact its graders depend on, so no grader can be met only through a network lookup. Applied by reading every case of the Phase 9 run set (74: the 68 interview cases including brainstorm-spec-stage-no-candidates, the seventh new case, and five out-of-tag cases). All 74 meet it: the graders judge what the reply does (questions asked, wording, ledger rows, files written), and every fact they name comes from the prompt or the scaffolded repository. Where a grader names a vendor (Loqate, Shopify, PagerDuty, Kafka) it grades behavior, not a vendor value; interview-recommendation-basis-and-flip accepts a fact the user stated, so a URL is never the only way to pass. No scaffold fetches anything (grep for curl, wget, git clone, pip, npm, http in every scaffold). Ten cases already carried the tag and fit; 64 gain it, one line each, nothing else changed. The per-case table with each reason goes in the PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
…acter budget validate-plugins failed the pre-push gate: the description was 359 characters (docs/contracts/plugin-option-naming/README.md budget 300); now 294, README options row regenerated with scripts/sync-plugin-options-docs.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
Case edits from the Phase 9 eval diagnosis (CASE and ENV rows), approved
by the owner on 2026-11-21 ("Approve the edits"). No grader check is
dropped; each change aligns a check with the approved design.
CASE rows:
- quality-bar-stated-production, quality-bar-stated-prototype,
intent-prototype-one-line, asks-a-round regex:
`(?:\*\*)?(?:Q1...` -> `(?:❓\s*)?(?:\*\*)?(?:Q1...`. The default layout
prints the emoji marker before Q<N> (layout-contract.md); the check
still needs a numbered question ending in `?`. A pass sample of the
emoji shape is added to each.
- intent-prototype-one-line (AC8 clash): the prompt now supplies the
intake round's answers and asks for round 2, and the scaffold writes
the ledger with Q1-Q2 answered `kind:: discovery`. asks-a-round's id
anchor `Q1\b|1[.)]` -> `Q[1-9]\d*\b|[1-9]\d*[.)]`, because round 2
opens at Q3; it still requires a numbered question. one-line-intent
and the no-quality-requirement check are unchanged; description and
expected_outcome now say round 2.
- coverage-prompt-asked (AC8 clash): the prompt supplies the intake
answers (quality bar, users) and asks for round 2; the coverage regex
is unchanged.
- coded-terms-round-answerable, answerable-without-session criterion 3:
"shown again with its context, recommendation, basis and
alternatives" -> the same, plus a withheld question meets
recommendation and basis with a `Withheld:` line naming the settling
evidence, still with context and alternatives (SKILL.md "A withheld
question carries no recommendation marker"; Brief AC6).
- round-topic-groups, topic-groups criterion 3: same `Withheld:`
acceptance; alternatives stay required.
- lock-restates-problem, restates-problem-and-asks:
`(?:that|this)\b[^\n?]{0,80}\?` -> `{0,300}`; a one-line problem title
of ~190 characters is still a restate-and-confirm question. Pass
sample of that length added.
- owner-deferred-document, document-beside-next-step criterion 3:
"does not choose an SSH key type" -> the same, plus naming example
key types without choosing or recommending one is not choosing. A
chosen or recommended type still fails.
- single-pacing-one-question: the one-question-full-layout pass sample
carried the Origin:/Basis: lines the case's own no-basis-line regex
forbids; it now shows the standard layout. Graders unchanged.
ENV rows:
- max_turns raised to 20 where a run used 12 or more turns in
b2-new-main or b3-old-kept-failing: closing-probe-something-else
(12 -> 20; measured 13 and 19), crux-on-rejection (12; 12 and 13),
document-dba-replica (15; 18), fact-looked-up-not-asked (16; 23),
prototype-split-by-question (12; 13 and 19),
quality-bar-asked-bookmark-sync (12; 20),
quality-bar-asked-ticket-dashboard (12; 19),
quality-bar-stated-production (12; 23),
quality-bar-stated-prototype (12; 21),
round1-unknowns-offers-wayfind (12; 17),
wayfind-size-suggestion (12; 15).
- adr-verdict-overruled, scenario-walk-two-purchase-kinds: the prompt
invokes the skill in words instead of a slash command, since
run-validity never counts a slash expansion as a Skill call; request
text and skill-fired grader unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
This was referenced Nov 21, 2026
This was referenced Nov 21, 2026
Skill-side fixes for the Phase 9 eval diagnosis (SKILL rows, and the ENV rows the skill's wording caused). Every change adds or clarifies a rule; none loosens a gate or a defense. SKILL rows: - brainstorm-spec-stage-no-candidates: lookups.md gap triage routes a doubt about a stated fix to Captured assumptions, never a question; problem-framing.md "Prescribed solution" allows at most one line naming no other cause or approach. - adr-verdict-overruled: SKILL.md overrule paragraph states the ADR candidate in full in the reply when no register row exists. - consumer-standard-override: loop.md requires the literal `- Overrides bundled default <principle> per <file>` line whenever a standard decides a recommendation, lock and unattended included. - context-unfamiliar-term-upstream: question-shape.md Context defines the step or topic a question decides about, and request facts cite `user, round 1`, never "(your request)" (lookups.md example aligned). - coverage-unattended-unexamined: coverage-prompt.md gives the unattended line word for word. - crux-on-rejection: loop.md Drift check gives a rejected technical recommendation one crux question; a preference rejection is accepted. - final-brief-changes-only: SKILL.md Out of scope omits accepted answers; the one-line problem comes from the ledger or is said to be not stated. - option-existing-control-drops: recommendation-basis.md labels a control read only in a document `documented, unverified`; SKILL.md links it. - prescribed-solution-rethink: lookups.md intake round makes the rethink-or-accept question the intake problem question, with the inferred problem and its rating in its Context; problem-framing.md points there. - recommendation-basis-and-flip: SKILL.md reason names the specific fact; a judgment-only Context fact is researched or withheld; coined terms and P/Q ids are glossed. - reported-problem-instance-question: problem-framing.md pairs the instance question with the confirmation ask; lookups.md asks it in the intake round. - scenario-walk-two-purchase-kinds: scenario-comparison.md builds only scenarios the facts settle; a policy-dependent one goes to Later: or its own question. - single-pacing-one-question: SKILL.md Question pacing keeps the round.sh Scope check line and a Later: line that never previews the frontier. ENV rows caused by skill wording: - bare-vague-build-asks: SKILL.md Emit checklist says Read the template, then Write it, never a shell copy. - discuss-sets-discussing, prototype-split-by-question: routing.md states `--limit` takes the resolved integer, `27` by default; config.md:208 says the default is `27` with Basis judgment (it read "default is `judgment`"). Docs over a round.py default: a code default would be a third home for 27 beside config.md and the schema. - closing-probe-something-else (turn overhead): `round.sh next-id --round N --limit M [--rounds-dir D]` prints the ids and then the round's Scope check line, counting the new rows, so one call replaces next-id plus scope-check and the count includes this round (the run printed "0 of 27"). Tested in TestScopeCheck; README and usage text updated. check-round-codes still runs on every round. - surface-unset-offer-one-line (opening-lines): SKILL.md has its own "Print `line` exactly as given" paragraph. proceeds-to-round (NOISE/UNVERIFIED) not addressed. Re-pinned digests of interview-defenses.test.sh, the owner approving the re-pin 2026-11-21 after the auto-mode classifier first refused it (old -> new, why the defense holds); every whole-line pin_exact still passes unchanged: - SKILL.md Stance section: 913182d8f122427687e3daa10eb5a74f25cd0af1df918c68a094d162ac4bb128 -> 7624ab78474a662bce553d6918e1378709f2466d28ee6fdf18ec7db789e920b5. The "Partial-round resolution" no-silent-resolve rule is untouched; the edits add wording on reasons, pacing, ids and document-only controls. - SKILL.md Action Router section: 368743e50f46c68695014cc10793b89a1cf2c0b2e84258798579c4fbe0e5c36d -> da5d48aeabdf5a52a6827112557feba7fc4736a285c9b13a5565def7bd4de359. The `lock` row's whole-line pin passes; only the Question-lanes paragraph was split to give "Print `line` exactly as given" its own sentence. - SKILL.md Step 3 section: d1aedd09a3cc54f12763212381e7ef7b0731165303cf4b18ba6e1da2c9526533 -> 927479bb736f24a9e1c16aef9570c608b2529b2c2ab04682982c338833beccb6. The confirmation-gate whole-line pin passes; the problem-source paragraph, the Out-of-scope narrowing and the no-ledger ADR candidate sit beside it. - loop.md open-question register section: 272e5524066f4072445dcf55f67574eb3ed49ceeee681fcbcc71f86d075df83f -> 2245b2d67eb587d33f4a226a0eb0e1ffb26d56613619a8ce2459c68590cbc463. A departing answer still stays open until confirmed; the crux rule is added after the read-back rule. - context/coverage-prompt.md: 39485364e728eebe86218c577aefc3dc91d345bf118fe6d9656cf84961ff3f6a -> 74402dc04b272878c6351109cdfbafbf2ace9bcdcc976f53d1f2246986526c20. The register exemption text is unchanged; the unattended line is now given word for word. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
The eval-fix commit raised these caps from 12 to 20, but measured new-skill runs used 20-23 turns (fact-looked-up 23, stated-production 23, stated-prototype 21, bookmark-sync 20). 30 matches most interview cases; timeout 900 where it was 600. Both arms rerun on the same cap, and the PR reports measured turns: the redesign spends more tool calls per round. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
Conflict in plugins/planning/tests/interview-defenses.test.sh, the SKILL.md Stance digest pin: re-pinned over the merged section (this branch's edits plus main's #17872 prescribed-solution routing, already reviewed and pinned on main): 7624ab78474a662bce553d6918e1378709f2466d28ee6fdf18ec7db789e920b5 -> 3c1623c043f6d17ec50189dc8e3b296e8f5efcf5653414279a648f478030a215; the no-silent-resolve rule's text is unchanged. Owner approved re-pinning the interview pins 2026-11-21. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
Fixes for the f1-new eval diagnosis (diagnosis-f1.md). Every change adds
or clarifies a rule; none loosens a gate, a pinned defense or a grader
beyond accepting the output the skill prescribes.
Skill text:
1. context-unfamiliar-term-upstream: SKILL.md:90 reads question-shape.md
before the first question in any lane, this conversation included
(one home for the Context rules kept). question-shape.md:77-81: a
Context claim cites its own source, never another question
("(Q4 recommendation)"); an undecided answer is not a Context claim.
2. option-existing-control-drops: SKILL.md:239, every other
recommendation whose reason relies on a doc-only control carries the
same `documented, unverified` label.
3. recommendation-basis-and-flip and round-quality-standard:
SKILL.md:322, a fact a `Second view:` line raises goes on the
affected question's `Flips if:`. SKILL.md:183, the quality bar is
never settled by evidence. SKILL.md:243, the printed label covers
every fact the reason relies on, and a reason whose deciding step
rests only on a `judgment` line is researched or withheld; SKILL.md:211
(where the round file is written) checks that before the round is
checked and shown. question-shape.md:77-79: in a message that resumes
past round N, round N's answers cite `user, round N` and other facts
cite `user, round N+1`, matching the existing single-pacing PASS
sample.
4. scenario-walk-two-purchase-kinds: scenario-comparison.md:57-63, each
scenario is its own bullet under `Context:`, names its own person and
amount, day or date, and the walk states no candidate definition
first; the example at :71-80 shows two bulleted scenarios.
Graders (owner-approved corrections):
5. intent-asked-recorded: registers-intent-row word list gains
`variations?`; new PASS sample is the run's own row ("(variation
intent)"). loop.md:360: the row's `question` is the title as shown.
6. numbering-tool-display-order: ids-in-display-order counts a
`Commits you to:` part id (`- Q<N>:`) as the next id. Replayed on the
run's reply: no match (PASS); with the part renumbered Q11, or the
next question renumbered Q11, it matches (FAIL). New PASS sample with
a Q9 part between Q8 and Q10, new FAIL sample whose part skips Q9.
7. single-pacing-one-question: one-question-full-layout allows the
round title line, `Scope check:` and a `Later:` line naming topics
outside the five decisions (SKILL.md:177, :181, :185 prescribe them);
a `Later:` line naming another frontier decision still FAILs. New
PASS and FAIL samples.
8. surface-unset-offer-one-line: proceeds-to-round criterion 3 accepts
an intake round of discovery questions (lookups.md:34-37); PASS
sample replaced with the resolver's opening line and an intake round
(rethink-or-accept problem question, quality bar, alert audience).
Re-pinned digests of interview-defenses.test.sh (old -> new, why the
defense holds); every whole-line pin_exact still passes unchanged:
- SKILL.md Stance section:
3c1623c043f6d17ec50189dc8e3b296e8f5efcf5653414279a648f478030a215 ->
52c35fdc4ddfa2204c6617d02c92e58003228e18f3fa1e0efd4d442429c38f2c.
The "Partial-round resolution" no-silent-resolve rule is untouched;
the edits add the quality-bar, doc-only-control, judgment-line and
draft-check wording.
- loop.md open-question register section:
2245b2d67eb587d33f4a226a0eb0e1ffb26d56613619a8ce2459c68590cbc463 ->
d8a7c8107e6ef2b36a7420cf48d686c703f80db1d3f8035234d522ba19ed7a27.
Only the Fields line gains what the question field holds; statuses
and gap rules are unchanged.
- SKILL.md Action Router section:
da5d48aeabdf5a52a6827112557feba7fc4736a285c9b13a5565def7bd4de359 ->
0834710ccd7fc6f929fc206bab2abfdb1e65312b3d909704c8ccfd794593b83c.
The `lock` row's whole-line pin passes; only the question-shape.md
read now covers the inline lane too.
- SKILL.md Step 2 section:
4f1230d595f67a615b884320885dcc743dd42080822d3197a2f4a823ab7cb95d ->
eb231ceeea553fefdaae908e4367fb3f8cb1c3a0456d2a632310d23bcce158f5.
The constraint ledger, candidate, tripwire and sweep rules are
unchanged; the recommender paragraph adds the Second view to Flips if
route.
Checks: interview-defenses.test.sh PASS=185 FAIL=0; validate-cases.py
on plugins/planning/evals exits 0 with no FAIL (regex samples grade as
labeled). check-round-codes.sh rules unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
typos flags `labelled` on SKILL.md:239, the line the F1 fixes edited. Re-pinned digest of interview-defenses.test.sh (old -> new, why the defense holds): - SKILL.md Stance section: 52c35fdc4ddfa2204c6617d02c92e58003228e18f3fa1e0efd4d442429c38f2c -> 0470e7c5bc8c694eb8823c5e48b911dca3c44e33c9205aaebd1d74e2774dec0f. One word's spelling; no rule changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
Prose-only rule additions did not change the model's behavior, so four
rules it kept breaking in the F2 eval run now meet the round check that
runs before each drafted round is shown.
check-round-codes.sh:
- undefined=<line>:<term>: a `## Terminology matrix` term a question
uses that its own Context never mentions fails the round.
- --pacing single: a header line other than the round title,
`Scope check:`, `Later:`, `Held:` or `Settled by evidence:` fails as
extra=<line>:single-header.
- check=<line>:<name> self-checks print and never change the exit
code: own-context-terms (each question after the first),
flips-if-fact (each `Flips if:` line), existing-control (each
`My recommendation:` line), single-pacing (with --pacing single).
SKILL.md's round-check paragraph passes --pacing single, lists the new
keys, and sends a round only after each printed item is answered; the
single-pacing rule points to the check.
Graders: asks-second-variant also matches the yes-accepts intent
question ("Should this be built for Loqate ... only, with no layer for
swapping providers?"), with a PASS and a new FAIL sample;
proceeds-to-round says the prescribed `Pacing:` line is not a repeat
of the surface offer, with a PASS sample and a FAIL sample in which
another line repeats the offer.
Re-pin (interview-defenses.test.sh):
- "SKILL.md Stance section is unchanged (the in-round no-silent-resolve
rule lives here)": 0470e7c5bc8c694eb8823c5e48b911dca3c44e33c9205aaebd1d74e2774dec0f
-> b32afc007ad388e8aec7267ab8593bc5b2b2ceb1bf41e224b09ccbd2ed519e5f.
The edits add the round check's --pacing flag, its two new failures
and its self-checks to the round-check paragraph, and a pointer from
the single-pacing rule to it. The in-round no-silent-resolve rule
(Partial-round resolution) is untouched, and every edit adds a check;
none removes or qualifies one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
Merges main through f4da1df into the interview redesign. The planning skill changes on main's side are 41c004f (interview friction fixes, #17736); the four conflicts all come from it. Conflict resolutions: - scripts/check-open-questions.sh: took main's structure (header comment moved into the usage() heredoc, round_field_re accepting the page export's "round <N> (added in round <M>)"); re-applied this branch's declared-gap text inside the heredoc's Exit 2 list, and kept the recheck:: count and tags-only tail lines ahead of main's round-field match. The diff against main is exactly this branch's changes. - skills/interview/SKILL.md: took this branch's text in both hunks (the pacing paragraph and Step 2's re-ground paragraph). Main narrowed the per-round contradiction pass ("at most one contradiction pass"; a pass only when a round cites an answered row or a confirmed constraint); this branch removes that pass (PLAN.md AC10 and B7, Phase 2; eval interview-sweep-reports-contradiction). This is a modify/delete: main's added rule is the skip, and a pass that never runs satisfies it; only base behavior (the pass running on a citing round) is dropped, which neither side added. SKILL.md is byte-identical to HEAD. - skills/interview/context/assumption-sweep.md: kept both sides: this branch's "ask it for every finding" paragraph and main's "Follow-up sweep scope" paragraph (the When bullet points at it); main's /planning:design deferral sentence plus this branch's contradiction bullet naming each side's row by id and plain name; the "Pre-round contradiction pass" section stays deleted, per the SKILL.md item above. - tests/interview-defenses.test.sh: digest conflicts resolved to the digests of the merged text (re-pins below). SKILL.md Stance and Step 2 keep this branch's digests (b32afc00, eb231cee): SKILL.md equals HEAD. Re-pins (no comment lines added to the test file): - loop.md open-question register section: d8a7c8107e6ef2b36a7420cf48d686c703f80db1d3f8035234d522ba19ed7a27 -> f8873518e28409c2ec4a6c9c42bb3e3494b1f55b70daee2a23a971298b4135a3 (main had 9707355761f65850d77cd0d8ef274a75749784ada6857bdcb11d0cb72eeb70f8). The section now holds this branch's text plus main's two row-shape additions: the round field may read "round <N> (added in round <M>)" from the page export, and a re-ask-only round writes no row and takes no round number. Neither resolves a row; ask-time writes, the register gate, the unattended ladder and every pinned line (auto-guard, rungs 3 and 4, confirmation rung, lock STOP-on-gap) are byte-identical, and the test passes them. - context/assumption-sweep.md: 19473131cf19c03e6f3f17d95c9012b143b505435d0d0eb8a21239774b4b6c0a -> 7741202424622a250ffaf696fd0bad31877dc4ad425e03490942d0766d151478 (main had b07bfe7ec5d290450afcb13ae97dbd51c310f95fe92b2f075878dee35bc74171). The file is this branch's version plus main's follow-up sweep scope and the /planning:design deferral of a design-level decision item. The deferral writes a deferred register row the register gate counts and the --brief cross-check requires, so no item is settled silently; a decision item still always becomes a register row, so case A's open-decision plant and case B's resolved-but-one plant survive. The pre-round pass stays deleted. Known leftover: main's inherited re-pin comments in interview-defenses.test.sh (the #17736 lines above the Stance, loop.md register, Step 2 and assumption-sweep pins) describe the narrowed pre-round pass this merge removes. They were left as is because the comment lint rejects comment edits there. Verification: interview-defenses (185/185), reattach-slice (25/25), check-open-questions.test.sh, check-round-codes.test.sh and surface-checks.sh (167/167) pass. pytest on surface/test_exporters.py and test_round.py: 668 passed, 3 failed. The three test_round failures (a failed sync names the recovery; two wake ledger-check tests) fail the same way on HEAD's tree before this merge: the Windows temp path's short versus long name and backslashes lost on the path bash runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… to the skill text Each change fixes a cause found in the skill text for an eval case that failed on the new skill in every batch: - SKILL.md existing-control rule: the documented-unverified label now covers every Context line that states a control known only from a document, not just recommendations (interview-option-existing-control-drops). - SKILL.md flips-if-fact check: it also asks that Flips if carry each fact a Second view line names (interview-recommendation-basis-and-flip). - SKILL.md round check and "Recommended answers": a reason that states a fact no Context line sources is researched or withheld, like a judgment line; the alternatives template's source slot allows judgment (interview-round-quality-standard). - SKILL.md single pacing and loop.md Frontier rounds item 1: under single pacing the restate names only what is decided, and no text gives the reason one question goes first (interview-single-pacing-one-question). - covers-d grader (owner-approved): accepts "outside the ... request" and "once ... ready"; adds that wording as a PASS sample, FAIL samples kept. Re-pins (plugins/planning/tests/interview-defenses.test.sh): - SKILL.md Stance section: b32afc007ad388e8aec7267ab8593bc5b2b2ceb1bf41e224b09ccbd2ed519e5f -> fe6615facced60e6498a079c0693db45d1559bc02820c4aae2112a4e2fca8f03. Holds: the edits only tighten the label, sourcing and pacing rules; the in-round no-silent-resolve rule is untouched. - loop.md Step 2 section: f458232e95be64764519502cbb91bf973ce0cef5f8d1e7dbf96b1e4f87eebb6f -> 90734b5ba721a00e1fc2d12a01900c540721a83cd4b62e838b1eb5a3541efa88. Holds: only the restate's content under single pacing narrows; the constraint ledger rows and the process-change definition are untouched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
…save line The surfaces resolver prints "Questions are saved to <path>." right after the one-line offer (surfaces/registry.mjs), and the skill prints its line verbatim. The proceeds-to-round rubric predates that line, and its "no other line repeats or expands that offer" read it as part of the offer. Criterion 1 now says the save line is not part of the offer, and a PASS sample carries it; every FAIL sample is kept. Evidence: replay runs in which the real judge graded the saved F3 reply moved from FAIL 3 of 3 to FAIL PASS FAIL once the line was removed; it is the only trait the F1, F2 and F3 failures share. Owner-approved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
…cking The eval binary runs regex graders on JavaScriptCore. On an 8271-character reply the old pattern backtracked for about 10 seconds and JavaScriptCore returned no match where V8 matched, so a correct reply failed (F3). The rewrite keeps every piece of the old pattern's meaning and changes only how lines are matched: a referenced line is found by lookahead and consumed once, and the run of referenced lines is taken whole through a lookahead-capture, so nothing is retried. The new pattern under bun gives the old pattern's node verdict on all 222 saved replies and samples, 18 adversarial inputs and 60,000 fuzzed inputs, in under 1 ms each. Owner-approved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #17964
Held (
do-not-merge): the merge gate (plan Phase 9 step 8) failed; see Verification. The owner's decision for a failed pilot: "run the full set only if all pass, else stop with the PR held".Summary
Redesigns
/planning:interviewfrom the owner-locked Brief and the approved plan (published copy below):round.sh next-idand are never reused or renumbered; a gap passes the register gate only where a- gap |line declares it, and the exporter writes- gap | Q<a>-Q<b> | never assignedfor ids nothing ever held.round.sh reopen, its old answer kept under## Reopened answersand every dependent flaggedrecheck::at any depth; register rows carry a fixed-shape tag tail (kind::,problem::,depends::,reopened::,recheck::).round.sh showat the newquestion_detailsetting (standardby default); the round-file check fails a question with no answer line and a header with noScope check:line.interview-recommenderagent per round, the per-round contradiction pass removed (the end sweep reports every finding), terms agreed where they first appear.---rule between questions, and one home per rule for every item the consistency audit found stated two ways.Fix
Phases 2 to 8 of the plan, one or more commits each; every edit inside a digest-pinned region of
plugins/planning/tests/interview-defenses.test.shis re-pinned in its commit, whose message gives old digest, new digest and why the defense still holds. Owner decisions behind the changes a review would read as loosened checks:Verification
Merge gate: not met; the PR is held. Old skill = main's planning plugin with this branch's cases; new skill = this branch. One run per case (opus-5-5, sonnet judge), each results file checked with
run-validity.py --runs 1 --scaffolded; 198 of the owner's 300-run cap used. The plan's two-of-three rule was not run on the last head: the owner's stop rule for a failed pilot ended the work first. Each of the six cases below failed in every batch it ran in, across three skill versions (B2, F1, F2, F3; round-quality-standard F1 to F3). Not settled: about 12 kept cases failed on new in F1 (two skill changes back), about 9 of them have no old-skill result, so "no kept case regresses" was never decided; 3 cases voided by a revoked login have no result.0839b432a6ran only the 8 pilot cases.F1 (full batch,
101a801939): 73 planning cases plus 1 work-items case on new. Rewritten cases passing on new: 5 of 12. New cases passing: 5 of 7. Three results were voided by a revoked login mid-batch and two by sandbox denials (both rerun VALID and passing).Two fix rounds followed, each piloted on the failing cases before any full rerun:
c7d4b3b4660839b432a6Still failing on new after F3 (case: grader):
The stand-in judge found the model breaking rules the skill states plainly in four of these, for example a one-question round that opens "Of the five open decisions...". Two text rounds did not change that.
Grader corrections the owner approved (to accept output the skill prescribes, each keeping a FAIL sample): interview-intent-asked-recorded
registers-intent-rowandasks-second-variant, interview-numbering-tool-display-orderids-in-display-order, interview-single-pacing-one-questionone-question-full-layout, interview-surface-unset-offer-one-lineproceeds-to-round(twice). The owner asked for each to run on both arms; the old-skill runs were not done before the stop.Measures at
101a801939, new against old, over the 23 cases with both (F1): median turns +3, median duration -4 s, median cost +0.15 USD list per run (subscription usage, not a bill). New-skill median: 22 turns, 128 s, 1.04 USD.Old results reused rather than rerun, to stay inside the cap: 9 unchanged rewritten cases (an older main) and the B3 old runs for kept cases unchanged since.
Found while evaluating
validate-cases.pypassed twocase.yamlfiles the CLI's YAML parser refuses (an unquoted scalar holding:).node: the shim fails in the sandbox's throwaway home, so runs need the real node bin first onPATH.Related
Eval audit table (all 68 interview cases)
Every case tagged
interviewinplugins/planning/evals/(68), classified before any paid run.The per-case facts behind each verdict (quoted grader and sample text) were extracted read-only and
judged in the main session.
Decision codes in the reasons:
Discovery:line and no recommendation (AC8).question_detail: standardprints noOrigin:,Basis:orChecked against:line; the recommendation keeps its reason (AC5, AC6).Each rewrite moves the prompt past round 1 where INTAKE applies ("round 1 is answered: ..."), keeps
every existing grader's requirement, and gains one assertion only the new skill passes, so it can
fail on the old skill (AC2):
(?:^|\n)[^\n]{0,8}Scope check:. The old skill never prints this line (grep -c "Scope check" plugins/planning/skills/interview/SKILL.mdprints 0 at 7844889).match: not_contains,(?:^|\n)[^\n]{0,8}(?:Basis|Origin):. The old per-question layout printsBasis:on every recommended question.Every rewrite lists
BashandAgentin itsallowed_tools, and Phase 9 passes--allow-tools Bash(
Agentis read-only and needs no grant;plugins/evals/skills/plugin-eval/SKILL.md:108-113), soround.shand the recommender can run. The new skill text still writes theScope check:lineand the
standardprint by the same rules whenround.shcannot run.brainstorm-spec-stage-no-candidatesinterview-adr-verdict-overruledinterview-bare-vague-build-asksinterview-bare-vague-explore-no-extrainterview-brief-keeps-adr-verdictsinterview-closing-probe-something-elseinterview-coded-terms-round-answerableinterview-consumer-standard-overrideinterview-context-unfamiliar-term-upstreammeround 1 asked for decision questions; the Context regex must also end a block at the newDiscovery:,Held:,Second view:,Departs from:,Scope check:and---linesinterview-coverage-prompt-askedautowith goal and criteria given; grades the pre-Brief coverage prompt, not a roundinterview-coverage-unattended-unexaminedinterview-crux-on-rejectioninterview-discuss-sets-discussingdiscussingstatus; the optional re-ask's "basis" is met by the reason; risk DETAIL notedinterview-document-dba-replica### Q<N>:id passesinterview-ears-convention-tagsinterview-ears-convention-tags-criteriainterview-fact-looked-up-not-askedinterview-final-brief-changes-onlyinterview-final-confirm-active-replyinterview-goal-keeps-user-whyinterview-handoff-no-design-questioninterview-handoff-small-contract-is-planinterview-handoff-unsure-midflowinterview-idea-routes-brainstorminterview-intent-asked-recordedinterview-intent-prototype-one-lineQ1question and no quality-requirement question; no recommendation neededinterview-intent-unattended-blockedinterview-later-line-known-roundsLater:line, which every round keepsinterview-lock-restates-problemlockproblem confirmation; untouchedinterview-negative-recommendation-yes-acceptsinterview-nfr-quality-requirementinterview-option-existing-control-dropsinterview-option-known-repetition-keepsinterview-owner-deferred-documentinterview-pacing-round-armquestion_detaillevels); its graders are frozen by its own rule, so they cannot follow the new layout; recorded as a dated amendment ininterview-pacing.mdinterview-pacing-single-arminterview-pacing-round-arminterview-pacing-single-arm-open-binterview-pacing-round-arminterview-pacing-single-arm-open-dinterview-pacing-round-arminterview-pacing-single-arm-step-2interview-pacing-round-arminterview-pacing-single-arm-step-3interview-pacing-round-arminterview-person-deferred-documentinterview-prd-seeds-briefinterview-prescribed-solution-rethinkinterview-prototype-split-by-questioninterview-quality-bar-asked-bookmark-syncinterview-quality-bar-asked-ticket-dashboardinterview-quality-bar-stated-productioninterview-quality-bar-stated-prototypeinterview-readback-dictated-answerinterview-recommendation-basis-and-flipinterview-reported-problem-instance-questioninterview-round-topic-groupsBasis:line" per question; reworded to "a recommendation with its reason (its basis)"interview-round1-unknowns-offers-wayfindinterview-scenario-walk-two-purchase-kindsinterview-scope-cut-moscowinterview-single-pacing-one-questionBasis:line"; reworded to its reasoninterview-skips-quick-behavior-changeinterview-skips-quick-change-typointerview-stated-problem-no-rethinkinterview-surface-unset-offer-one-lineinterview-term-clash-qualified-nameinterview-tiny-diff-not-cheap-to-retryinterview-unattended-default-finish-lineinterview-unattended-needs-human-stopsinterview-unattended-no-plugin-offerinterview-unattended-problem-blockedinterview-unknown-field-candidatesinterview-wayfind-size-suggestionCoverage kept for the retired pacing group: a new case
interview-round-quality-standardreusesthe round-arm prompt (round 2 of the invoice CSV export, four planted decisions) with graders for
the new
standardlayout: every planted decision asked; each question with aContext:line citinga stated fact, a recommendation whose printed reason names its evidence (or
Withheld:), andalternatives;
---between questions; aScope check:header line (new-only). It is not part of thefrozen experiment, so the freeze holds.
Totals: 50 keep, 12 rewrite, 6 retire. No case grades the removed contradiction pass (NOPASS); its
eval evidence is the new case
interview-sweep-reports-contradiction.Risks carried into Phase 9 (each a kept case whose judge reads "basis" against the printed reason):
interview-coded-terms-round-answerable,interview-discuss-sets-discussing. Requirements the newskill text must meet so kept cases hold (Phases 7 and 8): every round header keeps
Later:and theclosing probe; a recommendation's printed reason carries its evidence and source at every level;
routing suggestions (brainstorm, wayfind) are not intake questions and still appear in round 1; a
looked-up candidate for a problem field is stated with its source and a confirm request; the
glossary loads only once an interview starts, after the quick-change skip; with no
Agenttool themain session drafts the recommendation, and with no
Bashthe next id is read from the ledger(no case allows
Agent; 10 allow noBash).Second table: legacy cases and cases outside the tag
Legacy
plugins/planning/skills/interview/evals/evals.json(51 entries, a rubric with no graders, sono paid run). Case digests pin ids 1, 2, 3, 8, 12, 13, 15, 16 and 25 (
interview-defenses.test.shlines 684, 884, 922-949; the header comment says eight and is corrected in Phase 2); the roster pin
(line 913) covers all 51.
relentless-me-mode-frontier-roundsdependent-questions-split-across-roundsengineering-term-resolution-delegatesfacts-looked-up-decisions-askedmechanism-tripwire-asks-now-or-at-planninginherited-constraint-asked-not-assumedChecked against:is in the saved full block, not thestandardprintout-of-band-return-supersedes-before-the-replyresumed-recommendation-re-grounded-against-a-later-constraintChecked against:line is in the saved blockself-answer-from-mcp-and-research-and-offer-to-connectBasis:linepre-round-contradiction-passlate-return-restates-every-open-question-in-fullheld-question-release-restates-every-open-question-in-fulldocument-self-answerable-stays-a-roundNo pinned case is retired, so the plan's user-approval gate for retiring a pinned case does not come due.
Cases outside the tag whose inputs a decision changes, run once on the new skill in Phase 9:
work-items/evals/decompose-decisions-no-coined-idsQ2 | answered | ...); the tag tail must not change what it readsplanning/evals/wayfind-chart-seeds-from-interview-ledger## Problemsand## Reopened answersplanning/evals/wayfind-purpose-settled-facts-recordedplanning/evals/wayfind-ready-questions-asked-in-one-round-and-items-are-topicsplanning/evals/plan-close-out-posts-adr-verdictsQ1-Q4idsRubric-only entries outside the tag edited in Phase 6 or 7 where an expectation becomes false:
audit-answers
evals.jsonid 1 (accepts "each open question's recommended answer"; a discovery rowhas none, so
/planning:audit-answersmust skipkind:: discoveryrows).Cost estimate (no paid run before this)
Runs, one per case and one arm (
--ablation none), perplugins/evals/skills/plugin-eval/SKILL.md"Cost":Price basis: the suite's last recorded run, 14.47 USD list price for 36 runs on
claude-opus-5-5(
plugins/planning/evals/interview-pacing.md:178-179), about 0.40 USD a run with regex gradersonly. Most cases here add an llm judge (three calls a run), so 87 runs is roughly 35 to 55 USD list
price, and the full 109 roughly 45 to 70 USD. On this plan that is subscription usage, not a bill.
The tokens of the first 2 runs in Phase 9 resize it before any batch.
🤖 Generated with Claude Code
https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh