From 1a56d59350800e91349a0ef8c813e238619cae20 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 21 Nov 2026 02:21:58 -0500 Subject: [PATCH 01/30] refactor(planning): remove the interview's per-round contradiction pass The interview no longer dispatches a sub-agent before each round to check the draft against earlier answers. The end sweep keeps its Contradictions category and reconcile rule, and the answer-time conflict rule still catches a reply that conflicts with an earlier answer. Brief AC10: "the per-round contradiction pass is gone". - SKILL.md: drop the Step 2 paragraph and the pacing clause that named the pass; the sweep paragraph's pointer names the sweep-round label, whose rule lives in context/assumption-sweep.md. - context/assumption-sweep.md: drop "Pre-round contradiction pass". - context/product.md: the pacing note names only the sweep. - evals.json: remove case 36 pre-round-contradiction-pass, which graded the removed pass. Its replacement evidence is the new eval case interview-sweep-reports-contradiction (added later on this branch). - docs/contracts/verification-level: the interview's exemption names the sweep and the quote check only. Re-pins (interview-defenses.test.sh), old -> new, and why each defense still holds: - Stance: 53ff69a1cc4af1af5f7df38a03b272977431a6e8374c52df8f9ca044e70e7ef6 -> e059b8a9bdadf3a44593745098d9365d46b80fe2e2e9bdb3d449ab124d0481b0. Only the pacing clause naming the pass changed; the no-silent-resolve rule is byte-identical. - Step 2: 91e71ea48b46fdefdb4fa77f32cd6d48b47245f04c0c1bbdcf4884427211fff4 -> f332b13eda79cde79825d912827e70b18971870c60accde42f0ff5c79ca3bcea. The pass's paragraph is gone and the sweep paragraph points at its spoke; the constraint ledger, candidates, currency, tripwire and sweep rules are unchanged, and no row is resolved for the user. - context/assumption-sweep.md: 2054938d89851cd80602ba2d80aed3bd49722420cd836155c9ccdfe8e0f47e2a -> 4d1a6e8c8789645fb58bbae26822fc34d2fdcefda674e0d7210247dcccc8f0f3. Only the per-round section is deleted; scope, item shape and every disposition, USER-RESERVED included, are unchanged. - Case roster: 9f9747c65eb92352e3b940f53808bac632a4f2691a95dfd08ac74a152818d713 -> 7bd78ae7a7bc7a32988b8a7b526a189373ac3fa961f79f7aa56c826a280d588b. Only case 36 left the roster; cases 15 and 16 and every digested case are unchanged, and no case added contradicts them. The three test comments that described the pass are deleted; the reasons above replace per-pin comments, which the comment lint refuses. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh --- .../feat-interview-redesign-07635b38.md | 7 ++++++ docs/contracts/verification-level/README.md | 4 ++-- plugins/planning/skills/interview/SKILL.md | 6 ++--- .../interview/context/assumption-sweep.md | 23 ------------------- .../skills/interview/context/product.md | 4 ++-- .../skills/interview/evals/evals.json | 14 ----------- .../planning/tests/interview-defenses.test.sh | 11 ++++----- 7 files changed, 17 insertions(+), 52 deletions(-) create mode 100644 .changes/planning/feat-interview-redesign-07635b38.md diff --git a/.changes/planning/feat-interview-redesign-07635b38.md b/.changes/planning/feat-interview-redesign-07635b38.md new file mode 100644 index 00000000000..4d27302574d --- /dev/null +++ b/.changes/planning/feat-interview-redesign-07635b38.md @@ -0,0 +1,7 @@ +--- +bump: minor +--- + +### Removed + +- **The interview's per-round contradiction pass.** `/planning:interview` no longer dispatches a sub-agent before each round to check the draft against earlier answers. The end sweep still finds contradictions and asks each as one question naming both sides, and an answer that conflicts with an earlier one is still caught when it is given. The legacy eval case `pre-round-contradiction-pass` is removed with it. diff --git a/docs/contracts/verification-level/README.md b/docs/contracts/verification-level/README.md index 23304d8b8be..371a4200ca9 100644 --- a/docs/contracts/verification-level/README.md +++ b/docs/contracts/verification-level/README.md @@ -77,8 +77,8 @@ deliverable is not governed. These skills do not read the level: `/planning:audi `/performance:verify`, `/discipline:scrutinize-dont-coast`, `/review:quality-gate`, `/review:fanout`, `/planning:devils-advocate`, `/review:security-review` and `/planning:interview`. Each is itself the independent check when invoked, except -`/planning:interview`, whose contradiction pass, assumption sweep and quote check produce the -questions it exists to ask. +`/planning:interview`, whose assumption sweep and quote check produce the questions it exists to +ask. `/coupling:reduce` and the conventions lens of `/architecture:improve` do not read it either: reproducing a scan finding produces the finding they apply or propose for recording, so it runs on diff --git a/plugins/planning/skills/interview/SKILL.md b/plugins/planning/skills/interview/SKILL.md index d46927324e6..78cd3ae8e9a 100644 --- a/plugins/planning/skills/interview/SKILL.md +++ b/plugins/planning/skills/interview/SKILL.md @@ -171,7 +171,7 @@ Alternatives: **Name the evidence-settled decisions in the round header.** A decision whose evidence leaves only one viable answer gets no question. The round header lists each on one `Settled by evidence:` line, by id and plain name, with the answer and its Basis: `Settled by evidence: Q7 (log format), JSON lines, Basis: verified, src/log.ts:12`. Each takes its `Q` and an `answered` register row with the resolution `answered-from-evidence: . Basis: `. The user can reopen one by id: its row goes back to `open` and it is asked in full in the next round. Any decision with a live alternative is still asked. -**One question at a time (`question_pacing`).** Pacing: resolve `question_pacing` once, before the first round or the page start, per [`${CLAUDE_PLUGIN_ROOT}/reference/config.md`](../../reference/config.md) "Convention-home keys", with `${user_config.question_pacing}` as its user layer (unexpanded or empty means unset; the default is `round`), and report the value and its layer in one line. `auto` settles to `round` or `single` after the Step 1 survey and before the first round or the page start, and everything after reads that choice: `single` when the topic is broad and open-ended (many independent directions, little settled upstream, no spec or earlier answer to anchor on), `round` otherwise. State the choice and how to switch in one line ("Asking one at a time: this topic is wide open. Say 'give me the whole round' to switch."). `round` asks the frontier as one set. `single` computes the frontier the same way, then asks one question per message in blast-radius order: each message carries one question in the full layout, the `Held:` line when one applies, and the closing probe, with no topic header. One frontier is still one round: one round number, one `round-.md` file and one contradiction pass, sent over several messages. A round fired from inside another workflow's phase ignores `single` ("Rounds fire at phase boundaries"). Register each question `open` in the message that asks it, and recompute the frontier after each answer, since an answer can prune or add questions. The user's request in the session ("one at a time", "give me the whole round") overrides it for that session. The page honors `single` too: the session posts the round with its first question and adds each later one after the one before it is answered ([`context/lane-page.md`](context/lane-page.md)). +**One question at a time (`question_pacing`).** Pacing: resolve `question_pacing` once, before the first round or the page start, per [`${CLAUDE_PLUGIN_ROOT}/reference/config.md`](../../reference/config.md) "Convention-home keys", with `${user_config.question_pacing}` as its user layer (unexpanded or empty means unset; the default is `round`), and report the value and its layer in one line. `auto` settles to `round` or `single` after the Step 1 survey and before the first round or the page start, and everything after reads that choice: `single` when the topic is broad and open-ended (many independent directions, little settled upstream, no spec or earlier answer to anchor on), `round` otherwise. State the choice and how to switch in one line ("Asking one at a time: this topic is wide open. Say 'give me the whole round' to switch."). `round` asks the frontier as one set. `single` computes the frontier the same way, then asks one question per message in blast-radius order: each message carries one question in the full layout, the `Held:` line when one applies, and the closing probe, with no topic header. One frontier is still one round: one round number and one `round-.md` file, sent over several messages. A round fired from inside another workflow's phase ignores `single` ("Rounds fire at phase boundaries"). Register each question `open` in the message that asks it, and recompute the frontier after each answer, since an answer can prune or add questions. The user's request in the session ("one at a time", "give me the whole round") overrides it for that session. The page honors `single` too: the session posts the round with its first question and adds each later one after the one before it is answered ([`context/lane-page.md`](context/lane-page.md)). `Q` numbering runs continuously across rounds (Q1…Q4 in round one, Q5… in round two; visible depth). Close the round with one probe inviting a constraint that breaks the recommendations, worded "something else", never "anything else": "Is there something else that would change these recommendations?" (eval case `interview-closing-probe-something-else`). Put the note that the user may answer in any order before the probe. In the terminal lane an inline round is the last message of the turn that sends it and the probe its last line, because the user answers what the turn ends on: write its round file, run its checks and register its rows before it is sent (under `single`, each row in the message that asks it), and call no tool after it. A correction that changes a question already shown resends the whole round with the note in its header (under `single`, only that question, the note above it); any other note waits for the top of the next round. Wait for the answers before computing the next round. The recommendation+basis discipline holds per question. Inline prose carries the recommendation, its reasoning, AND the probe in one readable block; a card cannot. @@ -309,15 +309,13 @@ When the resolved lane's spoke shows Step 3's confirmation gate outside this con **Re-ground open recommendations before they are shown.** On resume, and before a queued round is shown, check each open recommendation: its `Basis:` cites a source read this session or a constraint row, and it does not rest on an artifact the constraint ledger now excludes. A consequential one that fails gets a research dispatch before the user sees it, or is shown marked "not yet researched". When the user adds a constraint mid-interview, write the `confirmed` row at once and re-check every open recommendation's `Checked against:` line against it; revise the ones it changes and say which. -**Check each draft round against earlier answers before it is shown.** Every round after the first, before it is asked inline or posted with `add-round`, gets a contradiction pass: a fresh-context non-fork sub-agent runs only the assumption sweep's contradiction category over the draft round and the register's `answered` rows ([`context/assumption-sweep.md`](context/assumption-sweep.md) "Pre-round contradiction pass"). Fix each contradiction in the draft before the round goes out: revise the recommendation so it holds with the earlier answer or constraint, or, for an earlier answer, ask one question that names both sides. A question that weighs earlier answers, whether it comes from this pass or the sweep, lists their `Q` ids as the answers it reconciles; a round of sweep items is labeled a sweep round, per that file's "Disposition". - **Composed artifacts are candidates.** A design from explore or research output, or read from an existing implementation, is a candidate, never a recommendation, until re-derived against the constraint ledger: check it against every row, recommend only what survives, and name the rows on its `Checked against:` line. A research or exploration brief dispatched mid-interview carries the constraint ledger verbatim. **Label evidence by when it was true.** When a recommendation rests on historical evidence (past metrics, merged PRs, an existing implementation) and the survey finds a recent process change or cannot tell, label each finding `past`, `current`, or `intended` before a recommendation rests on it. A process change is one the survey finds (a changelog, commit, ticket, or user statement dated after the evidence window) or cannot rule out. State or ask any currency you cannot establish; past-only evidence never alone grounds a recommendation. **What, not how: the mechanism tripwire.** "Does not plan implementation" has a trigger. When a question or its recommendation names a mechanism (a tool, an API, a credential or token scope, a runtime, a platform per step), ask once whether the user wants mechanism-level decisions settled now or at planning. Ask it in the same round as the first mechanism-naming question and register it like any other question. Its Context names the question that named the mechanism, and its `Basis:` cites the fact behind that mechanism in any form "Recommended answers" allows (`Basis: verified, user, round 1: deploys already run through one CI pipeline`); neither cites this rule or this skill, which the user cannot check. With no such fact, ask it withheld. Until it is answered, recommendations carry no mechanism part. When the answer is "at planning", no later recommendation fixes a mechanism, and every still-`open` mechanism row, commitment rows included, becomes `deferred`: carried to the Brief's `### Deferred questions` under the arbiter rule in an engineering session, listed as deferred in the shared-understanding summary in a general one. When the answer is "now", write `C | confirmed | mechanism decisions settled in this interview | user, round ` at once, and the Brief's Constraints records that this topic overrides "does not plan implementation". -**Sweep for assumptions before Step 3.** In `me` and `auto` with a register, including the `auto` Mixed path after its residue round, the loop ends with a fresh-context assumption sweep. Once no open question remains on the frontier, and before the Step 3 register gate and confirmation gate, dispatch a non-fork sub-agent to inventory undecided details, hidden defaults, contradictions, and hedged or free-text rows across the ledger, the recommendations, and any composed artifacts. You turn each item into a register row at the next contiguous `Q`, a stated fact, or, for hedged rows, a line in the confirmation restate. Any `open` row it adds returns the run to Step 2, and the next sweep covers only what the new answers changed; Step 3 is reached when a sweep adds no `open` row. A run that asked no question skips it, and so does `lock`; unattended, an item that is the user's decision is `blocked`. The acceptance-criteria coverage prompt is never a sweep item ([`context/coverage-prompt.md`](context/coverage-prompt.md)). Scope, item shape, and disposition: [`context/assumption-sweep.md`](context/assumption-sweep.md). +**Sweep for assumptions before Step 3.** In `me` and `auto` with a register, including the `auto` Mixed path after its residue round, the loop ends with a fresh-context assumption sweep. Once no open question remains on the frontier, and before the Step 3 register gate and confirmation gate, dispatch a non-fork sub-agent to inventory undecided details, hidden defaults, contradictions, and hedged or free-text rows across the ledger, the recommendations, and any composed artifacts. You turn each item into a register row at the next contiguous `Q`, a stated fact, or, for hedged rows, a line in the confirmation restate. Any `open` row it adds returns the run to Step 2, and the next sweep covers only what the new answers changed; Step 3 is reached when a sweep adds no `open` row. A run that asked no question skips it, and so does `lock`; unattended, an item that is the user's decision is `blocked`. The acceptance-criteria coverage prompt is never a sweep item ([`context/coverage-prompt.md`](context/coverage-prompt.md)). Scope, item shape, disposition and the sweep-round label: [`context/assumption-sweep.md`](context/assumption-sweep.md). ### Step 3. Recognize the stop condition diff --git a/plugins/planning/skills/interview/context/assumption-sweep.md b/plugins/planning/skills/interview/context/assumption-sweep.md index 4a91b644e2b..9884f7eff1c 100644 --- a/plugins/planning/skills/interview/context/assumption-sweep.md +++ b/plugins/planning/skills/interview/context/assumption-sweep.md @@ -144,26 +144,3 @@ The round that asks these rows is a **sweep round**. On the page, post it with answers` and shows each listed answer beside the question ([`surface.md`](surface.md) "A round"). In the terminal lane, open the round with that label and quote each listed earlier answer under its question. - -## Pre-round contradiction pass - -Before every round after the first is asked inline or posted with `add-round`, check the draft -round against what the user already answered, so a conflict is caught before the user sees it -rather than at the next sweep. - -- **Dispatch:** a fresh-context (non-fork) sub-agent, as in "Dispatch" above. Hand it the draft - round (each question with its recommendation, `Checked against:` line and `Commits you to:` - parts) and every `answered` register row with its resolution, plus the `confirmed` constraint - rows. Withhold the rationale, as above. -- **Scope:** only the **Contradictions** category of "What it looks for", read with the draft - round's recommendations standing in for answers: a recommendation that cannot hold beside an - `answered` row or a `confirmed` constraint. No other category runs here; the sweep still runs - once no open question remains on the frontier. -- **Return:** one line per conflict in the "Item shape" above, category `contradiction`, with the - draft question and the earlier row as its dependencies; nothing when there is none. -- **Disposition:** fix each in the draft before posting. A conflict with a `confirmed` constraint - is fixed by revising the recommendation so it holds, citing the constraint on its - `Checked against:` line. A conflict with an answered row is fixed the same way, or by replacing - the question with one that names both sides and lists that row's `Q` in `reconciles` (only - question ids go there). A draft conflict never becomes a register row of its own; the revised - question takes its place. diff --git a/plugins/planning/skills/interview/context/product.md b/plugins/planning/skills/interview/context/product.md index e9ea8319141..9e5ddf35842 100644 --- a/plugins/planning/skills/interview/context/product.md +++ b/plugins/planning/skills/interview/context/product.md @@ -16,8 +16,8 @@ only: question, and the early session-config recommendation do not run: the Reason to exist principles ask for the problem, and a session-config recommendation, when one fits, comes after the verdict. -2. **Pacing.** One question per turn, not a frontier round. The pre-round contradiction pass and - the assumption sweep do not run; pushing on each answer before moving on does that work here. +2. **Pacing.** One question per turn, not a frontier round. The assumption sweep does not run; + pushing on each answer before moving on does that work here. 3. **Recommendation.** Every question carries what a strong answer sounds like, with a `Basis:` line citing the principle it comes from (see "Question shape"). A strong answer is a standard for the user's own answer, not words put in their mouth, so the withheld-recommendation rule for diff --git a/plugins/planning/skills/interview/evals/evals.json b/plugins/planning/skills/interview/evals/evals.json index e84253431de..0cfe4222a8a 100644 --- a/plugins/planning/skills/interview/evals/evals.json +++ b/plugins/planning/skills/interview/evals/evals.json @@ -502,20 +502,6 @@ "The handoff message lists Q3 and Q4 by id and question and does not route to /planning:plan" ] }, - { - "id": 36, - "name": "pre-round-contradiction-pass", - "prompt": "/planning:interview me — add nightly exports of the billing ledger to the analytics warehouse.\n\n[Session state: an engineering session in the terminal lane. Round 1 is answered: Q1 (export window) `answered` as \"only between 01:00 and 03:00 UTC\", Q2 (who may read the export) `answered` as \"finance only\". The constraint ledger holds C1 `confirmed`: no export may include card numbers. The model has drafted round 2: Q3 recommends a full-table export that includes the `card_number` column, and Q4 recommends running the export at 04:00 UTC after the backup job. The frontier is not empty, and round 2 has not been shown.]", - "expected_output": "Before round 2 is shown, the skill dispatches a fresh-context (non-fork) sub-agent with the draft round and the answered rows and confirmed constraints, and has it check only for contradictions, withholding the rationale. The pass finds that Q3 contradicts C1 and Q4 contradicts Q1's answer. The skill fixes the draft before showing it: Q3's recommendation drops the card column, and Q4 either moves into the 01:00 to 03:00 window or is asked as one question that names both sides and cites Q1 as the earlier answer it reconciles. Round 2 is then shown with no recommendation that contradicts an earlier answer or a confirmed constraint. The full assumption sweep does not run here, since the frontier is not empty.", - "files": [], - "expectations": [ - "A fresh-context (non-fork) sub-agent checks the draft round 2 against the answered rows and confirmed constraints before round 2 is shown", - "The pre-round pass looks only for contradictions; it does not run the full assumption sweep's other categories", - "Round 2 as shown has no recommendation that includes the card number column, which C1 rules out", - "Round 2 as shown does not recommend a 04:00 UTC run as settled: Q4 is revised into Q1's window or asked as one question naming both sides that cites Q1", - "No conflict found by the pass is registered as a register row of its own before round 2 is asked" - ] - }, { "id": 37, "name": "sweep-flags-coined-term-after-accept-all", diff --git a/plugins/planning/tests/interview-defenses.test.sh b/plugins/planning/tests/interview-defenses.test.sh index 0bf05101bd1..430b4a8448b 100755 --- a/plugins/planning/tests/interview-defenses.test.sh +++ b/plugins/planning/tests/interview-defenses.test.sh @@ -532,7 +532,7 @@ pin_section "SKILL.md Stance section is unchanged (the in-round no-silent-resolv "$SKILL" \ "## Stance: supportive, depth-first, opinionated" \ "## The interview loop" \ - "53ff69a1cc4af1af5f7df38a03b272977431a6e8374c52df8f9ca044e70e7ef6" + "e059b8a9bdadf3a44593745098d9365d46b80fe2e2e9bdb3d449ab124d0481b0" pin_section "SKILL.md interview-loop preamble is unchanged (it governs every step below it)" \ "$SKILL" \ "## The interview loop" \ @@ -766,7 +766,6 @@ pin_section "SKILL.md \"does NOT do\" section is unchanged (the fudge prohibitio # the user's `confirm-understanding` event); that moves where the gate is shown, not whether # it runs, and the sweep still runs before it. # The re-grounding paragraph checks open recommendations on resume, before a queued round and on a new constraint; it adds a check and resolves no row. -# The pre-round contradiction pass revises a draft recommendation or asks one question naming both sides before a round is shown; it resolves no row and qualifies neither STOP-on-gap nor the auto-guard. # EARS and coverage-prompt rule text moved to context/ears.md and context/coverage-prompt.md, both digested whole (#10703); each former site keeps a one-sentence pointer, the coverage prompt still exempts only itself, and the auto-guard, STOP-on-gap and no-silent-resolve lines are byte-identical # Reworded phrases shared with mattpocock grilling (#9790): Stance names the wayfind trigger as an effort that will outrun a single session, Step 2 and the sweep run once no open question remains on the frontier, the coverage-prompt exemption still covers no other question, and the confirmation gate opens with the stop wording; every rule and its halt are unchanged # Question shape (#7475): every question opens with a referenced Context block, alternatives carry gain and cost, open questions are restated in full, a terminology matrix is agreed as a registered question, and codes show with their plain names (Checked against names constraints in words); each adds a requirement or a registered question, resolves no row, and the no-silent-resolve rule, the constraint-ledger rows and their re-check, and the late-commitment numbering are unchanged @@ -777,7 +776,7 @@ pin_section "SKILL.md Step 2 section is unchanged (constraint ledger, candidates "$SKILL" \ "### Step 2. Drive the frontier-rounds loop" \ "### Step 3. Recognize the stop condition" \ - "91e71ea48b46fdefdb4fa77f32cd6d48b47245f04c0c1bbdcf4884427211fff4" + "f332b13eda79cde79825d912827e70b18971870c60accde42f0ff5c79ca3bcea" # Frontier-rounds step 4 now requires a `Basis:` line and limits convention-only recommending # to non-consequential questions; an unsettled consequential one is asked open, withheld. It adds # a requirement and resolves nothing new. @@ -831,7 +830,6 @@ pin_section "loop.md relentless mechanics section is unchanged (late commitment "## Relentless \`me\` mode mechanics" \ "## The open-question register" \ "493b9e062228d39672b85ea3256e8a3c360fce9df183d08de9ea081666e20076" -# The sweep-round labeling and the appended pre-round contradiction pass leave the scope, item shape and USER-RESERVED disposition intact; a draft conflict becomes no register row and nothing is captured as an assumption. # Re-pinned for #8314: adds the coined-term category, item-shape value, disposition and a glossary dispatch input; every existing category and disposition is unchanged, so the open-decision (case A) and resolved-but-one (case B) plants survive. # EARS and coverage-prompt rule text moved to context/ears.md and context/coverage-prompt.md, both digested whole (#10703); each former site keeps a one-sentence pointer, the coverage prompt still exempts only itself, and the auto-guard, STOP-on-gap and no-silent-resolve lines are byte-identical # Re-pin for #8293: adds the quoted-source rule and fresh-context quote check for fact-settled items, a `decision` line that is never settled from facts, and the `## Assumption sweep` record the gate grades; every category, item-shape field and existing disposition is unchanged, and a decision still becomes a register row, so cases A and B survive. @@ -847,7 +845,7 @@ pin_section "loop.md relentless mechanics section is unchanged (late commitment # Row-shape bookkeeping (https://github.com/melodic-software/claude-code-plugins/issues/17279) only adds row-shape guidance to the register section and a note that sweep S ids restart per sweep; ask-time writes, the gate, the unattended ladder and the sweep's scope, item shape and dispositions are unchanged pin_file "context/assumption-sweep.md is unchanged (the sweep's scope, item shape, and disposition)" \ "$SWEEP" \ - "2054938d89851cd80602ba2d80aed3bd49722420cd836155c9ccdfe8e0f47e2a" + "4d1a6e8c8789645fb58bbae26822fc34d2fdcefda674e0d7210247dcccc8f0f3" # The coverage prompt's exemption from the register, the gate and the unattended ladder moved # here from SKILL.md "Acceptance-criteria capture"; it exempts that one prompt, and every other # question still writes its row at ask-time and brings the gate into scope. @@ -898,7 +896,6 @@ pin_case_digest "eval case B is unchanged (no criterion added that licenses the # Case 31 (ship-it-drops-questions-not-gates) moves the open rows to deferred as `USER-RESERVED` on the user's own ship-it, never answered with an invented answer, and keeps the confirmation gate, so it agrees with cases 15 and 16. # Cases 32-34 (wayfind-trigger-needs-two-rounds-running, declined-wayfind-not-repeated-until-escalation, never-switches-to-wayfind-on-its-own) grade only when wayfind is recommended and that the switch waits for the user; they resolve no row, so they agree with cases 15 and 16. # Case 35 (ship-it-confirmed-writes-deferred-brief) starts after the user confirmed and grades that the Brief keeps the deferred rows `USER-RESERVED` with no invented answer and that the handoff waits on the cross-check, so it agrees with cases 15 and 16. -# Case 36 (pre-round-contradiction-pass) has the draft fixed or asked as one question naming both sides before the round is shown and registers no row, so it agrees with cases 15 and 16. # Re-pinned for #8314: adds case 37 sweep-flags-coined-term-after-accept-all, which requires a coined-term register row and blocks the lock while it is open; it licenses no silent capture or fudged gap, so cases 15 and 16 still hold. # Case 38 (person-deferred-routes-to-document) defers the DBA retention window with an arbiter tag, resolves no row on the DBA behalf and only recommends the interview's document surface, so it agrees with cases 15 and 16. # Cases 46-50 move the document cases into this skill: each interviews only the send, writes the questions to a document under open register rows, and resolves none on the recipient's behalf; case 51 resolves a returned row only when its answer takes the recommendation, and reads back any other, which stays open until the user confirms it, so cases 15 and 16 hold. @@ -911,7 +908,7 @@ pin_case_digest "eval case B is unchanged (no criterion added that licenses the # Only the ADR handoff changed: the Domain-aware ADR bullet now points at round step 8, loop.md adds item 8 Decision record (sweep becomes 9), and eval 18 is renamed to the ADR-candidate fallback. No line about silent resolution, gaps, constraint rows or process changes was touched, and cases 15 and 16 are unchanged. # Document surface folded into the interview: the frontmatter adds the document trigger phrases and the `document` argument word; the Action Router adds one row that resolves a named surface and then runs `auto`, leaving the `lock` row and its STOP-on-gap reading untouched; the roster renames case 38 and adds cases 46-51, which register the recipient questions open and resolve none on anyone behalf, so neither defense is weakened pin_case_set "the eval-case roster is unchanged (no sibling case added that contradicts 15 or 16)" \ - "9f9747c65eb92352e3b940f53808bac632a4f2691a95dfd08ac74a152818d713" + "7bd78ae7a7bc7a32988b8a7b526a189373ac3fa961f79f7aa56c826a280d588b" # The roster pin catches a case ADDED. It cannot see an existing sibling REWRITTEN in # place: case 3 kept its name `lock-mode-does-not-fudge-gap` while its body was rewritten From 146fa27a8dc117ea88587cc30936b499931a6fa6 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 21 Nov 2026 03:11:21 -0500 Subject: [PATCH 02/30] feat(planning): stable question ids with next-id and declared gaps Question numbers are now handed out and kept, never reused or renumbered (issue #7541, owner ruling "Keep ids (do not refuse)"). - round.sh next-id --ledger [--dir ] [--count ] prints the next n ids: the highest number the ledger's register rows and `- gap |` lines hold (and, with --dir, every number the merged export writes and every pending proposal's Q) plus 1. It never fills a gap. It refuses a file without exactly one open-question register heading (check_ledger, as sync-ledger does) and a --count below 1. - exporters.numbering(ids, held): a row the ledger already holds keeps its ledger number and its [] lead whatever its page id; a Q id new to the ledger keeps N unless a ledger row already has that number (then it takes the next number and a [Q] lead); any other id takes the next number after the highest so far (ledger numbers and the Q ids kept in this export), in natural id order. A row leads with [] only when its number is not its own id. The second parameter is now the ledger's {id: number} map (ledger_numbers), since a bool cannot carry the numbers. ledger_gapped had no caller left and is removed with its test. - export-ledger --ledger X keeps each hand-written `- gap |` line of X, placed before the first row past its range, as sync-ledger keeps it. - check-open-questions.sh accepts a gap only where a hand-written `- gap | Q-Q | ` line in the register section declares it; an undeclared gap (or the undeclared part of one) still exits 2 and its message names the gap line to write; a malformed or fenced gap line declares nothing. - templates/checklist.md: the register header names the gap line. - surface/README.md: a next-id row in the command table. Intended reversals (the plan's numbering contract replaces the renumbering rule): 1. test_non_contiguous_ids_are_renumbered_with_the_original_id is rewritten as test_non_contiguous_ids_are_never_renumbered. 2. test_numbering_keeps_gapped_q_ids_only_for_a_gapped_ledger is rewritten as test_numbering_keeps_every_q_id_and_numbers_others_after_the_highest. The Brief test test_without_a_ledger_a_gap_keeps_the_register_number_and_ adds_the_id is updated to test_without_a_ledger_a_gap_keeps_every_id_with_ no_lead: it now asserts the gate refuses the undeclared gap, then passes with brief=ok once a gap line declares it. New tests: test_numbering_keeps_ledger_numbers_and_numbers_only_new_ids, test_a_renumbered_ledger_exports_twice_with_the_numbers_it_holds, test_export_ledger_keeps_each_hand_written_gap_line_in_place, test_a_q_id_new_to_the_ledger_keeps_its_number, test_a_new_q_id_whose_number_the_ledger_holds_takes_the_next_number, TestNextId (6 tests), and the gate's declared, partly declared, fenced and malformed gap-line cases plus the gap-line message check. Known limit: an export with no ledger text (export-ledger without --ledger, the report, a recap with a data dir and no ledger) numbers an imported renumbered ledger's rows from their ids, not the numbers the ledger held (rows [Q7] at Q1 and [J1] at Q2 export as Q7 and Q8). A later phase records each row's number in meta.seededFrom at import. Pre-flight: readers of register ids and of numbering() - surface/exporters.py: ROW regex in scan_register/parse_register (row number, cells); LEAD regex in ledger_register, import_ledger, ledger_row and recap_answer ([] lead -> question id); REGISTER_LEAD in locate_row/split_lead (supersede and resolve find a row by Q or its lead); numbering() called by register(), merged_register() and export_brief(); brief_label() shows "Q []" when number != id. - surface/round.py: export-ledger, ledger-check, sync-ledger, add-round --sync-ledger and recap go through merged_register(); supersede and resolve take Q or the [] lead (locate_row); record_ledger_sync reads Q from merge notes. - scripts/check-open-questions.sh: row_candidate_re and the id regex in the register loop (track_id: duplicate, out of order), the gap pass after the loop, deferred_ids for --brief, and the sweep's "register through Q" and Q disposition checks. - shared/session.py: QID regex Q[1-9][0-9]{0,5} validates shared-session question ids; it reads no ledger. - skills/audit-answers/SKILL.md: reads register rows by status (superseded-by-plan, hedged:) in prose; no id parsing. - skills/wayfind/SKILL.md: chart step 1 seeds from interview-checklist.md by row status in prose; no id parsing. - plugins/work-items/skills/decompose/SKILL.md: reads answered Q rows of interview-checklist.md in prose; no id parsing. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh --- .../feat-interview-redesign-07635b38.md | 4 + .../planning/scripts/check-open-questions.sh | 41 ++++- .../scripts/check-open-questions.test.sh | 36 +++++ .../skills/interview/templates/checklist.md | 2 +- plugins/planning/surface/README.md | 1 + plugins/planning/surface/exporters.py | 112 +++++++++----- plugins/planning/surface/round.py | 22 ++- plugins/planning/surface/test_exporters.py | 145 ++++++++++++++---- plugins/planning/surface/test_round.py | 75 +++++++++ 9 files changed, 367 insertions(+), 71 deletions(-) diff --git a/.changes/planning/feat-interview-redesign-07635b38.md b/.changes/planning/feat-interview-redesign-07635b38.md index 4d27302574d..e8ad786bd6f 100644 --- a/.changes/planning/feat-interview-redesign-07635b38.md +++ b/.changes/planning/feat-interview-redesign-07635b38.md @@ -2,6 +2,10 @@ bump: minor --- +### Added + +- **Stable question ids.** `round.sh next-id --ledger [--dir ] [--count ]` prints the next free ids after the highest one held, and the ledger export keeps every number a ledger row already has instead of renumbering. `check-open-questions.sh` accepts a gap in ids only where a hand-written `- gap | Q-Q | ` line declares it, and `export-ledger --ledger` keeps those lines. + ### Removed - **The interview's per-round contradiction pass.** `/planning:interview` no longer dispatches a sub-agent before each round to check the draft against earlier answers. The end sweep still finds contradictions and asks each as one question naming both sides, and an answer that conflicts with an earlier one is still caught when it is given. The legacy eval case `pre-round-contradiction-pass` is removed with it. diff --git a/plugins/planning/scripts/check-open-questions.sh b/plugins/planning/scripts/check-open-questions.sh index 416f934f4be..69d0fd13346 100755 --- a/plugins/planning/scripts/check-open-questions.sh +++ b/plugins/planning/scripts/check-open-questions.sh @@ -30,7 +30,8 @@ # Exit 2 = ungradeable: no ledger, no register section, a duplicate register or # deferred-questions heading, an unterminated fenced block, an empty # register, a malformed row, an unknown status, a duplicate, an -# out-of-order or a missing (gap) Q id, an unfenced `- Q |` row +# out-of-order or a missing (gap) Q id that no `- gap | Q-Q | ` +# line declares, a malformed gap line, an unfenced `- Q |` row # outside the register section (any heading ends the section, a # sub-heading inside it included), a named `--brief` that is # missing, or a `--surface` that is missing, comes without `--brief` @@ -753,6 +754,7 @@ superseded=0 seen_ids=" " deferred_ids="" ids_in_order="" +declared_gaps="" max_id=0 rounds_seen=" " max_round=0 @@ -791,6 +793,8 @@ track_id() { # the live branch below both test it, and they must agree: a shape skipped as # documentation has to be the same shape that would have been graded as data. row_candidate_re='^[[:space:]]*-[[:space:]]+[Qq][0-9]+([^0-9]|$)' +gap_candidate_re='^[[:space:]]*-[[:space:]]+[Gg]ap([^A-Za-z0-9]|$)' +gap_line_re='^[[:space:]]*-[[:space:]]+[Gg]ap[[:space:]]*\|[[:space:]]*[Qq]([1-9][0-9]*)[[:space:]]*-[[:space:]]*[Qq]([1-9][0-9]*)[[:space:]]*\|[[:space:]]*[^[:space:]]' skipped_fenced_row=0 while IFS= read -r line; do @@ -810,6 +814,15 @@ while IFS= read -r line; do continue fi + if [[ "$line" =~ $gap_candidate_re ]]; then + if [[ "$line" =~ $gap_line_re && "${BASH_REMATCH[1]}" -le "${BASH_REMATCH[2]}" ]]; then + declared_gaps="$declared_gaps${BASH_REMATCH[1]} ${BASH_REMATCH[2]}"$'\n' + else + row_error "malformed gap line (expected '- gap | Q-Q | '): $line" + fi + continue + fi + # A candidate row's shape is validated below. The prefilter deliberately does # not require the first pipe: a row that lost it (`- Q2 open | round 1 | ...`) # would otherwise be skipped silently, the contiguity check would never see @@ -921,16 +934,32 @@ for lineno in $outside_lines; do row_error "question row outside the register section (any heading ends it, a sub-heading included); move the row into the register, above its next heading" done +undeclared_gap() { + local gap="Q$1" + [[ "$2" -gt "$1" ]] && gap="$gap to Q$2" + row_errors="${row_errors}gap in question ids: no row for $gap; declare an intended gap with a '- gap | Q$1-Q$2 | ' line"$'\n' +} + +report_gap() { + local lo="$1" hi="$2" a b + while read -r a b; do + [[ -n "$a" && "$b" -ge "$lo" ]] || continue + [[ "$a" -gt "$hi" ]] && break + [[ "$a" -gt "$lo" ]] && undeclared_gap "$lo" $((a - 1)) + lo=$((b + 1)) + [[ "$lo" -gt "$hi" ]] && return + done <<<"$sorted_gaps" + undeclared_gap "$lo" "$hi" +} + # A gap is a number missing below the highest id; one sort over the whole set, # not a walk to the highest id, so a mistyped Q99999 costs nothing extra. +sorted_gaps="" +[[ -z "$declared_gaps" ]] || sorted_gaps="$(printf '%s' "$declared_gaps" | sort -n -k1,1)" prev=0 # shellcheck disable=SC2086 # deliberate: one id per word for n in $(printf '%s\n' $ids_in_order | sort -n); do - if [[ "$n" -gt $((prev + 1)) ]]; then - gap="Q$((prev + 1))" - [[ "$n" -gt $((prev + 2)) ]] && gap="$gap to Q$((n - 1))" - row_errors="${row_errors}gap in question ids: no row for $gap"$'\n' - fi + [[ "$n" -gt $((prev + 1)) ]] && report_gap $((prev + 1)) $((n - 1)) prev="$n" done diff --git a/plugins/planning/scripts/check-open-questions.test.sh b/plugins/planning/scripts/check-open-questions.test.sh index ff2435e580f..2eb8df5b4a8 100755 --- a/plugins/planning/scripts/check-open-questions.test.sh +++ b/plugins/planning/scripts/check-open-questions.test.sh @@ -213,6 +213,41 @@ EOF )" expect_exit "gap in Q numbering -> 2" 2 --ledger "$gap" +declared_gap="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | Who writes? | admin +- gap | Q2-Q4 | the shared session's ids ran ahead +- Q5 | answered | round 1 | What format? | markdown +EOF +)" +expect_exit "a declared gap -> 0" 0 --ledger "$declared_gap" +expect_stdout "a declared gap grades clean" "registered=2 " --ledger "$declared_gap" +part_gap="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | Who writes? | admin +- gap | Q2-Q3 | dropped before the page +- Q6 | answered | round 1 | What format? | markdown +EOF +)" +expect_exit "a gap declared only in part -> 2" 2 --ledger "$part_gap" +part_err="$(stderr_of --ledger "$part_gap")" +if [[ "$part_err" == *"no row for Q4 to Q5"* && "$part_err" == *"- gap | Q4-Q5 |"* && "$part_err" != *"no row for Q2"* ]]; then pass "only the undeclared part of a gap is reported"; else fail "only the undeclared part of a gap is reported (stderr: '$part_err')"; fi +fenced_gap="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | Who writes? | admin +```text +- gap | Q2-Q2 | an example +``` +- Q3 | answered | round 1 | What format? | markdown +EOF +)" +expect_exit "a fenced gap line declares nothing -> 2" 2 --ledger "$fenced_gap" +for bad_gap in '- gap | Q2 | no range' '- gap | Q3-Q2 | reversed' '- gap | Q2-Q2 |'; do + bad_gap_ledger="$(printf -- '- Q1 | answered | round 1 | Who writes? | admin\n%s\n- Q3 | answered | round 1 | What format? | markdown\n' "$bad_gap" | mkledger)" + expect_exit "malformed gap line '$bad_gap' -> 2" 2 --ledger "$bad_gap_ledger" + if [[ "$(stderr_of --ledger "$bad_gap_ledger")" == *"malformed gap line"* ]]; then pass "'$bad_gap' is named malformed"; else fail "'$bad_gap' is named malformed"; fi +done + # 14b. A row that lost its first pipe must exit 2, not vanish. Skipping it # silently would hide Q2 from the contiguity check and grade the register # clean with a question missing — the failure this gate exists to refuse. @@ -877,6 +912,7 @@ expect_stdout "row errors keep the counts of the valid rows" "registered=2 open= # messages, neither reported as the other. gap_err="$(stderr_of --ledger "$gap")" if [[ "$gap_err" == *"gap in question ids: no row for Q2"* && "$gap_err" != *"out of order"* ]]; then pass "a missing id reads as a gap"; else fail "a missing id reads as a gap (stderr: '$gap_err')"; fi +if [[ "$gap_err" == *"- gap | Q2-Q2 | "* ]]; then pass "an undeclared gap names the line that declares it"; else fail "an undeclared gap names the line that declares it (stderr: '$gap_err')"; fi swapped="$( mkledger <<'EOF' - Q2 | answered | round 1 | Format? | markdown diff --git a/plugins/planning/skills/interview/templates/checklist.md b/plugins/planning/skills/interview/templates/checklist.md index bf1f17a68c4..86604d2802e 100644 --- a/plugins/planning/skills/interview/templates/checklist.md +++ b/plugins/planning/skills/interview/templates/checklist.md @@ -38,7 +38,7 @@ One row per sourced term the rounds rely on, built and agreed with the user befo ## Open-question register -**Write a row the moment a round is asked, before any reply arrives.** The register is a byproduct of asking, not of resolving: a question that only lands on disk once it is answered cannot record the failure of never being answered. Statuses: `open` | `discussing` (the user asked to talk it through; blocks like `open`; `context/loop.md` "Discussing") | `answered` | `deferred` | `withdrawn` | `blocked` | `superseded-by-plan` (set by `/planning:plan` when a plan change displaces an answer, resolution `plan proposes: ; was: `; not terminal; row shape and how it clears: `context/loop.md` "Superseded by plan"). `Q` matches the terminal numbering and runs continuously across rounds with no gaps. +**Write a row the moment a round is asked, before any reply arrives.** The register is a byproduct of asking, not of resolving: a question that only lands on disk once it is answered cannot record the failure of never being answered. Statuses: `open` | `discussing` (the user asked to talk it through; blocks like `open`; `context/loop.md` "Discussing") | `answered` | `deferred` | `withdrawn` | `blocked` | `superseded-by-plan` (set by `/planning:plan` when a plan change displaces an answer, resolution `plan proposes: ; was: `; not terminal; row shape and how it clears: `context/loop.md` "Superseded by plan"). `Q` matches the terminal numbering and runs continuously across rounds; an intended gap needs a hand-written `- gap | Q-Q | ` line in this section. Each `Commits you to:` part of a recommendation is a row of its own with `(part of Q)` in the question field, numbered right after its headline when asked with it, and at the next free id when registered after later ids exist. Resolution rules: `context/loop.md` "Commitment rows"; late numbering: "Commitment rows written after later ids exist". On the page surface, the parts are the question's `commits` entries instead (see `context/loop.md` "Commitment parts on the page"). The page exports each row's resolution as named fields and every older grammar still imports (`context/surface.md` "Start and stop"). diff --git a/plugins/planning/surface/README.md b/plugins/planning/surface/README.md index 4a3c65c7441..b2808ce64df 100644 --- a/plugins/planning/surface/README.md +++ b/plugins/planning/surface/README.md @@ -86,6 +86,7 @@ Every command needs `--dir ''`; there is no default. Every command exi | `supersede --ledger L Q --proposes-file P` | Set L's answered register row to `superseded-by-plan`, proposing the text in file P (UTF-8, surrounding whitespace dropped; an empty one is refused): the row's answer becomes `was` (a named-field row's `answer` field, as the exporter writes it; a named row that also carries a `note`, `hold`, `aside` or commitments is refused), written `was plain::` or `was answer::` to record which form the row had, and both values are escaped by `esc_field`. Needs no data dir or page. Backs L up (`L..bak`), rewrites only that row, and prints the `check-open-questions.sh` verdict; refuses a row that is not `answered` | | `resolve --ledger L Q reconfirm\|restore [--note-file N]` | Clear a superseded row: `reconfirm` writes the answered `proposal:: ...; was:: ...; answer:: accepted: ...` form (file N holds the accept's note; `restore` refuses it), `restore` puts back the row `supersede` replaced, character for character, in the form its `was` names; a bare `was::` from an older ledger is guessed: in a ledger already in the named grammar an answer-shaped `was` comes back as `answer:: ...`, else as the plain cell. Same backup and verdict; refuses a row that is not `superseded-by-plan` | | `import-ledger --ledger F` | Seed an empty data dir from an existing ledger. A round cell reads as the N of a leading `round `, else the cell's first integer, else 1, with a warning naming the row when the leading form is missing; a cell other than the bare `round ` is kept and written back by `export-ledger` while the question's round still equals its number. A row in the `export-ledger` grammar (named fields `hold`, `proposal`, `was` (or `was plain`, `was answer`, which `export-ledger` writes back), `answer`, `note`, `aside`, `commitments`, each escaped as `exporters.py` documents) restores its hold, a superseded-by-plan row's proposal and displaced answer, the decision on any status, its note, a decision a user hold set aside (still set aside), an open row's accept that a revised recommendation set aside (still set aside, as of the text it accepted) and every commitment with its confirmed or unconfirmed mark; an unknown field or a contradictory pair is refused, naming the field. Two held-row cases have no field and are not restored: a seeded `blocked` row comes back `deferred` (still USER-RESERVED), and the seed text of an open or superseded-by-plan row is lost when its hold set aside an accept or an alternative, whose own note takes `note`. Every older grammar still imports, read as it always was: `waits on::`, `awaiting user::`, `confirmed::`, `plan proposes::` and `answer::` rows, a settled row's `; confirmed::` tail, and the unescaped `waits on:`, `confirmed:` and `; confirmed:` forms | +| `next-id --ledger L [--count N]` | Print the next N question ids (default 1), one per line: the highest id L's register rows and `- gap \| Q-Q \| ` lines hold plus 1, never a reused or gap-filling id; with `--dir`, also every number the page's merged export writes and every pending proposal's `Q`. Needs no data dir; refuses an L without exactly one open-question register heading and a count below 1 | | `doctor --ledger L` | Report, writing nothing, what the running version needs and L or the page lacks: a `missing:` line for each of L's `## Constraint ledger` and `## Open-question register` sections and for open questions without a Basis or a `Checked against:` line in `facts`, and a `note:` when L's `Planning version:` line or `meta.pluginVersion` (the version that first wrote the data dir) names another version; exit 1 on any `missing:` line. `--dir` need not exist yet | | `lease [--release \| --takeover]` | Print the watcher holding the lease, or `no lease`; `--release` clears it; `--takeover` clears it and runs `watch.sh` for this session (run it as the background task after `/clear`), and the old watcher exits 3 with a `handed over, not a failure` message | diff --git a/plugins/planning/surface/exporters.py b/plugins/planning/surface/exporters.py index 8f27a25e884..d7860a13d48 100644 --- a/plugins/planning/surface/exporters.py +++ b/plugins/planning/surface/exporters.py @@ -4,6 +4,7 @@ export_ledger the interview ledger: a decision-tree checklist, then the open-question register in the row shape scripts/check-open-questions.sh grades, then the deferred questions; given a ledger's text it merges into that ledger's register (merged_register) + and keeps each of its `- gap |` lines before the first row past its range sync_ledger a ledger's text with only its register rows replaced by the merged rows export_brief the PLAN.md `## Brief` sections and an empty `## Plan`; given a ledger's text it numbers as that ledger and carries its ledger-only rows. Problem, Goal, Work size, @@ -45,10 +46,10 @@ is not the bare `round ` in meta.seededFrom.roundCells, which export writes back while the question's round still equals the cell's number, so an imported ledger round-trips byte for byte. -Register ids: surface ids that are already Q1..Qn stay as they are, and so do gapped Q ids -when the ledger merged into already has a gap in its register; any other set is renumbered Q1..Qn -in natural id order (letters, then number) and each row's resolution leads with `[]`, which -import_ledger reads back so a re-export numbers the same way. +Register ids are never renumbered: a row the ledger merged into already holds keeps its number, +a Q id new to that ledger keeps N, and any other id takes the next number after the highest, +in natural id order (letters, then number). A row whose number is not its own id leads its +resolution with `[]`, which import_ledger reads back so a re-export numbers the same way. Every row export_ledger writes has one resolution grammar: named fields in this fixed order, each present only when it has a value, each value escaped on its own (esc_field), joined with `; `: @@ -130,6 +131,7 @@ # The note an accept made by an accept-audit event exports with (schema/event.schema.json). PENDING_NOTE = "pending agent validation" ROW = re.compile(r"^\s*-\s+[Qq]([0-9]+)\s*\|(.*)$") +GAP_LINE = re.compile(r"^\s*-\s+[Gg]ap\s*\|\s*[Qq][0-9]+\s*-\s*[Qq]([0-9]+)\s*\|") LEAD = re.compile(r"^\[([^\]\s]+)\]\s*(.*)$") FENCE = re.compile(r"^\s*(```|~~~)") MID_SENTENCE = re.compile(r"[.!?)\"'`\]]$") @@ -641,29 +643,36 @@ def settle(q, responses, events, seed_rows): return status if status == "superseded-by-plan" else "open", fields, "", False -def numbering(ids, gaps=False): - """({id: register number}, kept): ids that are already Q1..Qn keep their number, and so do - gapped Q ids when `gaps` (the ledger being written already has a gap), so a record citing - one still names the same question; any other set is numbered from 1 in natural id order.""" - if all(QN.match(i) for i in ids): - nums = sorted(int(i[1:]) for i in ids) - if gaps or nums == list(range(1, len(ids) + 1)): - return {i: int(i[1:]) for i in ids}, True - return {i: n for n, i in enumerate(sorted(ids, key=natural), start=1)}, False +def numbering(ids, held=None): + """{id: register number}, never reusing or changing a number, so a record citing one still + names the same question: an id `held` ({id: number} of the ledger being written) keeps its + ledger number; a Q id new to the ledger keeps N unless another row already has it; every + other id, in natural id order, takes the next number after the highest so far.""" + held = held or {} + pos = {i: held[i] for i in ids if i in held} + taken = set(held.values()) + for i in ids: + if i not in pos and QN.match(i) and int(i[1:]) not in taken: + pos[i] = int(i[1:]) + taken.add(pos[i]) + top = max(taken, default=0) + for i in sorted({i for i in ids if i not in pos}, key=natural): + top += 1 + pos[i] = top + return pos -def ledger_gapped(text): - """True when a ledger's register row numbers are not Q1..Qn.""" - ns = sorted(row[0] for row in parse_register(text or "")) - return ns != list(range(1, len(ns) + 1)) +def lead_of(qid, n): + """The `[] ` a register row leads with when its number is not its own id.""" + return "" if f"Q{n}" == qid else f"[{clean(qid)}] " -def register(doc, resp, extra=(), gaps=False): +def register(doc, resp, extra=(), numbers=None): """One dict per question, in register order, with its Q and settled status; `extra` - names ledger-only ids that share the numbering, and `gaps` is numbering's.""" + names ledger-only ids that share the numbering, and `numbers` is numbering's `held`.""" qs = sorted(doc.get("questions") or [], key=lambda q: natural(q["id"])) held = list(doc.get("pendingProposals") or []) - pos, kept = numbering([q["id"] for q in qs] + held + list(extra), gaps) + pos = numbering([q["id"] for q in qs] + held + list(extra), numbers) events = resp.get("events") or [] responses = resp.get("responses") or {} seed_rows = ((doc.get("meta") or {}).get("seededFrom") or {}).get("rows") or {} @@ -672,7 +681,7 @@ def register(doc, resp, extra=(), gaps=False): for q in qs: status, fields, note, reserved = settle(q, responses, events, seed_rows) marked = marked_commits(q, events) - lead = "" if kept else f"[{clean(q['id'])}] " + lead = lead_of(q["id"], pos[q["id"]]) res = encode(fields, marked) if q["id"] in unchecked: stale = f"stale:: {', '.join(unchecked[q['id']])}" @@ -727,11 +736,21 @@ def ledger_register(text): file order; the id is a row's `[]` lead, else its Q.""" rows = {} for n, status, _, title, res, cell in parse_register(text): - lead = LEAD.match(res) - rows[lead.group(1) if lead else f"Q{n}"] = (status, cell, title, res) + rows[row_id(n, res)] = (status, cell, title, res) return rows +def row_id(n, res): + """A register row's question id: its `[]` lead, else its Q.""" + lead = LEAD.match(res) + return lead.group(1) if lead else f"Q{n}" + + +def ledger_numbers(text): + """{question id: register number} of a ledger's rows; empty with no ledger.""" + return {row_id(n, res): n for n, _, _, _, res, _ in parse_register(text or "")} + + COLUMNS = ("status", "round", "title", "resolution") @@ -749,8 +768,8 @@ def merged_register(d, text=None): old = ledger_register(text) if text is not None else {} page_ids = {q["id"] for q in doc.get("questions") or []} extra = [i for i in old if i not in page_ids] - gaps = ledger_gapped(text) - pos, kept = numbering(sorted(page_ids) + doc["pendingProposals"] + extra, gaps) + numbers = ledger_numbers(text) + pos = numbering(sorted(page_ids) + doc["pendingProposals"] + extra, numbers) seeded = (doc.get("meta") or {}).get("seededFrom") or {} cells = seeded.get("roundCells") or {} responses = resp.get("responses") or {} @@ -768,7 +787,7 @@ def page_decided(q): ) rows = [] - for r in register(doc, resp, extra, gaps): + for r in register(doc, resp, extra, numbers): q = r["q"] rnd, added = q.get("round") or 1, q.get("addedInRound") page = { @@ -791,8 +810,6 @@ def page_decided(q): rows.append(row) for i in extra: status, cell, title, res = old[i] - if not kept and not LEAD.match(res): - res = f"[{clean(i)}] {res}".rstrip() rows.append( { "n": f"Q{pos[i]}", @@ -843,7 +860,12 @@ def export_ledger(d, text=None): mark = " " if r["status"] in UNSETTLED else "x" out.append(f"- [{mark}] {r['n']} {r['short']}: {r['status']}") out += ["", "## Open-question register", ""] - out += [row_line(r) for r in rows] + gaps = gap_lines(text or "") + for r in rows: + while gaps and gaps[0][0] < int(r["n"][1:]): + out.append(gaps.pop(0)[1]) + out.append(row_line(r)) + out += [line for _, line in gaps] out += ["", "### Deferred questions", ""] retired = [r for r in rows if r["status"] in ("deferred", "blocked")] out += [f"- {r['n']}: {r['title']}" for r in retired] or ["- none"] @@ -1313,9 +1335,9 @@ def export_brief(d, text=None): old = ledger_register(text) if text is not None else {} page_ids = {q["id"] for q in doc.get("questions") or []} extra = [i for i in old if i not in page_ids] - gaps = ledger_gapped(text) - rows = register(doc, resp, extra, gaps) - pos, _ = numbering(sorted(page_ids) + doc["pendingProposals"] + extra, gaps) + numbers = ledger_numbers(text) + rows = register(doc, resp, extra, numbers) + pos = numbering(sorted(page_ids) + doc["pendingProposals"] + extra, numbers) for i in extra: status, _, title, res = old[i] rows.append(ledger_row(f"Q{pos[i]}", i, status, title, res)) @@ -1735,9 +1757,9 @@ def round_drift(doc): ] -def scan_register(lines): +def scan_register(lines, row=ROW): """(heads, rows) over a ledger's lines outside fenced blocks: the index of every heading - naming the open-question register, and (index, row match) for each row under the first.""" + naming the open-question register, and (index, `row` match) for each row under the first.""" heads, rows, fenced, inside = [], [], False, False for i, line in enumerate(lines): if FENCE.match(line): @@ -1750,12 +1772,32 @@ def scan_register(lines): if inside: heads.append(i) continue - m = ROW.match(line) if inside and len(heads) == 1 else None + m = row.match(line) if inside and len(heads) == 1 else None if m: rows.append((i, m)) return heads, rows +def gap_lines(text): + """[(highest id it declares, line)] of a ledger's `- gap |` lines, in file order.""" + lines = text.splitlines() + return [(int(m.group(1)), lines[i].strip()) for i, m in scan_register(lines, GAP_LINE)[1]] + + +def highest_id(text, d=None): + """The highest question number a ledger's register rows and `- gap |` lines hold, and with a + data dir every number its merged export writes and every pending proposal's Q; 0 for + none.""" + lines = text.splitlines() + nums = [int(m.group(1)) for _, m in scan_register(lines)[1]] + nums += [n for n, _ in gap_lines(text)] + if d is not None: + doc, rows, _ = merged_register(d, text) + nums += [int(r["n"][1:]) for r in rows] + nums += [int(i[1:]) for i in doc["pendingProposals"] if QN.match(i)] + return max(nums, default=0) + + def parse_register(text): """Register rows as (n, status, round, title, resolution, round cell), skipping fenced blocks.""" diff --git a/plugins/planning/surface/round.py b/plugins/planning/surface/round.py index a9e5b54b142..c1fc236ef34 100644 --- a/plugins/planning/surface/round.py +++ b/plugins/planning/surface/round.py @@ -2896,6 +2896,15 @@ def cmd_resolve(d, a): ) +def cmd_next_id(d, a): + if a.count < 1: + sys.exit(f"refused: --count must be 1 or more, not {a.count}") + text = read_input(a.ledger, "ledger") + exporters.check_ledger(text, a.ledger) + top = exporters.highest_id(text, d) + print("\n".join(f"Q{top + k}" for k in range(1, a.count + 1))) + + def cmd_import_ledger(d, a): text = read_input(a.ledger, "ledger") with sidecar_lock(d): @@ -3396,6 +3405,17 @@ def main(argv=None): ) s.set_defaults(fn=cmd_ledger_check) + s = sub.add_parser( + "next-id", + help="print the next question ids, one per line: the highest id the ledger (and the data " + "dir, when --dir is given) holds plus 1; never a reused or gap-filling id", + ) + s.add_argument("--ledger", required=True, help="ledger markdown file") + s.add_argument( + "--count", type=int, default=1, help="how many ids to print (default 1)" + ) + s.set_defaults(fn=cmd_next_id) + s = sub.add_parser("import-ledger", help="seed an empty data dir from a ledger") s.add_argument("--ledger", required=True, help="ledger markdown file") s.set_defaults(fn=cmd_import_ledger) @@ -3518,7 +3538,7 @@ def main(argv=None): ) if extra: p.error(f"unrecognized arguments: {' '.join(extra)}") - if a.fn in (cmd_supersede, cmd_resolve): + if a.fn in (cmd_supersede, cmd_resolve) or (a.fn is cmd_next_id and not a.dir): a.fn(None, a) return if ( diff --git a/plugins/planning/surface/test_exporters.py b/plugins/planning/surface/test_exporters.py index 3a487e04909..6aad4c4010f 100644 --- a/plugins/planning/surface/test_exporters.py +++ b/plugins/planning/surface/test_exporters.py @@ -253,7 +253,7 @@ def test_a_terminal_decision_with_a_rev_is_set_aside_by_rev(self): held = {**term, "rev": 5} self.assertIsNone(exporters.latest_decision({**q, "terminal": held}, {})) - def test_non_contiguous_ids_are_renumbered_with_the_original_id(self): + def test_non_contiguous_ids_are_never_renumbered(self): self.session( [question("J2"), question("J1"), question("Q7")], [ @@ -264,39 +264,120 @@ def test_non_contiguous_ids_are_renumbered_with_the_original_id(self): ) ledger = self.export("ledger") rows = register_rows(ledger) - self.assertEqual([r.split(" | ")[0] for r in rows], ["- Q1", "- Q2", "- Q3"]) - self.assertIn("[J1]", rows[0]) - self.assertIn("[J2]", rows[1]) - self.assertIn("[Q7]", rows[2]) + self.assertEqual([r.split(" | ")[0] for r in rows], ["- Q7", "- Q8", "- Q9"]) + self.assertNotIn("[Q7]", rows[0]) + self.assertIn("[J1]", rows[1]) + self.assertIn("[J2]", rows[2]) rc, out = self.check("--ledger", ledger) - self.assertEqual(rc, 0, out) + self.assertEqual(rc, 2, out) + self.assertIn("no row for Q1 to Q6", out) + self.assertIn("- gap | Q1-Q6 |", out) - def test_numbering_keeps_gapped_q_ids_only_for_a_gapped_ledger(self): - gapped = ["Q500", "Q1", "Q2", "Q501"] + def test_numbering_keeps_ledger_numbers_and_numbers_only_new_ids(self): self.assertEqual( - exporters.numbering(gapped, gaps=True), - ({"Q1": 1, "Q2": 2, "Q500": 500, "Q501": 501}, True), + exporters.numbering(["J1", "Q7"], {"Q7": 1, "J1": 2}), {"Q7": 1, "J1": 2} ) self.assertEqual( - exporters.numbering(gapped), - ({"Q1": 1, "Q2": 2, "Q500": 3, "Q501": 4}, False), + exporters.numbering(["Q1", "Q2", "Q600"]), + {"Q1": 1, "Q2": 2, "Q600": 600}, ) self.assertEqual( - exporters.numbering(["Q2", "Q1", "Q3"]), - ({"Q1": 1, "Q2": 2, "Q3": 3}, True), + exporters.numbering( + ["Q1", "Q2", "Q3", "Q4", "J1"], {"Q1": 1, "Q2": 2, "Q3": 3} + ), + {"Q1": 1, "Q2": 2, "Q3": 3, "Q4": 4, "J1": 5}, ) self.assertEqual( - exporters.numbering(["Q7", "J1"], gaps=True), ({"J1": 1, "Q7": 2}, False) + exporters.numbering(["J1", "Q3"], {"J1": 3}), {"J1": 3, "Q3": 4} ) - def test_ledger_gapped_reads_the_register_row_numbers(self): - def ledger(*ns): - rows = "".join(f"- Q{n} | open | round 1 | Q{n}? |\n" for n in ns) - return "# Interview ledger\n\n## Open-question register\n\n" + rows + RENUMBERED = ( + "# Interview ledger\n\n## Open-question register\n\n" + "- Q1 | answered | round 1 | Question Q7? | [Q7] free-text: first\n" + "- Q2 | answered | round 1 | Question J1? | [J1] free-text: second\n" + ) - self.assertTrue(exporters.ledger_gapped(ledger(1, 5))) - self.assertFalse(exporters.ledger_gapped(ledger(1, 2))) - self.assertFalse(exporters.ledger_gapped(None)) + def test_a_renumbered_ledger_exports_twice_with_the_numbers_it_holds(self): + self.session([question("J1"), question("Q7")], []) + text = self.RENUMBERED + for export in ("first", "second"): + _, rows, _ = exporters.merged_register(self.dir, text) + self.assertEqual( + [(r["n"], r["id"]) for r in rows], + [("Q1", "Q7"), ("Q2", "J1")], + f"{export} export", + ) + self.assertTrue(rows[0]["resolution"].startswith("[Q7]"), rows[0]) + self.assertTrue(rows[1]["resolution"].startswith("[J1]"), rows[1]) + text = exporters.export_ledger(self.dir, text) + + def test_export_ledger_keeps_each_hand_written_gap_line_in_place(self): + self.session([question("Q1"), question("Q5"), question("Q9")], []) + ledger = self.tmp / "ledger.md" + ledger.write_text( + "# Interview ledger\n\n## Open-question register\n\n" + "- Q1 | open | round 1 | Question Q1? |\n" + "- gap | Q2-Q4 | dropped before the page\n" + "- Q5 | open | round 1 | Question Q5? |\n" + "- gap | Q6-Q8 | the shared session's ids ran ahead\n" + "- Q9 | open | round 1 | Question Q9? |\n", + encoding="utf-8", + newline="\n", + ) + out = self.tmp / "exported.md" + rc, text = self.rp("export-ledger", "--ledger", str(ledger), "--out", str(out)) + self.assertEqual(rc, 0, text) + register = out.read_text(encoding="utf-8").split("## Open-question register")[1] + lines = [x for x in register.splitlines() if x.startswith("- ")] + self.assertEqual( + [x.split(" | ")[:2] for x in lines[:5]], + [ + ["- Q1", "open"], + ["- gap", "Q2-Q4"], + ["- Q5", "open"], + ["- gap", "Q6-Q8"], + ["- Q9", "open"], + ], + ) + self.assertIn("- gap | Q6-Q8 | the shared session's ids ran ahead", lines) + rc, verdict = self.check("--ledger", out) + self.assertEqual(rc, 1, verdict) + self.assertIn("registered=3 open=3", verdict) + + def test_a_q_id_new_to_the_ledger_keeps_its_number(self): + self.session([question("Q1"), question("Q2"), question("Q600")], []) + _, rows, _ = exporters.merged_register( + self.dir, + "# Interview ledger\n\n## Open-question register\n\n" + "- Q1 | open | round 1 | Question Q1? |\n" + "- Q2 | open | round 1 | Question Q2? |\n", + ) + self.assertEqual([r["n"] for r in rows], ["Q1", "Q2", "Q600"]) + self.assertFalse([r for r in rows if r["resolution"].startswith("[")]) + + def test_a_new_q_id_whose_number_the_ledger_holds_takes_the_next_number(self): + self.session([question("Q1"), question("J1"), question("Q2")], []) + _, rows, _ = exporters.merged_register( + self.dir, + "# Interview ledger\n\n## Open-question register\n\n" + "- Q1 | open | round 1 | Question Q1? |\n" + "- Q2 | open | round 1 | Question J1? | [J1]\n", + ) + self.assertEqual( + [(r["n"], r["id"]) for r in rows], + [("Q1", "Q1"), ("Q2", "J1"), ("Q3", "Q2")], + ) + self.assertTrue(rows[2]["resolution"].startswith("[Q2]"), rows[2]) + + def test_numbering_keeps_every_q_id_and_numbers_others_after_the_highest(self): + self.assertEqual( + exporters.numbering(["Q500", "Q1", "Q2", "Q501"]), + {"Q1": 1, "Q2": 2, "Q500": 500, "Q501": 501}, + ) + self.assertEqual( + exporters.numbering(["Q2", "Q1", "Q3"]), {"Q1": 1, "Q2": 2, "Q3": 3} + ) + self.assertEqual(exporters.numbering(["Q7", "J1"]), {"Q7": 7, "J1": 8}) def test_gapped_q_ids_keep_their_number_with_no_lead(self): self.session( @@ -830,7 +911,7 @@ def test_the_page_constraints_lead_and_the_ledger_adds_only_other_ids(self): self.assertNotIn("internal tool only", constraints) self.assertIn("- C2 (inherited):", constraints) - def test_without_a_ledger_a_gap_keeps_the_register_number_and_adds_the_id(self): + def test_without_a_ledger_a_gap_keeps_every_id_with_no_lead(self): events, revs = confirmed( [ event(1, "Q1", "accept"), @@ -842,12 +923,20 @@ def test_without_a_ledger_a_gap_keeps_the_register_number_and_adds_the_id(self): [question("Q1"), question("Q3"), question("Q4")], events, restatements=revs ) text = self.export("brief").read_text(encoding="utf-8") - self.assertIn("- Q2 [Q3]: Question Q3?, defer until later;", text) - self.assertIn("- Q3 [Q4] Short Q4: accepted:", text) - self.assertNotIn("[Q3] [Q3]", text) - rc, out = self.check( - "--ledger", self.export("ledger"), "--brief", self.export("brief") + self.assertIn("- Q3: Question Q3?, defer until later;", text) + self.assertIn("- Q4 Short Q4: accepted:", text) + self.assertNotIn("[Q3]", text) + self.assertNotIn("[Q4]", text) + ledger = self.export("ledger") + brief = self.export("brief") + rc, out = self.check("--ledger", ledger, "--brief", brief) + self.assertEqual(rc, 2, out) + self.assertIn("- gap | Q2-Q2 |", out) + declared = ledger.read_text(encoding="utf-8").replace( + "\n- Q3 |", "\n- gap | Q2-Q2 | dropped before the page\n- Q3 |", 1 ) + ledger.write_text(declared, encoding="utf-8", newline="\n") + rc, out = self.check("--ledger", ledger, "--brief", brief) self.assertEqual(rc, 0, out) self.assertIn("brief=ok", out) diff --git a/plugins/planning/surface/test_round.py b/plugins/planning/surface/test_round.py index be17d733dad..17b1569b0f1 100644 --- a/plugins/planning/surface/test_round.py +++ b/plugins/planning/surface/test_round.py @@ -4377,6 +4377,81 @@ def test_add_round_sync_keeps_gapped_ids_and_writes_the_page_ids(self): self.assertFalse([r for r in rows if "[Q" in r], rows) +class TestNextId(DirCase): + """next-id prints the ids after the highest the ledger, and the data dir when given, holds.""" + + REGISTER = "# Interview ledger\n\n## Open-question register\n\n" + + def ledger(self, rows): + path = self.tmp / "interview-checklist.md" + path.write_bytes((self.REGISTER + rows).encode("utf-8")) + return str(path) + + def next_id(self, *args): + p = subprocess.run( + [sys.executable, str(ROUND), "next-id", *args], + capture_output=True, + text=True, + timeout=60, + ) + return p.returncode, p.stdout, p.stderr + + def test_a_stray_high_id_is_never_filled_below(self): + ledger = self.ledger( + "- Q1 | open | round 1 | One? |\n" + "- Q2 | open | round 1 | Two? |\n" + "- Q600 | open | round 1 | Stray? |\n" + ) + rc, out, err = self.next_id("--ledger", ledger) + self.assertEqual(rc, 0, out + err) + self.assertEqual(out, "Q601\n") + + def test_count_prints_one_id_per_line(self): + ledger = self.ledger( + "- Q1 | open | round 1 | One? |\n- Q2 | open | round 1 | Two? |\n" + ) + rc, out, err = self.next_id("--ledger", ledger, "--count", "3") + self.assertEqual(rc, 0, out + err) + self.assertEqual(out, "Q3\nQ4\nQ5\n") + + def test_a_declared_gap_is_never_reused(self): + ledger = self.ledger( + "- Q1 | open | round 1 | One? |\n" + "- gap | Q2-Q9 | the shared session's ids ran ahead\n" + ) + rc, out, err = self.next_id("--ledger", ledger) + self.assertEqual(rc, 0, out + err) + self.assertEqual(out, "Q10\n") + + def test_the_data_dir_counts_its_page_ids_and_their_export_numbers(self): + doc = base_doc() + doc["questions"].append(question("J1", group="g2", round=1)) + self.write_doc(doc) + ledger = self.ledger( + "- Q1 | open | round 1 | One? |\n- Q2 | open | round 1 | Two? |\n" + ) + rc, out, err = self.next_id("--ledger", ledger, "--dir", str(self.dir)) + self.assertEqual(rc, 0, out + err) + self.assertEqual(out, "Q5\n") + + def test_a_file_with_no_register_heading_is_refused(self): + path = self.tmp / "notes.md" + path.write_bytes(b"# Notes\n\n- Q1 | open | round 1 | One? |\n") + rc, out, err = self.next_id("--ledger", str(path)) + self.assertNotEqual(rc, 0, out + err) + self.assertEqual(out, "") + self.assertIn("open-question register heading", err) + + def test_a_count_below_one_is_refused(self): + ledger = self.ledger("- Q1 | open | round 1 | One? |\n") + rc, out, err = self.next_id("--ledger", ledger, "--count", "0") + self.assertNotEqual(rc, 0, out + err) + self.assertEqual(out, "") + self.assertIn("--count", err) + rc, out, err = self.next_id("--ledger", ledger, "--count", "1") + self.assertEqual((rc, out), (0, "Q2\n"), err) + + class TestSupersedeResolve(DirCase): """supersede and resolve edit one register row of a ledger and need no data dir.""" From acdc7c3d53eea877d49365789313f78fac2f55f8 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 21 Nov 2026 03:47:18 -0500 Subject: [PATCH 03/30] fix(planning): declared gaps reserve their ids in numbering and next-id Code-design review findings on 146fa27a8d, all in Phase 3's files. - numbering(ids, held, gaps) treats every number inside a declared `- gap | Q-Q |` range as taken: a new non-Q id never lands in a gap, and a page Q id inside one is numbered after the highest number with its [] lead. page_numbering() builds the id list once from the page, its pending proposals, the ledger-only ids and the ledger's numbers and gaps; register() takes that numbering from its caller, so merged_register and export_brief no longer compute it twice, and export_report still gets the page's own. - highest_id (next-id) also counts each [Q] lead in the register and, with --dir, each page question's own Q id. - One gap-line grammar: Python's GAP_LINE/parse_gap now match the gate (no leading zero at either end, a <= b, a non-empty reason, both ends captured). - Gate: a declared range that covers a register row, or whose high end is past the highest row, exits 2 naming the gap line; the gap candidate is `gap` followed by optional spaces and `|`, so a prose bullet such as `- gap, see below` is not read as a gap line. - register_lines() places gap lines by range for both writers: export_ledger and sync_ledger now agree (sync_ledger used to leave a gap line where it stood while the rows moved above it). - Docstrings: numbering says what happens with no ledger; the module header states the gap-line order. Tests: test_a_new_id_never_lands_in_a_declared_gap, test_a_page_q_id_inside_a_declared_gap_is_numbered_after_it, test_export_ledger_keeps_a_gap_line_past_the_last_row_and_the_gate_refuses_it, test_sync_ledger_places_each_gap_line_by_its_range, test_the_gap_line_grammar_matches_the_gate (runs the gate and parse_gap on the gate test's valid and malformed gap lines), TestNextId.test_a_renumbered_row_lead_is_never_reused, TestNextId.test_a_pending_proposal_id_is_never_reused, and gate cases for a gap covering a row, a gap past the highest row and a prose `- gap,` bullet. The decision that a gapped export fails the gate until a hand-written gap line declares it is unchanged. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh --- .../planning/scripts/check-open-questions.sh | 21 ++- .../scripts/check-open-questions.test.sh | 26 ++++ plugins/planning/surface/exporters.py | 123 +++++++++++------- plugins/planning/surface/test_exporters.py | 95 ++++++++++++++ plugins/planning/surface/test_round.py | 20 +++ 5 files changed, 237 insertions(+), 48 deletions(-) diff --git a/plugins/planning/scripts/check-open-questions.sh b/plugins/planning/scripts/check-open-questions.sh index 69d0fd13346..88865663e04 100755 --- a/plugins/planning/scripts/check-open-questions.sh +++ b/plugins/planning/scripts/check-open-questions.sh @@ -31,7 +31,8 @@ # deferred-questions heading, an unterminated fenced block, an empty # register, a malformed row, an unknown status, a duplicate, an # out-of-order or a missing (gap) Q id that no `- gap | Q-Q | ` -# line declares, a malformed gap line, an unfenced `- Q |` row +# line declares, a malformed gap line, a declared gap that covers a +# row or runs past the highest row, an unfenced `- Q |` row # outside the register section (any heading ends the section, a # sub-heading inside it included), a named `--brief` that is # missing, or a `--surface` that is missing, comes without `--brief` @@ -793,7 +794,7 @@ track_id() { # the live branch below both test it, and they must agree: a shape skipped as # documentation has to be the same shape that would have been graded as data. row_candidate_re='^[[:space:]]*-[[:space:]]+[Qq][0-9]+([^0-9]|$)' -gap_candidate_re='^[[:space:]]*-[[:space:]]+[Gg]ap([^A-Za-z0-9]|$)' +gap_candidate_re='^[[:space:]]*-[[:space:]]+[Gg]ap[[:space:]]*\|' gap_line_re='^[[:space:]]*-[[:space:]]+[Gg]ap[[:space:]]*\|[[:space:]]*[Qq]([1-9][0-9]*)[[:space:]]*-[[:space:]]*[Qq]([1-9][0-9]*)[[:space:]]*\|[[:space:]]*[^[:space:]]' skipped_fenced_row=0 @@ -816,7 +817,7 @@ while IFS= read -r line; do if [[ "$line" =~ $gap_candidate_re ]]; then if [[ "$line" =~ $gap_line_re && "${BASH_REMATCH[1]}" -le "${BASH_REMATCH[2]}" ]]; then - declared_gaps="$declared_gaps${BASH_REMATCH[1]} ${BASH_REMATCH[2]}"$'\n' + declared_gaps="$declared_gaps${BASH_REMATCH[1]} ${BASH_REMATCH[2]} $lineno"$'\n' else row_error "malformed gap line (expected '- gap | Q-Q | '): $line" fi @@ -942,7 +943,7 @@ undeclared_gap() { report_gap() { local lo="$1" hi="$2" a b - while read -r a b; do + while read -r a b _; do [[ -n "$a" && "$b" -ge "$lo" ]] || continue [[ "$a" -gt "$hi" ]] && break [[ "$a" -gt "$lo" ]] && undeclared_gap "$lo" $((a - 1)) @@ -963,6 +964,18 @@ for n in $(printf '%s\n' $ids_in_order | sort -n); do prev="$n" done +read -ra row_ids <<<"$ids_in_order" +while read -r a b gap_line; do + [[ -n "$a" ]] || continue + [[ "$b" -gt "$max_id" ]] && row_errors="${row_errors}line $gap_line: declared gap Q$a-Q$b runs past the highest row Q$max_id"$'\n' + for n in ${row_ids[@]+"${row_ids[@]}"}; do + if [[ "$n" -ge "$a" && "$n" -le "$b" ]]; then + row_errors="${row_errors}line $gap_line: declared gap Q$a-Q$b covers the row Q$n"$'\n' + break + fi + done +done <<<"$declared_gaps" + if [[ -n "$row_errors" ]]; then # Exit 2 stays load-bearing for Step 3 and the --brief path; the verdict # keeps the counts of the rows that parsed, so a row error reads apart from diff --git a/plugins/planning/scripts/check-open-questions.test.sh b/plugins/planning/scripts/check-open-questions.test.sh index 2eb8df5b4a8..156b25ad08f 100755 --- a/plugins/planning/scripts/check-open-questions.test.sh +++ b/plugins/planning/scripts/check-open-questions.test.sh @@ -247,6 +247,32 @@ for bad_gap in '- gap | Q2 | no range' '- gap | Q3-Q2 | reversed' '- gap | Q2-Q2 expect_exit "malformed gap line '$bad_gap' -> 2" 2 --ledger "$bad_gap_ledger" if [[ "$(stderr_of --ledger "$bad_gap_ledger")" == *"malformed gap line"* ]]; then pass "'$bad_gap' is named malformed"; else fail "'$bad_gap' is named malformed"; fi done +covering_gap="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | Who writes? | admin +- gap | Q2-Q3 | dropped before the page +- Q3 | answered | round 1 | What format? | markdown +EOF +)" +expect_exit "a declared gap covering a row -> 2" 2 --ledger "$covering_gap" +if [[ "$(stderr_of --ledger "$covering_gap")" == *"line 10: declared gap Q2-Q3 covers the row Q3"* ]]; then pass "a gap covering a row is named with its line"; else fail "a gap covering a row is named with its line (stderr: '$(stderr_of --ledger "$covering_gap")')"; fi +trailing_gap="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | Who writes? | admin +- Q2 | answered | round 1 | What format? | markdown +- gap | Q3-Q5 | ids a peer session holds +EOF +)" +expect_exit "a declared gap past the highest row -> 2" 2 --ledger "$trailing_gap" +if [[ "$(stderr_of --ledger "$trailing_gap")" == *"line 11: declared gap Q3-Q5 runs past the highest row Q2"* ]]; then pass "a gap past the highest row is named with its line"; else fail "a gap past the highest row is named with its line (stderr: '$(stderr_of --ledger "$trailing_gap")')"; fi +prose_gap="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | Who writes? | admin +- gap, see below +- Q2 | answered | round 1 | What format? | markdown +EOF +)" +expect_exit "a prose bullet starting with gap is not a gap line -> 0" 0 --ledger "$prose_gap" # 14b. A row that lost its first pipe must exit 2, not vanish. Skipping it # silently would hide Q2 from the contiguity check and grade the register diff --git a/plugins/planning/surface/exporters.py b/plugins/planning/surface/exporters.py index d7860a13d48..e1d60a97c50 100644 --- a/plugins/planning/surface/exporters.py +++ b/plugins/planning/surface/exporters.py @@ -4,8 +4,10 @@ export_ledger the interview ledger: a decision-tree checklist, then the open-question register in the row shape scripts/check-open-questions.sh grades, then the deferred questions; given a ledger's text it merges into that ledger's register (merged_register) - and keeps each of its `- gap |` lines before the first row past its range - sync_ledger a ledger's text with only its register rows replaced by the merged rows + and keeps its well-formed `- gap |` lines, in range order, each right before + the first row numbered past its range (after the last row when none is) + sync_ledger a ledger's text with only its register rows and gap lines replaced by the + merged rows and those gap lines, placed as export_ledger places them export_brief the PLAN.md `## Brief` sections and an empty `## Plan`; given a ledger's text it numbers as that ledger and carries its ledger-only rows. Problem, Goal, Work size, Constraints, Acceptance criteria and Out-of-scope come from the newest confirmed @@ -47,8 +49,9 @@ question's round still equals the cell's number, so an imported ledger round-trips byte for byte. Register ids are never renumbered: a row the ledger merged into already holds keeps its number, -a Q id new to that ledger keeps N, and any other id takes the next number after the highest, -in natural id order (letters, then number). A row whose number is not its own id leads its +a Q id new to that ledger keeps N unless a row has it or a declared gap covers it, and any +other id takes the next number after the highest, in natural id order (letters, then number). +With no ledger every Q id keeps N. A row whose number is not its own id leads its resolution with `[]`, which import_ledger reads back so a re-export numbers the same way. Every row export_ledger writes has one resolution grammar: named fields in this fixed order, each @@ -131,7 +134,9 @@ # The note an accept made by an accept-audit event exports with (schema/event.schema.json). PENDING_NOTE = "pending agent validation" ROW = re.compile(r"^\s*-\s+[Qq]([0-9]+)\s*\|(.*)$") -GAP_LINE = re.compile(r"^\s*-\s+[Gg]ap\s*\|\s*[Qq][0-9]+\s*-\s*[Qq]([0-9]+)\s*\|") +GAP_LINE = re.compile( + r"^\s*-\s+[Gg]ap\s*\|\s*[Qq]([1-9][0-9]*)\s*-\s*[Qq]([1-9][0-9]*)\s*\|\s*\S" +) LEAD = re.compile(r"^\[([^\]\s]+)\]\s*(.*)$") FENCE = re.compile(r"^\s*(```|~~~)") MID_SENTENCE = re.compile(r"[.!?)\"'`\]]$") @@ -643,36 +648,47 @@ def settle(q, responses, events, seed_rows): return status if status == "superseded-by-plan" else "open", fields, "", False -def numbering(ids, held=None): +def numbering(ids, held=None, gaps=()): """{id: register number}, never reusing or changing a number, so a record citing one still - names the same question: an id `held` ({id: number} of the ledger being written) keeps its - ledger number; a Q id new to the ledger keeps N unless another row already has it; every - other id, in natural id order, takes the next number after the highest so far.""" + names the same question. An id `held` ({id: number} of the ledger being written) keeps its + ledger number. A Q id new to the ledger keeps N unless a ledger row has that number or + one of the ledger's declared `gaps` ((low, high) ranges) covers it. Every other id, in + natural id order, takes the next number after the highest held, kept or declared. With no + ledger every Q id keeps N and the rest number after the highest of them.""" held = held or {} pos = {i: held[i] for i in ids if i in held} taken = set(held.values()) for i in ids: - if i not in pos and QN.match(i) and int(i[1:]) not in taken: - pos[i] = int(i[1:]) - taken.add(pos[i]) - top = max(taken, default=0) + n = int(i[1:]) if QN.match(i) else None + if i not in pos and n and n not in taken and not any(a <= n <= b for a, b in gaps): + pos[i] = n + taken.add(n) + top = max([*taken, *(b for _, b in gaps)], default=0) for i in sorted({i for i in ids if i not in pos}, key=natural): top += 1 pos[i] = top return pos +def page_numbering(doc, extra=(), text=None): + """numbering() of the page's question ids, its pending proposals and the ledger-only `extra` + ids, against the row numbers and declared gaps of the ledger `text`.""" + ids = [q["id"] for q in doc.get("questions") or []] + ids += [*(doc.get("pendingProposals") or []), *extra] + gaps = [(a, b) for a, b, _, _ in gap_lines(text or "")] + return numbering(ids, ledger_numbers(text), gaps) + + def lead_of(qid, n): """The `[] ` a register row leads with when its number is not its own id.""" return "" if f"Q{n}" == qid else f"[{clean(qid)}] " -def register(doc, resp, extra=(), numbers=None): - """One dict per question, in register order, with its Q and settled status; `extra` - names ledger-only ids that share the numbering, and `numbers` is numbering's `held`.""" +def register(doc, resp, pos=None): + """One dict per question, in register order, with its Q and settled status; `pos` is the + page_numbering() the caller already made, else the page's own.""" qs = sorted(doc.get("questions") or [], key=lambda q: natural(q["id"])) - held = list(doc.get("pendingProposals") or []) - pos = numbering([q["id"] for q in qs] + held + list(extra), numbers) + pos = pos or page_numbering(doc) events = resp.get("events") or [] responses = resp.get("responses") or {} seed_rows = ((doc.get("meta") or {}).get("seededFrom") or {}).get("rows") or {} @@ -768,8 +784,7 @@ def merged_register(d, text=None): old = ledger_register(text) if text is not None else {} page_ids = {q["id"] for q in doc.get("questions") or []} extra = [i for i in old if i not in page_ids] - numbers = ledger_numbers(text) - pos = numbering(sorted(page_ids) + doc["pendingProposals"] + extra, numbers) + pos = page_numbering(doc, extra, text) seeded = (doc.get("meta") or {}).get("seededFrom") or {} cells = seeded.get("roundCells") or {} responses = resp.get("responses") or {} @@ -787,7 +802,7 @@ def page_decided(q): ) rows = [] - for r in register(doc, resp, extra, numbers): + for r in register(doc, resp, pos): q = r["q"] rnd, added = q.get("round") or 1, q.get("addedInRound") page = { @@ -849,6 +864,19 @@ def page_decided(q): return doc, rows, notes +def register_lines(rows, gaps): + """The register's lines: each row's, with each declared gap line (gap_lines' tuples) right + before the first row numbered past its range, in range order, and after the last row when + none is.""" + gaps = sorted(gaps, key=lambda g: (g[1], g[2])) + out = [] + for r in rows: + while gaps and gaps[0][1] < int(r["n"][1:]): + out.append(gaps.pop(0)[3]) + out.append(row_line(r)) + return out + [g[3] for g in gaps] + + def export_ledger(d, text=None): doc, rows, _ = merged_register(d, text) title = (doc.get("meta") or {}).get("title") @@ -860,12 +888,7 @@ def export_ledger(d, text=None): mark = " " if r["status"] in UNSETTLED else "x" out.append(f"- [{mark}] {r['n']} {r['short']}: {r['status']}") out += ["", "## Open-question register", ""] - gaps = gap_lines(text or "") - for r in rows: - while gaps and gaps[0][0] < int(r["n"][1:]): - out.append(gaps.pop(0)[1]) - out.append(row_line(r)) - out += [line for _, line in gaps] + out += register_lines(rows, gap_lines(text or "")) out += ["", "### Deferred questions", ""] retired = [r for r in rows if r["status"] in ("deferred", "blocked")] out += [f"- {r['n']}: {r['title']}" for r in retired] or ["- none"] @@ -955,19 +978,21 @@ def check_ledger(text, where): def sync_ledger(d, text, where): - """(text, notes): the ledger's text with its live register rows, and nothing else, replaced - by merged_register's rows, written where the first row stood. A register with no rows yet gets + """(text, notes): the ledger's text with its live register rows and declared gap lines, and + nothing else, replaced by merged_register's rows and those gap lines placed as register_lines + places them, written where the first of them stood. A register with no rows yet gets them under its heading, with a blank line before any text that follows, which would otherwise read as part of the last row's list item.""" lines = text.splitlines(keepends=True) heads, found = check_ledger(text, where) _, rows, notes = merged_register(d, text) - drop = {i for i, _ in found} + gaps = gap_lines(text) + drop = {i for i, _ in found} | {g[2] for g in gaps} at = min(drop) if drop else heads[0] + 1 if not drop and at < len(lines) and not lines[at].strip(): at += 1 eol = "\r\n" if lines and lines[0].endswith("\r\n") else "\n" - new = [row_line(r) + eol for r in rows] + new = [line + eol for line in register_lines(rows, gaps)] if new and not drop and at < len(lines) and lines[at].strip(): new.append(eol) keep = [line for i, line in enumerate(lines) if i not in drop] @@ -1335,9 +1360,8 @@ def export_brief(d, text=None): old = ledger_register(text) if text is not None else {} page_ids = {q["id"] for q in doc.get("questions") or []} extra = [i for i in old if i not in page_ids] - numbers = ledger_numbers(text) - rows = register(doc, resp, extra, numbers) - pos = numbering(sorted(page_ids) + doc["pendingProposals"] + extra, numbers) + pos = page_numbering(doc, extra, text) + rows = register(doc, resp, pos) for i in extra: status, _, title, res = old[i] rows.append(ledger_row(f"Q{pos[i]}", i, status, title, res)) @@ -1778,24 +1802,35 @@ def scan_register(lines, row=ROW): return heads, rows +def parse_gap(line): + """(low, high) of a `- gap | Q-Q | ` line in the grammar check-open-questions.sh + accepts (no leading zero, a <= b, a reason), else None.""" + m = GAP_LINE.match(line) + ends = (int(m.group(1)), int(m.group(2))) if m else None + return ends if ends and ends[0] <= ends[1] else None + + def gap_lines(text): - """[(highest id it declares, line)] of a ledger's `- gap |` lines, in file order.""" + """[(low, high, line index, line)] of each well-formed `- gap |` line in a ledger's register + section outside fenced blocks, in file order.""" lines = text.splitlines() - return [(int(m.group(1)), lines[i].strip()) for i, m in scan_register(lines, GAP_LINE)[1]] + found = [(parse_gap(lines[i]), i) for i, _ in scan_register(lines, GAP_LINE)[1]] + return [(*ends, i, lines[i].strip()) for ends, i in found if ends] def highest_id(text, d=None): - """The highest question number a ledger's register rows and `- gap |` lines hold, and with a - data dir every number its merged export writes and every pending proposal's Q; 0 for - none.""" - lines = text.splitlines() - nums = [int(m.group(1)) for _, m in scan_register(lines)[1]] - nums += [n for n, _ in gap_lines(text)] + """The highest question number a ledger holds: its register rows' Q, each `[Q]` lead + and each `- gap |` line's high end; with a data dir also every number its merged export + writes, every page question's own Q and every pending proposal's. 0 for none.""" + ids = [] + for n, _, _, _, res, _ in parse_register(text): + ids += [f"Q{n}", row_id(n, res)] + nums = [b for _, b, _, _ in gap_lines(text)] if d is not None: doc, rows, _ = merged_register(d, text) nums += [int(r["n"][1:]) for r in rows] - nums += [int(i[1:]) for i in doc["pendingProposals"] if QN.match(i)] - return max(nums, default=0) + ids += [q["id"] for q in doc["questions"]] + doc["pendingProposals"] + return max(nums + [int(i[1:]) for i in ids if QN.match(i)], default=0) def parse_register(text): diff --git a/plugins/planning/surface/test_exporters.py b/plugins/planning/surface/test_exporters.py index 6aad4c4010f..200c5cc9e85 100644 --- a/plugins/planning/surface/test_exporters.py +++ b/plugins/planning/surface/test_exporters.py @@ -344,6 +344,101 @@ def test_export_ledger_keeps_each_hand_written_gap_line_in_place(self): self.assertEqual(rc, 1, verdict) self.assertIn("registered=3 open=3", verdict) + GAPPED = ( + "# Interview ledger\n\n## Open-question register\n\n" + "- Q1 | open | round 1 | Question Q1? |\n" + "- gap | Q2-Q9 | the shared session's ids ran ahead\n" + ) + + def test_a_new_id_never_lands_in_a_declared_gap(self): + self.session([question("Q1"), question("J1")], []) + _, rows, _ = exporters.merged_register(self.dir, self.GAPPED) + self.assertEqual([(r["n"], r["id"]) for r in rows], [("Q1", "Q1"), ("Q10", "J1")]) + self.assertTrue(rows[1]["resolution"].startswith("[J1]"), rows[1]) + self.assertEqual( + exporters.numbering(["Q1", "J1"], {"Q1": 1}, [(2, 9)]), {"Q1": 1, "J1": 10} + ) + + def test_a_page_q_id_inside_a_declared_gap_is_numbered_after_it(self): + self.session([question("Q1"), question("Q5")], []) + _, rows, _ = exporters.merged_register(self.dir, self.GAPPED) + self.assertEqual([(r["n"], r["id"]) for r in rows], [("Q1", "Q1"), ("Q10", "Q5")]) + self.assertTrue(rows[1]["resolution"].startswith("[Q5]"), rows[1]) + + def test_export_ledger_keeps_a_gap_line_past_the_last_row_and_the_gate_refuses_it(self): + self.session([question("Q1"), question("Q2")], []) + ledger = self.tmp / "ledger.md" + ledger.write_text( + "# Interview ledger\n\n## Open-question register\n\n" + "- Q1 | open | round 1 | Question Q1? |\n" + "- Q2 | open | round 1 | Question Q2? |\n" + "- gap | Q3-Q5 | ids a peer session holds\n", + encoding="utf-8", + newline="\n", + ) + out = self.tmp / "exported.md" + rc, text = self.rp("export-ledger", "--ledger", str(ledger), "--out", str(out)) + self.assertEqual(rc, 0, text) + register = out.read_text(encoding="utf-8").split("## Open-question register")[1] + lines = [x for x in register.split("###")[0].splitlines() if x.startswith("- ")] + self.assertEqual(lines[-1], "- gap | Q3-Q5 | ids a peer session holds") + self.assertEqual([x.split(" | ")[0] for x in lines], ["- Q1", "- Q2", "- gap"]) + rc, verdict = self.check("--ledger", out) + self.assertEqual(rc, 2, verdict) + self.assertIn("runs past the highest row Q2", verdict) + + def test_sync_ledger_places_each_gap_line_by_its_range(self): + self.session([question("Q1"), question("Q5"), question("Q6")], []) + ledger = self.tmp / "ledger.md" + ledger.write_text( + "# Interview ledger\n\n## Open-question register\n\n" + "- Q1 | open | round 1 | Question Q1? |\n" + "- gap | Q2-Q4 | dropped before the page\n" + "- Q5 | open | round 1 | Question Q5? |\n\n" + "## Notes\n\nkept as written\n", + encoding="utf-8", + newline="\n", + ) + rc, text = self.rp("sync-ledger", "--ledger", str(ledger)) + self.assertEqual(rc, 0, text) + synced = ledger.read_text(encoding="utf-8") + register = synced.split("## Open-question register")[1].split("## Notes")[0] + lines = [x for x in register.splitlines() if x.startswith("- ")] + self.assertEqual( + [x.split(" | ")[0] for x in lines], ["- Q1", "- gap", "- Q5", "- Q6"] + ) + self.assertIn("- gap | Q2-Q4 | dropped before the page", lines) + self.assertTrue(synced.endswith("## Notes\n\nkept as written\n"), synced) + rc, verdict = self.check("--ledger", ledger) + self.assertEqual(rc, 1, verdict) + + GATE_GAP_LINES = { + "- gap | Q2-Q4 | the shared session's ids ran ahead": (2, 4), + "- gap | Q2-Q3 | dropped before the page": (2, 3), + "- gap | Q2-Q2 | an example": (2, 2), + "- gap | Q2 | no range": None, + "- gap | Q3-Q2 | reversed": None, + "- gap | Q2-Q2 |": None, + "- gap | Q02-Q3 | a leading zero": None, + } + + def test_the_gap_line_grammar_matches_the_gate(self): + for line, ends in self.GATE_GAP_LINES.items(): + with self.subTest(line=line): + self.assertEqual(exporters.parse_gap(line), ends) + ledger = self.tmp / "gate-gap.md" + ledger.write_text( + "# Interview ledger\n\n## Open-question register\n\n" + "- Q1 | answered | round 1 | One? | yes\n" + f"{line}\n" + "- Q5 | answered | round 1 | Five? | yes\n", + encoding="utf-8", + newline="\n", + ) + _, verdict = self.check("--ledger", ledger) + self.assertEqual("malformed gap line" in verdict, ends is None, verdict) + self.assertIsNone(exporters.parse_gap("- gap, see below")) + def test_a_q_id_new_to_the_ledger_keeps_its_number(self): self.session([question("Q1"), question("Q2"), question("Q600")], []) _, rows, _ = exporters.merged_register( diff --git a/plugins/planning/surface/test_round.py b/plugins/planning/surface/test_round.py index 17b1569b0f1..78b9ed41d25 100644 --- a/plugins/planning/surface/test_round.py +++ b/plugins/planning/surface/test_round.py @@ -4434,6 +4434,26 @@ def test_the_data_dir_counts_its_page_ids_and_their_export_numbers(self): self.assertEqual(rc, 0, out + err) self.assertEqual(out, "Q5\n") + def test_a_renumbered_row_lead_is_never_reused(self): + ledger = self.ledger( + "- Q1 | answered | round 1 | Seven? | [Q7] free-text: first\n" + "- Q2 | answered | round 1 | Jay? | [J1] free-text: second\n" + ) + rc, out, err = self.next_id("--ledger", ledger) + self.assertEqual(rc, 0, out + err) + self.assertEqual(out, "Q8\n") + + def test_a_pending_proposal_id_is_never_reused(self): + doc = base_doc() + doc["questions"].append(question("Q9", group="g2", round=1, proposed=True)) + self.write_doc(doc) + ledger = self.ledger( + "- Q1 | open | round 1 | One? |\n- Q2 | open | round 1 | Two? |\n" + ) + rc, out, err = self.next_id("--ledger", ledger, "--dir", str(self.dir)) + self.assertEqual(rc, 0, out + err) + self.assertEqual(out, "Q10\n") + def test_a_file_with_no_register_heading_is_refused(self): path = self.tmp / "notes.md" path.write_bytes(b"# Notes\n\n- Q1 | open | round 1 | One? |\n") From 9afd621478e30ff1cee9e8453487a9c84472d86d Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 21 Nov 2026 05:22:34 -0500 Subject: [PATCH 04/30] feat(planning): register tags and round.sh reopen A register row's resolution can now end in five fixed-shape tags, and an answer a later fact contradicts reopens under its own number with every row built on it flagged (Phase 4 of the interview redesign plan). - exporters.py: one anchored pattern (TAG_TAIL, beside STALE_TAIL) peels a tail of `kind:: discovery`, `problem:: P`, `depends:: Q[, ...]`, `reopened:: round ` and `recheck:: Q[, ...]` from the right of a stale-free resolution (split_tags; join_tags writes it back). The tags are not in the ordered named-field list. parse_register still returns the cell with its tail, so rows copied as they stand (ledger-only rows, a kept conflict) keep it; each parse point peels it first. import_ledger keeps each row's tail in meta.seededFrom.tags (declared in questions.schema.json); register() writes it back after the answer and before `stale::`. A merge into a ledger keeps the ledger's own tail for a row it holds, so a tag the terminal lane wrote after an import (a reopen, a --rechecked) is not undone by the next sync. - edit_row is split into locate + rewrite_row, which peels the tail before change() and re-appends it, so supersede and resolve keep tags (before, a named row with a tag was refused as an unknown field). - reopen_row / round.sh reopen --ledger L Q --round --fact-file P: answered row -> open with `reopened:: round `; `- Q | round | was: | fact: ` appended to `## Reopened answers` (created right after the register when absent); `recheck:: Q` merged into every row whose depends:: reaches Q at any depth (visited set, so a cycle ends); prints `marked Q` per row. `--rechecked Q...` (clear_recheck) removes recheck:: and refuses a row with none. Refuses a row that is not answered and an empty fact. Follows the cmd_supersede -> supersede_row precedent: no data dir, backup, gate verdict. - check-open-questions.sh: a resolution ending in a tail with recheck:: (before any stale::) counts as `recheck=` and blocks like stale::; a settled row whose resolution is only tags has an empty resolution under --procedure; rows_outside skips a `## Reopened answers` section, so its `- Q | round | ...` lines are not rows outside the register. `- P |` rows in `## Problems` were never graded and still are not. - check-round-codes.sh: `- P |` rows join the ledger codes. - The two scripts' header comments are not extended for the tags, the recheck count or P codes: the pre-commit comment lint refuses new comment lines, so this message and surface/README.md carry the contract. - Phase 3 carry-over (a): export_ledger and sync_ledger both refuse a gap line the gate reads as malformed (gap_lines with a writer's `where`); before, export dropped it and sync kept it in place. - Phase 3 carry-over (b): surface/README.md's next-id row lists every source highest_id counts ([Q] leads, page question ids); new reopen row; the import-ledger row names the tag tail. - templates/checklist.md: `## Problems` and `## Reopened answers` sections; the register header names the tags and points at loop.md. Not in this commit (reported to the orchestrator, blocked by existing tests the plan does not name as intended reversals): the exporter's `- gap | Q-Q | never assigned` lines (new D2) and carry-over (c), the trailing gap line the gate and numbering disagree on. Sanity check 1 as written (register byte-identical after import and export) does not hold for a plain-text answer: the exporter writes every row in its named grammar, so `free-text: staff; then contractors` comes back as `answer:: free-text: staff\; then contractors`, as it did before this phase. Every tag, every named-field row and `## Reopened answers` round-trip byte for byte, and the exported ledger is a fixed point. Pre-flight: parsers of the resolution cell and the tail - exporters.py parse_register: drops stale, keeps the tag tail in the cell (rows written back as they stand keep it). - exporters.py ledger_register / ledger_numbers / highest_id / row_id: read the [] lead at the start and the row number; the tail is opaque to them. - exporters.py merged_register: compares and copies cells whole; takes each held row's tail from the ledger (split_tags) for register(). - exporters.py import_ledger: LEAD, then split_tags, then parse_named / parse_hold / split_tail / PROPOSES on the peeled answer. - exporters.py parse_named: called only on peeled text (import_ledger, recap_answer, ledger_row, displaced_answer and resolve_row through rewrite_row). - exporters.py drop_stale: unchanged; runs first, tags sit before stale. - exporters.py LEAD / REGISTER_LEAD (split_lead, locate_row): the lead is at the start; split_tags keeps it on the resolution. - exporters.py encode: writes named fields only; register() appends the tail after it (join_tags). - exporters.py recap_answer, ledger_row: peel before parse_named and readable(), so a recap or Brief line shows no tags. - check-open-questions.sh: stale_tail_re unchanged (the tail is before stale); recheck_tail_re and tags_only_re read the tail. - skills/audit-answers/SKILL.md: reads rows by status and a `hedged:` substring in prose; opaque text, unaffected. - skills/wayfind/SKILL.md: seeds from the checklist's settled answers in prose; opaque text. - plugins/work-items/skills/decompose/SKILL.md: reads answered Q rows in prose; opaque text. Tests added - test_exporters.py TestRegisterTags: test_split_tags_peels_only_a_tail_of_whole_tags, test_import_keeps_each_tail_beside_the_answer, test_an_import_then_export_writes_every_tag_back_byte_for_byte, test_an_import_then_sync_leaves_the_register_and_reopened_answers_byte_identical, test_a_merge_keeps_the_ledgers_newer_tail, test_supersede_and_restore_keep_the_tail_on_a_named_row, test_the_brief_reads_a_ledger_only_row_without_its_tags, test_a_recap_reads_the_answer_without_its_tags. - test_exporters.py TestExportLedger: test_export_ledger_refuses_a_malformed_gap_line, test_sync_ledger_refuses_a_malformed_gap_line. - test_round.py TestReopen: test_a_reopen_flags_every_row_built_on_it_three_deep, test_rechecked_clears_only_the_named_rows, test_a_second_reopen_adds_to_the_reopened_answers_section, test_only_an_answered_row_reopens, test_a_reopen_lists_its_entry_under_the_sections_own_text, test_a_reopen_keeps_crlf_line_endings. - check-open-questions.test.sh: tags, Problems rows and Reopened answers rows grade clean; recheck rows counted beside stale (escaped `\;` not a mark); a recheck row blocks; cleared recheck is clean; tags-only resolution is empty under --procedure. - check-round-codes.test.sh: a bare problem id fails; a named one passes. No existing test changed. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh --- .../feat-interview-redesign-07635b38.md | 1 + .../planning/scripts/check-open-questions.sh | 15 +- .../scripts/check-open-questions.test.sh | 56 ++++ plugins/planning/scripts/check-round-codes.sh | 2 +- .../scripts/check-round-codes.test.sh | 29 ++ .../skills/interview/templates/checklist.md | 12 +- plugins/planning/surface/README.md | 5 +- plugins/planning/surface/exporters.py | 255 +++++++++++++++--- plugins/planning/surface/round.py | 65 ++++- .../surface/schema/questions.schema.json | 5 + plugins/planning/surface/test_exporters.py | 158 +++++++++++ plugins/planning/surface/test_round.py | 139 ++++++++++ 12 files changed, 700 insertions(+), 42 deletions(-) diff --git a/.changes/planning/feat-interview-redesign-07635b38.md b/.changes/planning/feat-interview-redesign-07635b38.md index e8ad786bd6f..e9cedfbaa14 100644 --- a/.changes/planning/feat-interview-redesign-07635b38.md +++ b/.changes/planning/feat-interview-redesign-07635b38.md @@ -5,6 +5,7 @@ bump: minor ### Added - **Stable question ids.** `round.sh next-id --ledger [--dir ] [--count ]` prints the next free ids after the highest one held, and the ledger export keeps every number a ledger row already has instead of renumbering. `check-open-questions.sh` accepts a gap in ids only where a hand-written `- gap | Q-Q | ` line declares it, and `export-ledger --ledger` keeps those lines. +- **Register tags and `round.sh reopen`.** A register row's resolution can end in `kind::`, `problem::`, `depends::`, `reopened::` and `recheck::` tags, which import and export carry unchanged. `round.sh reopen --ledger Q --round --fact-file ` reopens an answered row under its own number, keeps the old answer and the fact in `## Reopened answers`, and flags every row built on it with `recheck::`, which blocks `check-open-questions.sh` until `reopen --rechecked` clears it. ### Removed diff --git a/plugins/planning/scripts/check-open-questions.sh b/plugins/planning/scripts/check-open-questions.sh index 88865663e04..9331bb2bb59 100755 --- a/plugins/planning/scripts/check-open-questions.sh +++ b/plugins/planning/scripts/check-open-questions.sh @@ -244,7 +244,9 @@ rows_outside() { awk -v lo="$1" -v hi="$2" "$fence_awk"' fence_line($0) { next } fenced { next } + /^#+[[:space:]]/ { reopened = (tolower($0) ~ /^#+[[:space:]]+reopened answers[[:space:]]*$/) } NR >= lo && NR <= hi { next } + reopened { next } /^[[:space:]]*-[[:space:]]+[Qq][0-9]+[[:space:]]*[|]/ { print NR } ' "$3" } @@ -746,7 +748,13 @@ fi registered=0 open_count=0 stale_count=0 -stale_tail_re='(^|\] |(^|[^\\])(\\\\)*; )stale:: C[0-9]+(, C[0-9]+)*$' +recheck_count=0 +stale_list='stale:: C[0-9]+(, C[0-9]+)*' +stale_tail_re='(^|\] |(^|[^\\])(\\\\)*; )'"$stale_list"'$' +qlist='Q[1-9][0-9]*(, Q[1-9][0-9]*)*' +tag_re='(kind:: discovery|problem:: P[1-9][0-9]*|depends:: '"$qlist"'|reopened:: round [1-9][0-9]*|recheck:: '"$qlist"')' +recheck_tail_re='(^|\] |(^|[^\\])(\\\\)*; )('"$tag_re"'; )*recheck:: '"$qlist"'(; '"$tag_re"')*(; '"$stale_list"')?$' +tags_only_re='^(\[[^]]*\][[:space:]]*)?'"$tag_re"'(; '"$tag_re"')*(; '"$stale_list"')?$' answered=0 deferred=0 withdrawn=0 @@ -885,6 +893,8 @@ while IFS= read -r line; do trim_into "$resolution" resolution="$trimmed" [[ "$resolution" =~ $stale_tail_re ]] && stale_count=$((stale_count + 1)) + [[ "$resolution" =~ $recheck_tail_re ]] && recheck_count=$((recheck_count + 1)) + [[ "$resolution" =~ $tags_only_re ]] && resolution="" if [[ "$round_field" =~ ^[Rr]ound[[:space:]]+([1-9][0-9]*)$ ]]; then round_num="${BASH_REMATCH[1]}" rounds_seen="$rounds_seen$round_num " @@ -1165,8 +1175,9 @@ fi verdict="registered=$registered open=$open_count deferred=$deferred blocked=$blocked withdrawn=$withdrawn answered=$answered superseded=$superseded brief=$brief_state" [[ "$stale_count" -gt 0 ]] && verdict="$verdict stale=$stale_count" +[[ "$recheck_count" -gt 0 ]] && verdict="$verdict recheck=$recheck_count" -if [[ $((open_count + superseded + stale_count)) -gt 0 ]]; then +if [[ $((open_count + superseded + stale_count + recheck_count)) -gt 0 ]]; then echo "$verdict status=open procedure=$procedure_state" exit 1 fi diff --git a/plugins/planning/scripts/check-open-questions.test.sh b/plugins/planning/scripts/check-open-questions.test.sh index 156b25ad08f..778f23ebcee 100755 --- a/plugins/planning/scripts/check-open-questions.test.sh +++ b/plugins/planning/scripts/check-open-questions.test.sh @@ -114,6 +114,62 @@ EOF expect_exit "a mark after an escaped backslash still blocks -> 1" 1 --ledger "$stale_backslash" expect_stdout "open verdict counts the open row" "open=1" --ledger "$one_open" +tagged="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | a | free-text: x; y; kind:: discovery; problem:: P1 +- Q2 | answered | round 1 | b | answer:: accepted: yes; depends:: Q1; reopened:: round 2 +- Q3 | answered | round 2 | c | [J1] accepted: yes; depends:: Q1, Q2; stale:: C1 + +## Problems + +- P1 | staff cannot find the queue | agreed + +## Reopened answers + +- Q2 | round 2 | was: accepted: no | fact: ops.md:3 says yes +EOF +)" +expect_exit "tags, Problems rows and Reopened answers rows -> 1 only for the stale row" 1 --ledger "$tagged" +expect_stdout "tags are not counted as recheck" "superseded=0 brief=unchecked stale=1 status=open" --ledger "$tagged" +recheck_row="$( + mkledger <<'EOF' +- Q1 | open | round 1 | a | reopened:: round 2 +- Q2 | answered | round 1 | b | accepted: yes; depends:: Q1; recheck:: Q1 +- Q3 | answered | round 1 | c | answer:: free-text: C:\\; recheck:: Q1; stale:: C2 +- Q4 | answered | round 1 | d | answer:: said\; recheck:: Q1 +EOF +)" +expect_stdout "recheck rows are counted beside stale" "stale=1 recheck=2 status=open" --ledger "$recheck_row" +rechecked="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | a | accepted: yes; reopened:: round 2 +- Q2 | answered | round 1 | b | accepted: yes; depends:: Q1; recheck:: Q1 +EOF +)" +expect_exit "a recheck row on an answered register blocks -> 1" 1 --ledger "$rechecked" +cleared="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | a | accepted: yes; reopened:: round 2 +- Q2 | answered | round 1 | b | accepted: yes; depends:: Q1 +EOF +)" +expect_exit "once recheck:: is cleared the register is clean -> 0" 0 --ledger "$cleared" +tags_only="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | a | depends:: Q2 +- Q2 | answered | round 1 | b | [J1] kind:: discovery +- Q3 | answered | round 1 | c | yes; kind:: discovery +EOF +)" +tags_only_err="$(stderr_of --ledger "$tags_only" --procedure)" +if [[ "$tags_only_err" == *"Q1: answered row has an empty resolution"* && + "$tags_only_err" == *"Q2: answered row has an empty resolution"* && + "$tags_only_err" != *"Q3: answered"* ]]; then + pass "a resolution of tags alone is empty under --procedure" +else + fail "a resolution of tags alone is empty under --procedure ($tags_only_err)" +fi + # 8. Mixed terminal statuses all count as resolved. mixed="$( mkledger <<'EOF' diff --git a/plugins/planning/scripts/check-round-codes.sh b/plugins/planning/scripts/check-round-codes.sh index 9d2064863d8..99c1607e5e4 100755 --- a/plugins/planning/scripts/check-round-codes.sh +++ b/plugins/planning/scripts/check-round-codes.sh @@ -168,7 +168,7 @@ LC_ALL=C awk ' if (fence($0) || in_fence) next } FILENAME == ledger_file { - if ($0 ~ /^[ \t]*- [QCFS][0-9]+[ \t]*\|/) { + if ($0 ~ /^[ \t]*- [QCFSP][0-9]+[ \t]*\|/) { c = $0 sub(/^[ \t]*- /, "", c) sub(/[ \t]*\|.*/, "", c) diff --git a/plugins/planning/scripts/check-round-codes.test.sh b/plugins/planning/scripts/check-round-codes.test.sh index f5cc84733b5..0dc17ef3a52 100755 --- a/plugins/planning/scripts/check-round-codes.test.sh +++ b/plugins/planning/scripts/check-round-codes.test.sh @@ -199,6 +199,35 @@ else fail "a scenario id shown with nothing beside it is bare ($out)" fi +cat >"$TMP/problem-ledger.md" <<'MD' +## Problems + +- P1 | staff cannot find the queue | agreed + +## Open-question register + +- Q1 | open | round 1 | Which queue? | problem:: P1 +MD +round problem-bare.md \ + 'Q2: Does the alert fix it?' \ + 'Context: this serves P1 and Q1 (the queue question).' +out="$(bash "$SUT" --ledger "$TMP/problem-ledger.md" "$TMP/problem-bare.md" 2>&1)" +got=$? +if [[ "$got" -eq 1 ]] && grep -q '^bare=2:P1$' <<<"$out" && grep -q 'codes=3 bare=1 status=fail' <<<"$out"; then + pass "a bare problem id fails" +else + fail "a bare problem id fails ($out)" +fi +round problem-named.md \ + 'Q2: Does the alert fix it?' \ + 'Context: this serves P1, staff cannot find the queue, and Q1 (the queue question).' +if out="$(bash "$SUT" --ledger "$TMP/problem-ledger.md" "$TMP/problem-named.md" 2>&1)" && + grep -q 'bare=0 status=ok' <<<"$out"; then + pass "a named problem id passes" +else + fail "a named problem id passes ($out)" +fi + if [[ "$fails" -gt 0 ]]; then printf '%d failure(s)\n' "$fails" >&2 exit 1 diff --git a/plugins/planning/skills/interview/templates/checklist.md b/plugins/planning/skills/interview/templates/checklist.md index 86604d2802e..0936382c840 100644 --- a/plugins/planning/skills/interview/templates/checklist.md +++ b/plugins/planning/skills/interview/templates/checklist.md @@ -28,6 +28,12 @@ One line: what the user disclosed at intake about where they start (SKILL.md Sta - +## Problems + +One row per problem the questions serve, `- P | | proposed|agreed`, numbered from P1; a question names its row with `problem:: P`. Replace the example row with this run's problems. + +- P1 | instructors cannot see which comments need a reply | agreed + ## Terminology matrix One row per sourced term the rounds rely on, built and agreed with the user before the first decision question (a definition the session coins gets no row and is agreed in a withheld question); a later round that adds a sourced term adds a `proposed` row and gets it agreed. Status: `proposed` | `agreed`. With no such term, replace the table with `none`. Rules: `context/loop.md` "Terminology matrix". Replace the example row with this run's terms. @@ -38,7 +44,7 @@ One row per sourced term the rounds rely on, built and agreed with the user befo ## Open-question register -**Write a row the moment a round is asked, before any reply arrives.** The register is a byproduct of asking, not of resolving: a question that only lands on disk once it is answered cannot record the failure of never being answered. Statuses: `open` | `discussing` (the user asked to talk it through; blocks like `open`; `context/loop.md` "Discussing") | `answered` | `deferred` | `withdrawn` | `blocked` | `superseded-by-plan` (set by `/planning:plan` when a plan change displaces an answer, resolution `plan proposes: ; was: `; not terminal; row shape and how it clears: `context/loop.md` "Superseded by plan"). `Q` matches the terminal numbering and runs continuously across rounds; an intended gap needs a hand-written `- gap | Q-Q | ` line in this section. +**Write a row the moment a round is asked, before any reply arrives.** The register is a byproduct of asking, not of resolving: a question that only lands on disk once it is answered cannot record the failure of never being answered. Statuses: `open` | `discussing` (the user asked to talk it through; blocks like `open`; `context/loop.md` "Discussing") | `answered` | `deferred` | `withdrawn` | `blocked` | `superseded-by-plan` (set by `/planning:plan` when a plan change displaces an answer, resolution `plan proposes: ; was: `; not terminal; row shape and how it clears: `context/loop.md` "Superseded by plan"). `Q` matches the terminal numbering and runs continuously across rounds; an intended gap needs a hand-written `- gap | Q-Q | ` line in this section. A resolution may end in tags, in this order: `kind:: discovery`, `problem:: P`, `depends:: Q, Q`, `reopened:: round ` and `recheck:: Q` (`round.sh reopen` writes the last two; rules: `context/loop.md`). Each `Commits you to:` part of a recommendation is a row of its own with `(part of Q)` in the question field, numbered right after its headline when asked with it, and at the next free id when registered after later ids exist. Resolution rules: `context/loop.md` "Commitment rows"; late numbering: "Commitment rows written after later ids exist". On the page surface, the parts are the question's `commits` entries instead (see `context/loop.md` "Commitment parts on the page"). The page exports each row's resolution as named fields and every older grammar still imports (`context/surface.md` "Start and stop"). @@ -68,6 +74,10 @@ bash "${CLAUDE_PLUGIN_ROOT}/scripts/check-open-questions.sh" \ --brief //PLAN.md ``` +## Reopened answers + +`round.sh reopen` adds one line per answer a later fact reopened, `- Q | round | was: | fact: `; never write one by hand. + ## Self-answered facts One line per candidate question the self-answer step answered instead of asking: no register row, `- F | answered-from- | | | `. Rules: `context/loop.md` "Self-answer gate". diff --git a/plugins/planning/surface/README.md b/plugins/planning/surface/README.md index b2808ce64df..0bd5e9ed4f2 100644 --- a/plugins/planning/surface/README.md +++ b/plugins/planning/surface/README.md @@ -85,8 +85,9 @@ Every command needs `--dir ''`; there is no default. Every command exi | `sync-ledger --ledger L` | Rewrite only L's register rows from page state, merged as `export-ledger --ledger` merges; prose, fenced blocks and other sections stay as they are. Then records L's absolute path in `meta.syncedLedger` and runs the `ledger-check` comparison against it, so the header's ledger line shows. Refuses a ledger without exactly one register heading; exit 6 on a conflict | | `supersede --ledger L Q --proposes-file P` | Set L's answered register row to `superseded-by-plan`, proposing the text in file P (UTF-8, surrounding whitespace dropped; an empty one is refused): the row's answer becomes `was` (a named-field row's `answer` field, as the exporter writes it; a named row that also carries a `note`, `hold`, `aside` or commitments is refused), written `was plain::` or `was answer::` to record which form the row had, and both values are escaped by `esc_field`. Needs no data dir or page. Backs L up (`L..bak`), rewrites only that row, and prints the `check-open-questions.sh` verdict; refuses a row that is not `answered` | | `resolve --ledger L Q reconfirm\|restore [--note-file N]` | Clear a superseded row: `reconfirm` writes the answered `proposal:: ...; was:: ...; answer:: accepted: ...` form (file N holds the accept's note; `restore` refuses it), `restore` puts back the row `supersede` replaced, character for character, in the form its `was` names; a bare `was::` from an older ledger is guessed: in a ledger already in the named grammar an answer-shaped `was` comes back as `answer:: ...`, else as the plain cell. Same backup and verdict; refuses a row that is not `superseded-by-plan` | -| `import-ledger --ledger F` | Seed an empty data dir from an existing ledger. A round cell reads as the N of a leading `round `, else the cell's first integer, else 1, with a warning naming the row when the leading form is missing; a cell other than the bare `round ` is kept and written back by `export-ledger` while the question's round still equals its number. A row in the `export-ledger` grammar (named fields `hold`, `proposal`, `was` (or `was plain`, `was answer`, which `export-ledger` writes back), `answer`, `note`, `aside`, `commitments`, each escaped as `exporters.py` documents) restores its hold, a superseded-by-plan row's proposal and displaced answer, the decision on any status, its note, a decision a user hold set aside (still set aside), an open row's accept that a revised recommendation set aside (still set aside, as of the text it accepted) and every commitment with its confirmed or unconfirmed mark; an unknown field or a contradictory pair is refused, naming the field. Two held-row cases have no field and are not restored: a seeded `blocked` row comes back `deferred` (still USER-RESERVED), and the seed text of an open or superseded-by-plan row is lost when its hold set aside an accept or an alternative, whose own note takes `note`. Every older grammar still imports, read as it always was: `waits on::`, `awaiting user::`, `confirmed::`, `plan proposes::` and `answer::` rows, a settled row's `; confirmed::` tail, and the unescaped `waits on:`, `confirmed:` and `; confirmed:` forms | -| `next-id --ledger L [--count N]` | Print the next N question ids (default 1), one per line: the highest id L's register rows and `- gap \| Q-Q \| ` lines hold plus 1, never a reused or gap-filling id; with `--dir`, also every number the page's merged export writes and every pending proposal's `Q`. Needs no data dir; refuses an L without exactly one open-question register heading and a count below 1 | +| `import-ledger --ledger F` | Seed an empty data dir from an existing ledger. A round cell reads as the N of a leading `round `, else the cell's first integer, else 1, with a warning naming the row when the leading form is missing; a cell other than the bare `round ` is kept and written back by `export-ledger` while the question's round still equals its number. A row in the `export-ledger` grammar (named fields `hold`, `proposal`, `was` (or `was plain`, `was answer`, which `export-ledger` writes back), `answer`, `note`, `aside`, `commitments`, each escaped as `exporters.py` documents) restores its hold, a superseded-by-plan row's proposal and displaced answer, the decision on any status, its note, a decision a user hold set aside (still set aside), an open row's accept that a revised recommendation set aside (still set aside, as of the text it accepted) and every commitment with its confirmed or unconfirmed mark; an unknown field or a contradictory pair is refused, naming the field. Two held-row cases have no field and are not restored: a seeded `blocked` row comes back `deferred` (still USER-RESERVED), and the seed text of an open or superseded-by-plan row is lost when its hold set aside an accept or an alternative, whose own note takes `note`. Every older grammar still imports, read as it always was: `waits on::`, `awaiting user::`, `confirmed::`, `plan proposes::` and `answer::` rows, a settled row's `; confirmed::` tail, and the unescaped `waits on:`, `confirmed:` and `; confirmed:` forms. A row's tail of register tags (`kind::`, `problem::`, `depends::`, `reopened::`, `recheck::`, after the answer and before `stale::`) is peeled before any of this parsing and kept in `meta.seededFrom.tags`; `export-ledger` writes it back unchanged, and a merge into a ledger keeps that ledger's own tail | +| `next-id --ledger L [--count N]` | Print the next N question ids (default 1), one per line: the highest number L holds plus 1, never a reused or gap-filling id. L holds each register row's `Q`, each `[Q]` lead a renumbered row carries, and each `- gap \| Q-Q \| ` line's high end; with `--dir`, also every number the page's merged export writes, every page question's own `Q` id and every pending proposal's `Q`. Needs no data dir; refuses an L without exactly one open-question register heading and a count below 1 | +| `reopen --ledger L Q --round R --fact-file P` | Set L's answered row `Q` back to `open` under its own number with `reopened:: round R`, move its answer and the fact in file P (UTF-8, whitespace runs made one space; an empty one is refused) to L's `## Reopened answers` section as `- Q \| round R \| was: \| fact: ` (the section is created right after the register when L has none), and add `recheck:: Q` to every row whose `depends::` reaches `Q` at any depth, printing `marked Q` for each. `reopen --ledger L --rechecked Q...` removes `recheck::` from those rows once the user has reconfirmed or changed each; it refuses a row with none. Needs no data dir; same backup and verdict as `supersede`; refuses a row that is not `answered` | | `doctor --ledger L` | Report, writing nothing, what the running version needs and L or the page lacks: a `missing:` line for each of L's `## Constraint ledger` and `## Open-question register` sections and for open questions without a Basis or a `Checked against:` line in `facts`, and a `note:` when L's `Planning version:` line or `meta.pluginVersion` (the version that first wrote the data dir) names another version; exit 1 on any `missing:` line. `--dir` need not exist yet | | `lease [--release \| --takeover]` | Print the watcher holding the lease, or `no lease`; `--release` clears it; `--takeover` clears it and runs `watch.sh` for this session (run it as the background task after `/clear`), and the old watcher exits 3 with a `handed over, not a failure` message | diff --git a/plugins/planning/surface/exporters.py b/plugins/planning/surface/exporters.py index e1d60a97c50..37d602ce38f 100644 --- a/plugins/planning/surface/exporters.py +++ b/plugins/planning/surface/exporters.py @@ -4,10 +4,12 @@ export_ledger the interview ledger: a decision-tree checklist, then the open-question register in the row shape scripts/check-open-questions.sh grades, then the deferred questions; given a ledger's text it merges into that ledger's register (merged_register) - and keeps its well-formed `- gap |` lines, in range order, each right before - the first row numbered past its range (after the last row when none is) + and keeps its `- gap |` lines, in range order, each right before + the first row numbered past its range (after the last row when none is); + it refuses a gap line the gate would read as malformed sync_ledger a ledger's text with only its register rows and gap lines replaced by the - merged rows and those gap lines, placed as export_ledger places them + merged rows and those gap lines, placed as export_ledger places them; it + refuses a malformed gap line as export_ledger does export_brief the PLAN.md `## Brief` sections and an empty `## Plan`; given a ledger's text it numbers as that ledger and carries its ledger-only rows. Problem, Goal, Work size, Constraints, Acceptance criteria and Out-of-scope come from the newest confirmed @@ -48,6 +50,11 @@ is not the bare `round ` in meta.seededFrom.roundCells, which export writes back while the question's round still equals the cell's number, so an imported ledger round-trips byte for byte. +A resolution may end, before `stale::`, in a tail of register tags (TAG_NAMES: `kind:: discovery`, +`problem:: P`, `depends:: Q[, ...]`, `reopened:: round `, `recheck:: Q[, ...]`), +peeled from the right (split_tags) before any field is parsed. Import keeps each row's tail in +meta.seededFrom.tags and export writes it back; a merge into a ledger keeps the ledger's tail. + Register ids are never renumbered: a row the ledger merged into already holds keeps its number, a Q id new to that ledger keeps N unless a row has it or a declared gap covers it, and any other id takes the next number after the highest, in natural id order (letters, then number). @@ -137,6 +144,7 @@ GAP_LINE = re.compile( r"^\s*-\s+[Gg]ap\s*\|\s*[Qq]([1-9][0-9]*)\s*-\s*[Qq]([1-9][0-9]*)\s*\|\s*\S" ) +GAP_CANDIDATE = re.compile(r"^\s*-\s+[Gg]ap\s*\|") LEAD = re.compile(r"^\[([^\]\s]+)\]\s*(.*)$") FENCE = re.compile(r"^\s*(```|~~~)") MID_SENTENCE = re.compile(r"[.!?)\"'`\]]$") @@ -182,6 +190,14 @@ STALE_TAIL = re.compile( rf"(?:(?<=\] )|^){STALE_LIST}|((?:^|[^\\])(?:\\\\)*); {STALE_LIST}" ) +TAG_NAMES = ("kind", "problem", "depends", "reopened", "recheck") +QLIST = r"Q[1-9][0-9]*(?:, Q[1-9][0-9]*)*" +TAG = ( + rf"(?:kind:: discovery|problem:: P[1-9][0-9]*|depends:: {QLIST}" + rf"|reopened:: round [1-9][0-9]*|recheck:: {QLIST})" +) +TAG_LIST = rf"{TAG}(?:; {TAG})*$" +TAG_TAIL = re.compile(rf"^({TAG_LIST})|((?:^|[^\\])(?:\\\\)*); ({TAG_LIST})") def drop_stale(res): @@ -189,6 +205,44 @@ def drop_stale(res): return STALE_TAIL.sub(lambda m: m.group(1) or "", res).rstrip() +def split_tags(res): + """(resolution, tag tail) of a stale-free register resolution, its `[]` lead kept on the + resolution; the tail is "" when no tag ends it.""" + lead, body = split_lead(res) + m = TAG_TAIL.search(body) + if not m: + return res, "" + if m.group(1) is not None: + return lead.rstrip(), m.group(1) + return lead + body[: m.start()] + m.group(2), m.group(3) + + +def join_tags(res, tail): + """`res` with the tag tail `tail` after it, as split_tags reads it back.""" + if not tail: + return res + lead, body = split_lead(res) + if body: + return f"{res}; {tail}" + return f"{lead.rstrip()} {tail}" if lead else tail + + +def parse_tags(tail, where): + """{tag name: value} of a tag tail; refuses a tag written twice.""" + tags = {} + for part in split_fields(tail) if tail else (): + name, _, value = part.partition(":: ") + if name in tags: + refuse(f"tag {name + '::'!r} repeated", where) + tags[name] = value + return tags + + +def format_tags(tags): + """The tag tail of {tag name: value}, in TAG_NAMES order, empty values left out.""" + return "; ".join(f"{n}:: {tags[n]}" for n in TAG_NAMES if tags.get(n)) + + def clean(s): """One line with no register separators: pipes become slashes, all whitespace one space.""" return " ".join(str(s or "").replace("|", "/").split()) @@ -684,21 +738,24 @@ def lead_of(qid, n): return "" if f"Q{n}" == qid else f"[{clean(qid)}] " -def register(doc, resp, pos=None): +def register(doc, resp, pos=None, tails=None): """One dict per question, in register order, with its Q and settled status; `pos` is the - page_numbering() the caller already made, else the page's own.""" + page_numbering() the caller already made, else the page's own. Each row carries its tag tail + from `tails` ({id: tail}), else the one its import kept.""" qs = sorted(doc.get("questions") or [], key=lambda q: natural(q["id"])) pos = pos or page_numbering(doc) events = resp.get("events") or [] responses = resp.get("responses") or {} - seed_rows = ((doc.get("meta") or {}).get("seededFrom") or {}).get("rows") or {} + seeded = (doc.get("meta") or {}).get("seededFrom") or {} + seed_rows = seeded.get("rows") or {} + tails = {**(seeded.get("tags") or {}), **(tails or {})} unchecked = unchecked_constraints(doc) rows = [] for q in qs: status, fields, note, reserved = settle(q, responses, events, seed_rows) marked = marked_commits(q, events) lead = lead_of(q["id"], pos[q["id"]]) - res = encode(fields, marked) + res = join_tags(encode(fields, marked), tails.get(q["id"])) if q["id"] in unchecked: stale = f"stale:: {', '.join(unchecked[q['id']])}" res = f"{res}; {stale}" if res else stale @@ -777,7 +834,9 @@ def merged_register(d, text=None): ledger's title and its round cell while that cell still names the page's round; and a row the ledger settled that the page shows otherwise, with no decision of the page's own since any import, keeps the ledger's status and resolution, a conflict (a page decision since - then is newer than the ledger row and is written over it). notes are (kind, line) pairs, one per difference between the ledger + then is newer than the ledger row and is written over it). A row the ledger holds keeps the + ledger's tag tail, so a tag the terminal lane wrote after an import survives the merge. + notes are (kind, line) pairs, one per difference between the ledger and the merge or the page: kind `change` (the merge rewrites the ledger), `conflict` or `kept` (the merge keeps the ledger's text the page lacks).""" doc, resp = read(d) @@ -802,7 +861,8 @@ def page_decided(q): ) rows = [] - for r in register(doc, resp, pos): + tails = {i: split_tags(res)[1] for i, (_, _, _, res) in old.items()} + for r in register(doc, resp, pos, tails): q = r["q"] rnd, added = q.get("round") or 1, q.get("addedInRound") page = { @@ -877,7 +937,8 @@ def register_lines(rows, gaps): return out + [g[3] for g in gaps] -def export_ledger(d, text=None): +def export_ledger(d, text=None, where="the ledger"): + gaps = gap_lines(text or "", where) doc, rows, _ = merged_register(d, text) title = (doc.get("meta") or {}).get("title") out = ["# Interview ledger", ""] @@ -888,7 +949,7 @@ def export_ledger(d, text=None): mark = " " if r["status"] in UNSETTLED else "x" out.append(f"- [{mark}] {r['n']} {r['short']}: {r['status']}") out += ["", "## Open-question register", ""] - out += register_lines(rows, gap_lines(text or "")) + out += register_lines(rows, gaps) out += ["", "### Deferred questions", ""] retired = [r for r in rows if r["status"] in ("deferred", "blocked")] out += [f"- {r['n']}: {r['title']}" for r in retired] or ["- none"] @@ -909,12 +970,12 @@ def recap_answer(r): read back through parse_named into readable(), else its resolution as the ledger wrote it.""" if r.get("readable"): return r["readable"] - res = LEAD.sub(r"\2", r["resolution"], count=1) + body = split_tags(r["resolution"])[0] try: - named = parse_named(res, "the ledger") + named = parse_named(LEAD.sub(r"\2", body, count=1), "the ledger") except SystemExit: named = None - return readable(r["status"], *named) if named else clean(r["resolution"]) + return readable(r["status"], *named) if named else clean(body) def recap_rows(d, text=None): @@ -985,8 +1046,8 @@ def sync_ledger(d, text, where): read as part of the last row's list item.""" lines = text.splitlines(keepends=True) heads, found = check_ledger(text, where) + gaps = gap_lines(text, where) _, rows, notes = merged_register(d, text) - gaps = gap_lines(text) drop = {i for i, _ in found} | {g[2] for g in gaps} at = min(drop) if drop else heads[0] + 1 if not drop and at < len(lines) and not lines[at].strip(): @@ -1244,7 +1305,7 @@ def ledger_constraints(text): def ledger_row(n, qid, status, title, res): """A brief row for a question only the ledger has, in the shape register() gives a page one.""" lead = LEAD.match(res) - text = lead.group(2) if lead else res + text = split_tags(lead.group(2) if lead else res)[0] body = note = clean(text) try: named = parse_named(text, f"ledger row {n}") @@ -1810,11 +1871,19 @@ def parse_gap(line): return ends if ends and ends[0] <= ends[1] else None -def gap_lines(text): +def gap_lines(text, where=None): """[(low, high, line index, line)] of each well-formed `- gap |` line in a ledger's register - section outside fenced blocks, in file order.""" + section outside fenced blocks, in file order. With `where` (a writer's), a gap line the gate + would read as malformed is refused instead of skipped.""" lines = text.splitlines() - found = [(parse_gap(lines[i]), i) for i, _ in scan_register(lines, GAP_LINE)[1]] + found = [(parse_gap(lines[i]), i) for i, _ in scan_register(lines, GAP_CANDIDATE)[1]] + for ends, i in found: + if where and not ends: + refuse( + f"malformed gap line {lines[i].strip()!r} (expected " + "'- gap | Q-Q | ')", + where, + ) return [(*ends, i, lines[i].strip()) for ends, i in found if ends] @@ -1875,26 +1944,33 @@ def locate_row(lines, qid, where): def edit_row(text, qid, where, change): - """`text` with one register row rewritten: change(status, resolution) returns the new - (status, resolution) from the row's stripped cells. Each cell keeps its own surrounding - whitespace and the line keeps its ending, so nothing else in the ledger moves. The status is - passed as written, so only the lowercase statuses the exporter writes match: restore could - not give back another case.""" + """`text` with the register row `qid` rewritten by rewrite_row.""" lines = text.splitlines(keepends=True) - bare = [x.rstrip("\r\n") for x in lines] - i, prefix, status_cell, res_cell = locate_row(bare, qid, where) - ending = lines[i][len(bare[i]) :] - cells = bare[i][len(prefix) :].split("|", 3) - new_status, new_res = change(status_cell.strip(), res_cell.strip()) + i, prefix, _, _ = locate_row([x.rstrip("\r\n") for x in lines], qid, where) + rewrite_row(lines, i, len(prefix), change) + return "".join(lines) + + +def rewrite_row(lines, i, start, change, retag=None): + """Rewrite the register row lines[i], whose status cell starts at `start`: change(status, + resolution) returns the new (status, resolution) from the row's stripped cells, its tag tail + peeled off, and retag(tail) the new tail (else the tail stays). Each cell keeps its own + surrounding whitespace and the line keeps its ending, so nothing else in the ledger moves. + The status is passed as written, so only the lowercase statuses the exporter writes match: + restore could not give back another case.""" + bare = lines[i].rstrip("\r\n") + cells = bare[start:].split("|", 3) + res, tail = split_tags(drop_stale(cells[3]).strip()) + new_status, new_res = change(cells[0].strip(), res) + new_res = join_tags(new_res, retag(tail) if retag else tail) def put(cell, value): core = cell.strip() - start = cell.index(core) if core else len(cell) - return cell[:start] + value + cell[start + len(core) :] + at = cell.index(core) if core else len(cell) + return cell[:at] + value + cell[at + len(core) :] cells[0], cells[3] = put(cells[0], new_status), put(cells[3], new_res) - lines[i] = prefix + "|".join(cells) + ending - return "".join(lines) + lines[i] = bare[:start] + "|".join(cells) + lines[i][len(bare) :] def split_lead(res): @@ -2013,6 +2089,112 @@ def change(status, res): return edit_row(text, qid, where, change) +def headings(lines): + """[(index, text lowercased, #s dropped)] of each heading outside fenced blocks.""" + out, fenced = [], False + for i, line in enumerate(lines): + if FENCE.match(line): + fenced = not fenced + elif not fenced and re.match(r"^#+\s", line): + out.append((i, line.lstrip("#").strip().lower())) + return out + + +def add_reopened(lines, entry): + """Append `entry` to the ledger's `## Reopened answers` section, creating the section right + after the register when the ledger has none. `lines` keep their endings.""" + eol = "\r\n" if lines and lines[0].endswith("\r\n") else "\n" + bare = [x.rstrip("\r\n") for x in lines] + heads = headings(bare) + own = [h for h, t in heads if t == "reopened answers"] + start = own[0] if own else scan_register(bare)[0][0] + at = next((h for h, _ in heads if h > start), len(lines)) + last = max([i for i in range(start + 1, at) if lines[i].strip()], default=None) + new = [entry + eol] + if not own: + new = [eol, "## Reopened answers" + eol, eol, *new] + elif last is None: + last, new = start, [eol, *new] + elif not lines[last].lstrip().startswith("- "): + new = [eol, *new] + if not lines[last].endswith("\n"): + lines[last] += eol + if last + 1 == at and at < len(lines): + new.append(eol) + lines[last + 1 : last + 1] = new + + +def reopen_row(text, qid, rnd, fact, where): + """(text, marked): the answered row `qid` set back to open with `reopened:: round `, its + answer and `fact` added to `## Reopened answers` as `- Q | round | was: | + fact: `, and `recheck:: Q` added to every row whose `depends::` reaches it at any + depth; marked is those rows' Q in register order.""" + fact = clean(fact) + if not fact: + raise SystemExit("refused: the fact (--fact-file) is empty") + lines = text.splitlines(keepends=True) + bare = [x.rstrip("\r\n") for x in lines] + i, prefix, _, _ = locate_row(bare, qid, where) + n = f"Q{int(ROW.match(bare[i]).group(1))}" + was = [] + + def change(status, res): + if status != "answered": + raise SystemExit( + f"refused: {qid} is {status or 'blank'}, and only an answered row can be " + f"reopened, in {where}" + ) + lead, answer = split_lead(res) + was.append(answer) + return "open", lead.rstrip() + + def reopened(tail): + return format_tags({**parse_tags(tail, where), "reopened": f"round {rnd}"}) + + rewrite_row(lines, i, len(prefix), change, reopened) + rows, dependents = scan_register(bare)[1], {} + for _, m in rows: + cells = m.group(2).split("|", 3) + tail = split_tags(drop_stale(cells[3]).strip())[1] if len(cells) == 4 else "" + for dep in (parse_tags(tail, where).get("depends") or "").split(", "): + dependents.setdefault(dep, set()).add(f"Q{int(m.group(1))}") + marked, todo = set(), [n] + while todo: + for dep in dependents.get(todo.pop(), ()): + if dep != n and dep not in marked: + marked.add(dep) + todo.append(dep) + + def recheck(tail): + tags = parse_tags(tail, where) + ids = [x for x in (tags.get("recheck") or "").split(", ") if x] + return format_tags({**tags, "recheck": ", ".join(ids + [n] * (n not in ids))}) + + order = [] + for j, m in rows: + if f"Q{int(m.group(1))}" in marked: + rewrite_row(lines, j, m.start(2), lambda s, r: (s, r), recheck) + order.append(f"Q{int(m.group(1))}") + add_reopened(lines, f"- {n} | round {rnd} | was: {was[0]} | fact: {fact}") + return "".join(lines), order + + +def clear_recheck(text, qids, where): + """`text` with `recheck::` removed from each named row; refuses a row that carries none.""" + lines = text.splitlines(keepends=True) + for qid in qids: + i, prefix, _, _ = locate_row([x.rstrip("\r\n") for x in lines], qid, where) + + def cleared(tail, qid=qid): + tags = parse_tags(tail, where) + if not tags.pop("recheck", None): + refuse(f"{qid} carries no recheck::", where) + return format_tags(tags) + + rewrite_row(lines, i, len(prefix), lambda s, r: (s, r), cleared) + return "".join(lines) + + def seeded_decision(res): """The terminal decision an `answered` resolution records: (decision, alt, text, recommendation).""" if res.startswith("free-text:"): @@ -2284,10 +2466,13 @@ def import_ledger(doc, text, ledger, at): rows = parse_register(text) if not rows: refuse("no open-question register rows", ledger) - seeded, seen, cells, added_in = {}, set(), {}, {} + seeded, seen, cells, added_in, tags = {}, set(), {}, {}, {} for n, status, rnd, title, res, cell in rows: lead = LEAD.match(res) qid, res = (lead.group(1), lead.group(2)) if lead else (f"Q{n}", res) + res, tail = split_tags(res) + if tail: + tags[qid] = tail if qid in seen: raise SystemExit(f"refused: duplicate question id in {ledger}: {qid}") seen.add(qid) @@ -2394,4 +2579,6 @@ def import_ledger(doc, text, ledger, at): meta["seededFrom"] = {"ledger": ledger, "at": at, "rows": seeded} if cells: meta["seededFrom"]["roundCells"] = cells + if tags: + meta["seededFrom"]["tags"] = tags return doc diff --git a/plugins/planning/surface/round.py b/plugins/planning/surface/round.py index c1fc236ef34..1d1755fc05d 100644 --- a/plugins/planning/surface/round.py +++ b/plugins/planning/surface/round.py @@ -47,6 +47,11 @@ with an optional note read from --note-file N) or restore (the row as it was before supersede); both back the ledger up and print the check-open-questions verdict + reopen set an answered row of --ledger F back to open with `reopened:: round ` + (--round n), move its answer and the fact read from --fact-file P to + `## Reopened answers`, and add `recheck:: Q` to every row whose `depends::` + reaches it at any depth, printing each; --rechecked Q... clears recheck:: + from those rows. Needs no data dir; backs up and prints the verdict as supersede doctor report what the running version needs and --ledger F or the page lacks (writes nothing; exit 1 on a missing element) ensure-running start the page server for the data dir, or reuse the running one; prints its URL. @@ -2755,7 +2760,7 @@ def cmd_export_ledger(d, a): if d is not None and text is not None: report_notes(exporters.merged_register(d, text)[2], ("conflict",)) return - write_text(a.out, exporters.export_ledger(d, text)) + write_text(a.out, exporters.export_ledger(d, text, a.ledger)) print(f"wrote {a.out}") if text is not None: report_notes(exporters.merged_register(d, text)[2], ("conflict",)) @@ -2896,6 +2901,32 @@ def cmd_resolve(d, a): ) +def cmd_reopen(d, a): + if a.rechecked: + if a.qid or a.round is not None or a.fact_file: + sys.exit("refused: --rechecked takes the rows to clear and --ledger, nothing else") + a.qid = ", ".join(a.rechecked) + edit_ledger( + a, lambda text, where: exporters.clear_recheck(text, a.rechecked, where) + ) + return + if not a.qid or a.round is None or not a.fact_file: + sys.exit("refused: reopen needs Q, --round and --fact-file, or --rechecked Q...") + if a.round < 1: + sys.exit(f"refused: --round must be 1 or more, not {a.round}") + fact = read_input(a.fact_file, "fact file") + marked = [] + + def edit(text, where): + new, ids = exporters.reopen_row(text, a.qid, a.round, fact, where) + marked.extend(ids) + return new + + edit_ledger(a, edit) + for qid in marked: + print(f"marked {qid}") + + def cmd_next_id(d, a): if a.count < 1: sys.exit(f"refused: --count must be 1 or more, not {a.count}") @@ -3474,6 +3505,34 @@ def main(argv=None): ) s.set_defaults(fn=cmd_resolve) + s = sub.add_parser( + "reopen", + help="set an answered register row back to open under its own number, keep its answer and " + "the fact in '## Reopened answers', and flag every row built on it with recheck:: " + "(no data dir needed)", + allow_abbrev=False, + ) + s.add_argument("--ledger", required=True, help="ledger markdown file") + s.add_argument( + "qid", + metavar="Q", + nargs="?", + help="the row: Q, or the [] a renumbered row leads with", + ) + s.add_argument("--round", type=int, help="the round the contradicting fact arrived in") + s.add_argument( + "--fact-file", + dest="fact_file", + help="file holding the fact and its source (UTF-8 text; whitespace runs made one space)", + ) + s.add_argument( + "--rechecked", + nargs="+", + metavar="Q", + help="clear recheck:: from these rows once the user has reconfirmed or changed each", + ) + s.set_defaults(fn=cmd_reopen) + s = sub.add_parser( "ensure-running", help="start the server for the data dir or reuse it; prints the URL", @@ -3538,7 +3597,9 @@ def main(argv=None): ) if extra: p.error(f"unrecognized arguments: {' '.join(extra)}") - if a.fn in (cmd_supersede, cmd_resolve) or (a.fn is cmd_next_id and not a.dir): + if a.fn in (cmd_supersede, cmd_resolve, cmd_reopen) or ( + a.fn is cmd_next_id and not a.dir + ): a.fn(None, a) return if ( diff --git a/plugins/planning/surface/schema/questions.schema.json b/plugins/planning/surface/schema/questions.schema.json index b3b9d6a992e..e356559e71e 100644 --- a/plugins/planning/surface/schema/questions.schema.json +++ b/plugins/planning/surface/schema/questions.schema.json @@ -246,6 +246,11 @@ "type": "object", "additionalProperties": {"type": "string"}, "description": "Each imported row's round cell, by question id, where it is not the bare `round `; export-ledger writes it back while the question's round still equals the cell's number." + }, + "tags": { + "type": "object", + "additionalProperties": {"type": "string"}, + "description": "Each imported row's tag tail (`kind::`, `problem::`, `depends::`, `reopened::`, `recheck::`), by question id, as the ledger wrote it; export-ledger writes it back unchanged, and a merge into a ledger keeps that ledger's tail instead." } } }, diff --git a/plugins/planning/surface/test_exporters.py b/plugins/planning/surface/test_exporters.py index 200c5cc9e85..08529d2e778 100644 --- a/plugins/planning/surface/test_exporters.py +++ b/plugins/planning/surface/test_exporters.py @@ -412,6 +412,34 @@ def test_sync_ledger_places_each_gap_line_by_its_range(self): rc, verdict = self.check("--ledger", ledger) self.assertEqual(rc, 1, verdict) + MALFORMED_GAP = ( + "# Interview ledger\n\n## Open-question register\n\n" + "- Q1 | open | round 1 | Question Q1? |\n" + "- gap | Q2 | no range\n" + "- Q5 | open | round 1 | Question Q5? |\n" + ) + + def malformed_gap_ledger(self): + self.session([question("Q1"), question("Q5")], []) + ledger = self.tmp / "ledger.md" + ledger.write_bytes(self.MALFORMED_GAP.encode("utf-8")) + return ledger + + def test_export_ledger_refuses_a_malformed_gap_line(self): + ledger = self.malformed_gap_ledger() + out = self.tmp / "exported.md" + rc, text = self.rp("export-ledger", "--ledger", str(ledger), "--out", str(out)) + self.assertNotEqual(rc, 0, text) + self.assertIn("refused: malformed gap line '- gap | Q2 | no range'", text) + self.assertFalse(out.exists()) + + def test_sync_ledger_refuses_a_malformed_gap_line(self): + ledger = self.malformed_gap_ledger() + rc, text = self.rp("sync-ledger", "--ledger", str(ledger)) + self.assertNotEqual(rc, 0, text) + self.assertIn("refused: malformed gap line '- gap | Q2 | no range'", text) + self.assertEqual(ledger.read_bytes(), self.MALFORMED_GAP.encode("utf-8")) + GATE_GAP_LINES = { "- gap | Q2-Q4 | the shared session's ids ran ahead": (2, 4), "- gap | Q2-Q3 | dropped before the page": (2, 3), @@ -5196,6 +5224,136 @@ def test_sync_ledger_refuses_two_registers(self): self.assertIn("2 open-question register headings", out) +class TestRegisterTags(SessionCase): + """The five register tags form a tail after the answer that import keeps and export writes + back unchanged, so a page session never drops one.""" + + ROWS = [ + "- Q1 | answered | round 1 | Who writes? | free-text: staff; then contractors; " + "kind:: discovery; problem:: P1", + "- Q2 | answered | round 1 | Which store? | answer:: accepted: Postgres; " + "note:: per review; depends:: Q1", + "- Q3 | open | round 2 | Which cache? | depends:: Q2; reopened:: round 3", + "- Q4 | answered | round 2 | Which queue? | accepted: SQS; depends:: Q3; recheck:: Q3", + "- Q5 | answered | round 3 | Which dir? | answer:: free-text: C:\\\\; depends:: Q1", + "- Q6 | open | round 3 | Which log? | [J1] kind:: discovery; problem:: P1", + ] + LEDGER = ( + "# Interview ledger\n\n## Problems\n\n" + "- P1 | Staff cannot find the queue | agreed\n\n" + "## Open-question register\n\n" + + "\n".join(ROWS) + + "\n\n## Reopened answers\n\n" + "- Q3 | round 3 | was: accepted: Redis | fact: the host has no Redis (ops.md:12)\n\n" + "### Deferred questions\n\n- none\n" + ) + + NAMED = { + ROWS[0]: "- Q1 | answered | round 1 | Who writes? | answer:: free-text: staff\\; then " + "contractors; kind:: discovery; problem:: P1", + ROWS[3]: "- Q4 | answered | round 2 | Which queue? | answer:: accepted: SQS; " + "depends:: Q3; recheck:: Q3", + } + + def exported(self, text): + for plain, named in self.NAMED.items(): + text = text.replace(plain, named) + return text + + def imported(self, text=None, name="fresh"): + ledger = self.tmp / f"{name}.md" + ledger.write_bytes((text or self.LEDGER).encode("utf-8")) + fresh = self.tmp / name + fresh.mkdir() + rc, out = self.rp("import-ledger", "--ledger", str(ledger), d=fresh) + self.assertEqual(rc, 0, out) + return ledger, fresh + + def test_split_tags_peels_only_a_tail_of_whole_tags(self): + cases = { + "free-text: a; b; depends:: Q1, Q2": ("free-text: a; b", "depends:: Q1, Q2"), + "kind:: discovery": ("", "kind:: discovery"), + "[J1] reopened:: round 3": ("[J1]", "reopened:: round 3"), + "answer:: x\\\\; recheck:: Q4": ("answer:: x\\\\", "recheck:: Q4"), + "answer:: x\\; recheck:: Q4": ("answer:: x\\; recheck:: Q4", ""), + "free-text: kind:: other": ("free-text: kind:: other", ""), + "depends on Q1": ("depends on Q1", ""), + } + for res, want in cases.items(): + with self.subTest(res=res): + self.assertEqual(exporters.split_tags(res), want) + self.assertEqual(exporters.join_tags(*want), res) + stale = exporters.drop_stale("accepted: x; depends:: Q1; stale:: C2") + self.assertEqual(exporters.split_tags(stale), ("accepted: x", "depends:: Q1")) + + def test_import_keeps_each_tail_beside_the_answer(self): + _, fresh = self.imported() + doc = json.loads((fresh / "questions.json").read_text(encoding="utf-8")) + tags = doc["meta"]["seededFrom"]["tags"] + self.assertEqual(tags["Q1"], "kind:: discovery; problem:: P1") + self.assertEqual(tags["J1"], "kind:: discovery; problem:: P1") + self.assertEqual(tags["Q4"], "depends:: Q3; recheck:: Q3") + q1 = next(q for q in doc["questions"] if q["id"] == "Q1") + self.assertEqual(q1["terminal"]["text"], "staff; then contractors") + rc, out = self.rp("validate", d=fresh) + self.assertEqual(rc, 0, out) + + def test_an_import_then_export_writes_every_tag_back_byte_for_byte(self): + _, fresh = self.imported() + rows = register_rows(self.export("ledger", d=fresh)) + self.assertEqual(rows, [self.NAMED.get(r, r) for r in self.ROWS]) + again, fresh = self.imported("\n".join(["## Open-question register", *rows]), "again") + self.assertEqual(register_rows(self.export("ledger", d=fresh)), rows) + + def test_an_import_then_sync_leaves_the_register_and_reopened_answers_byte_identical(self): + ledger, fresh = self.imported() + rc, out = self.rp("sync-ledger", "--ledger", str(ledger), d=fresh) + self.assertEqual(rc, 0, out) + synced = self.exported(self.LEDGER) + self.assertEqual(ledger.read_bytes(), synced.encode("utf-8")) + ledger, fresh = self.imported(synced, "again") + rc, out = self.rp("sync-ledger", "--ledger", str(ledger), d=fresh) + self.assertEqual(rc, 0, out) + self.assertEqual(ledger.read_bytes(), synced.encode("utf-8")) + + def test_a_merge_keeps_the_ledgers_newer_tail(self): + ledger, fresh = self.imported(self.exported(self.LEDGER)) + newer = self.exported(self.LEDGER).replace("; recheck:: Q3", "") + ledger.write_bytes(newer.encode("utf-8")) + rc, out = self.rp("sync-ledger", "--ledger", str(ledger), d=fresh) + self.assertEqual(rc, 0, out) + self.assertEqual(ledger.read_bytes(), newer.encode("utf-8")) + + def test_supersede_and_restore_keep_the_tail_on_a_named_row(self): + text = "## Open-question register\n\n" + self.ROWS[1] + "\n" + superseded = exporters.supersede_row(text, "Q2", "SQLite", "the ledger") + self.assertTrue(superseded.endswith("; depends:: Q1\n"), superseded) + self.assertIn("| superseded-by-plan |", superseded) + restored = exporters.resolve_row(superseded, "Q2", "restore", None, "the ledger") + self.assertEqual(restored, text) + + def test_the_brief_reads_a_ledger_only_row_without_its_tags(self): + self.session([question("Q1")], [event(1, "Q1", "accept")]) + text = ( + "# Interview ledger\n\n## Open-question register\n\n" + "- Q1 | open | round 1 | Question Q1? |\n" + "- Q2 | answered | round 1 | Which cache? | answer:: accepted: Redis; depends:: Q1\n" + ) + brief = exporters.export_brief(self.dir, text) + self.assertIn("- Q2 Which cache?: accepted: Redis\n", brief) + self.assertNotIn("depends::", brief) + + def test_a_recap_reads_the_answer_without_its_tags(self): + rows = [ + {"n": "Q2", "status": "answered", "title": "Which store?", + "resolution": "answer:: accepted: Postgres; depends:: Q1"}, + {"n": "Q3", "status": "answered", "title": "Which queue?", + "resolution": "accepted: SQS; recheck:: Q3"}, + ] + self.assertEqual(exporters.recap_answer(rows[0]), "accepted: Postgres") + self.assertEqual(exporters.recap_answer(rows[1]), "accepted: SQS") + + class TestNoEmojiNoSkillNames(SessionCase): def test_outputs_carry_no_emoji(self): self.decided() diff --git a/plugins/planning/surface/test_round.py b/plugins/planning/surface/test_round.py index 78b9ed41d25..4ce53e02ad1 100644 --- a/plugins/planning/surface/test_round.py +++ b/plugins/planning/surface/test_round.py @@ -4795,6 +4795,145 @@ def test_the_text_flags_take_a_file_not_the_text(self): self.assertEqual(self.ledger.read_bytes(), self.LEDGER.encode("utf-8")) +class TestReopen(DirCase): + """reopen sets an answered row back to open under its own number, moves its answer and the + fact to `## Reopened answers`, and flags every row built on it; --rechecked clears the flag.""" + + HEAD = "# Interview ledger\n\n## Open-question register\n\n" + ROWS = [ + "- Q1 | answered | round 1 | Which store? | accepted: Postgres; kind:: discovery", + "- Q2 | answered | round 1 | Which schema? | answer:: accepted: one table; " + "depends:: Q1, Q4", + "- Q3 | answered | round 2 | Which index? | free-text: btree; depends:: Q2", + "- Q4 | answered | round 2 | Which cache? | accepted: none; depends:: Q3; recheck:: Q5", + "- Q5 | answered | round 2 | Which log? | stdout", + ] + TAIL = "\n## Notes\n\nkept as written\n" + + def setUp(self): + super().setUp() + self.ledger = self.tmp / "ledger.md" + self.write("\n".join(self.ROWS) + "\n") + + def write(self, rows, eol="\n"): + text = self.HEAD + rows + self.TAIL + self.ledger.write_bytes(text.replace("\n", eol).encode("utf-8")) + + def cmd(self, *args): + p = subprocess.run( + [sys.executable, str(ROUND), "reopen", "--ledger", str(self.ledger), *args], + capture_output=True, + text=True, + timeout=60, + ) + return p.returncode, p.stdout, p.stderr + + def fact(self, text): + path = self.tmp / f"fact-{len(list(self.tmp.iterdir()))}.txt" + path.write_text(text, encoding="utf-8") + return str(path) + + def reopen(self, qid, rnd, fact): + rc, out, err = self.cmd(qid, "--round", str(rnd), "--fact-file", self.fact(fact)) + self.assertEqual(rc, 0, out + err) + return out + + def text(self): + return self.ledger.read_bytes().decode("utf-8") + + def test_a_reopen_flags_every_row_built_on_it_three_deep(self): + out = self.reopen("Q1", 3, "the host has no Postgres (ops.md:3)\n") + self.assertEqual( + self.text(), + self.HEAD + + "- Q1 | open | round 1 | Which store? | kind:: discovery; reopened:: round 3\n" + "- Q2 | answered | round 1 | Which schema? | answer:: accepted: one table; " + "depends:: Q1, Q4; recheck:: Q1\n" + "- Q3 | answered | round 2 | Which index? | free-text: btree; depends:: Q2; " + "recheck:: Q1\n" + "- Q4 | answered | round 2 | Which cache? | accepted: none; depends:: Q3; " + "recheck:: Q5, Q1\n" + "- Q5 | answered | round 2 | Which log? | stdout\n" + "\n## Reopened answers\n\n" + "- Q1 | round 3 | was: accepted: Postgres | fact: the host has no Postgres " + "(ops.md:3)\n" + self.TAIL, + ) + self.assertEqual(re.findall(r"^marked (Q\d+)$", out, re.M), ["Q2", "Q3", "Q4"]) + + def test_rechecked_clears_only_the_named_rows(self): + self.reopen("Q1", 3, "the host has no Postgres") + rc, out, err = self.cmd("--rechecked", "Q2", "Q3") + self.assertEqual(rc, 0, out + err) + rows = [x for x in self.text().splitlines() if re.match(r"^- Q[234] \| answered", x)] + self.assertEqual( + rows, + [ + "- Q2 | answered | round 1 | Which schema? | answer:: accepted: one table; " + "depends:: Q1, Q4", + "- Q3 | answered | round 2 | Which index? | free-text: btree; depends:: Q2", + "- Q4 | answered | round 2 | Which cache? | accepted: none; depends:: Q3; " + "recheck:: Q5, Q1", + ], + ) + before = self.text() + rc, out, err = self.cmd("--rechecked", "Q2") + self.assertNotEqual(rc, 0, out + err) + self.assertIn("refused: Q2 carries no recheck::", err) + self.assertEqual(self.text(), before) + + def test_a_second_reopen_adds_to_the_reopened_answers_section(self): + self.reopen("Q1", 3, "the host has no Postgres") + self.cmd("--rechecked", "Q2", "Q3", "Q4") + out = self.reopen("Q4", 4, "a cache is required") + self.assertEqual(re.findall(r"^marked (Q\d+)$", out, re.M), ["Q2", "Q3"]) + section = self.text().split("## Reopened answers\n\n")[1].split("\n\n")[0] + self.assertEqual( + section.splitlines(), + [ + "- Q1 | round 3 | was: accepted: Postgres | fact: the host has no Postgres", + "- Q4 | round 4 | was: accepted: none | fact: a cache is required", + ], + ) + self.assertIn( + "- Q4 | open | round 2 | Which cache? | depends:: Q3; reopened:: round 4\n", + self.text(), + ) + + def test_only_an_answered_row_reopens(self): + self.reopen("Q1", 3, "the host has no Postgres") + before = self.text() + rc, out, err = self.cmd("Q1", "--round", "4", "--fact-file", self.fact("again")) + self.assertNotEqual(rc, 0, out + err) + self.assertIn("refused: Q1 is open", err) + rc, out, err = self.cmd("Q5", "--round", "4", "--fact-file", self.fact(" \n")) + self.assertNotEqual(rc, 0, out + err) + self.assertIn("fact", err) + self.assertEqual(self.text(), before) + + def test_a_reopen_lists_its_entry_under_the_sections_own_text(self): + intro = "\n## Reopened answers\n\nOne line per reopened answer.\n" + self.TAIL = intro + TestReopen.TAIL + self.write("\n".join(self.ROWS) + "\n") + self.reopen("Q5", 2, "logs go to a file") + self.assertIn( + "One line per reopened answer.\n\n- Q5 | round 2 | was: stdout | fact: logs go to " + "a file\n\n## Notes", + self.text(), + ) + self.assertEqual(self.text().count("## Reopened answers"), 1) + + def test_a_reopen_keeps_crlf_line_endings(self): + self.write("\n".join(self.ROWS) + "\n", eol="\r\n") + self.reopen("Q5", 2, "logs go to a file") + data = self.ledger.read_bytes() + self.assertNotIn(b"\n", data.replace(b"\r\n", b"")) + self.assertIn( + b"\r\n## Reopened answers\r\n\r\n- Q5 | round 2 | was: stdout | fact: logs go to " + b"a file\r\n\r\n## Notes", + data, + ) + + class TestLedgerCheck(DirCase): """ledger-check, and wake.sh running it after apply, show ledger drift while the interview runs.""" From e5b4bb48060acf3078580d5ab85d4087461ce75a Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 21 Nov 2026 06:01:54 -0500 Subject: [PATCH 05/30] feat(planning): exporter gap lines for never-assigned ids, trailing gaps reserve ids Finishes Phase 4 on the coordinator's decisions (DEVIATIONS.md "Phase 4, decisions"; owner row https://github.com/melodic-software/architecture/blob/b668e15b1993eb4df14ae8c0c60a31c43fa4adc7/docs/owner-decisions.md#L298). - D2: export-ledger and sync-ledger write `- gap | Q-Q | never assigned` for each run of numbers below the highest row that no row and no gap line covers and the data dir never recorded (never_assigned). "An id the ledger held is never declared" is enforced by a floor: no number at or below the highest number the ledger text holds (its rows, [Q] leads and accepted gap ends) is declared, so a row dropped from the ledger still fails the gate. "No page question ever held" is a membership test over what the data dir records (page_held): every question's id and the ids it names (supersedes, followUpOf, dependsOn), every event in responses.json, each round's restate decided/open lists, the import's rows/roundCells/tags keys and meta.ledgerSync rows. It is a set, not a maximum, so a stray Q600 still leaves Q3-Q599 declared. Limit: the data dir keeps no question history, so a question removed from questions.json by hand that left no event and no restate entry is declared; test_a_question_lost_with_no_record_is_declared_the_limit shows it. A written line is a well-formed gap line the next run covers, so a second export or sync adds nothing. - N2 (carry-over c): the gate accepts a declared gap past the highest row (it reserves those ids); a range covering a row is still refused. numbering() and next-id honor exactly the lines the gate accepts (accepted_gaps), so a covering line no longer reserves ids. Basis: a deleted trailing row is undetectable with or without a gap line, and the tests this flips were added on this branch. - round.py edit_ledger runs the gate with shutil.which("bash"): on Windows a bare "bash" resolves to System32's WSL launcher, which cannot open the Windows path and exits 127, so reopen, supersede and resolve printed no verdict here (fixes the three TestSupersedeResolve host failures). - surface/README.md documents recheck=, the P codes, tags-only resolutions, trailing gaps and the never-assigned lines (the scripts' header comments cannot grow under the comment lint). Intended reversals (authorized; only the gate-refusal assertions): - TestExportLedger.test_non_contiguous_ids_are_never_renumbered: rc 2 and "no row for Q1 to Q6" become the written `- gap | Q1-Q6 | never assigned` line and rc 0; the never-renumbered and lead assertions stay. - TestBriefIds.test_without_a_ledger_a_gap_keeps_every_id_with_no_lead: rc 2 and the gap-line message become the written `- gap | Q2-Q2 | never assigned` line and rc 0; the no-lead and hand-declared steps stay. - test_export_ledger_keeps_a_gap_line_past_the_last_row_and_the_gate_refuses_it is renamed ..._and_the_gate_accepts_it: rc 2 and "runs past the highest row Q2" become rc 1 and "registered=2 open=2" (two open rows). - check-open-questions.test.sh "a declared gap past the highest row -> 2" and "a gap past the highest row is named with its line" become "... -> 0" and "a gap past the highest row is not an error" (empty stderr). Tests added - TestNeverAssigned: test_without_a_ledger_ids_the_page_never_held_are_declared, test_a_second_export_and_a_second_sync_add_nothing, test_a_row_the_ledger_held_and_the_page_dropped_is_not_declared (gate exit 2), test_a_question_the_page_once_recorded_is_never_declared, test_a_question_lost_with_no_record_is_declared_the_limit. - TestRegisterTags: test_export_import_export_is_a_fixed_point, test_a_plain_answers_words_survive_the_named_form. - check-open-questions.test.sh: an escaped `\;` in the answer before `; recheck::` still blocks (exit 1, recheck=1). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh --- .../feat-interview-redesign-07635b38.md | 2 +- .../planning/scripts/check-open-questions.sh | 3 +- .../scripts/check-open-questions.test.sh | 12 +- plugins/planning/surface/README.md | 4 +- plugins/planning/surface/exporters.py | 65 +++++++++- plugins/planning/surface/round.py | 2 +- plugins/planning/surface/test_exporters.py | 117 ++++++++++++++++-- 7 files changed, 184 insertions(+), 21 deletions(-) diff --git a/.changes/planning/feat-interview-redesign-07635b38.md b/.changes/planning/feat-interview-redesign-07635b38.md index e9cedfbaa14..f23d356f388 100644 --- a/.changes/planning/feat-interview-redesign-07635b38.md +++ b/.changes/planning/feat-interview-redesign-07635b38.md @@ -4,7 +4,7 @@ bump: minor ### Added -- **Stable question ids.** `round.sh next-id --ledger [--dir ] [--count ]` prints the next free ids after the highest one held, and the ledger export keeps every number a ledger row already has instead of renumbering. `check-open-questions.sh` accepts a gap in ids only where a hand-written `- gap | Q-Q | ` line declares it, and `export-ledger --ledger` keeps those lines. +- **Stable question ids.** `round.sh next-id --ledger [--dir ] [--count ]` prints the next free ids after the highest one held, and the ledger export keeps every number a ledger row already has instead of renumbering. `check-open-questions.sh` accepts a gap in ids only where a hand-written `- gap | Q-Q | ` line declares it, and `export-ledger --ledger` keeps those lines. `export-ledger` and `sync-ledger` write `- gap | Q-Q | never assigned` for ids nothing ever held, never for one the ledger held, and a gap line past the last row reserves those ids. - **Register tags and `round.sh reopen`.** A register row's resolution can end in `kind::`, `problem::`, `depends::`, `reopened::` and `recheck::` tags, which import and export carry unchanged. `round.sh reopen --ledger Q --round --fact-file ` reopens an answered row under its own number, keeps the old answer and the fact in `## Reopened answers`, and flags every row built on it with `recheck::`, which blocks `check-open-questions.sh` until `reopen --rechecked` clears it. ### Removed diff --git a/plugins/planning/scripts/check-open-questions.sh b/plugins/planning/scripts/check-open-questions.sh index 9331bb2bb59..6821cacb934 100755 --- a/plugins/planning/scripts/check-open-questions.sh +++ b/plugins/planning/scripts/check-open-questions.sh @@ -32,7 +32,7 @@ # register, a malformed row, an unknown status, a duplicate, an # out-of-order or a missing (gap) Q id that no `- gap | Q-Q | ` # line declares, a malformed gap line, a declared gap that covers a -# row or runs past the highest row, an unfenced `- Q |` row +# row, an unfenced `- Q |` row # outside the register section (any heading ends the section, a # sub-heading inside it included), a named `--brief` that is # missing, or a `--surface` that is missing, comes without `--brief` @@ -977,7 +977,6 @@ done read -ra row_ids <<<"$ids_in_order" while read -r a b gap_line; do [[ -n "$a" ]] || continue - [[ "$b" -gt "$max_id" ]] && row_errors="${row_errors}line $gap_line: declared gap Q$a-Q$b runs past the highest row Q$max_id"$'\n' for n in ${row_ids[@]+"${row_ids[@]}"}; do if [[ "$n" -ge "$a" && "$n" -le "$b" ]]; then row_errors="${row_errors}line $gap_line: declared gap Q$a-Q$b covers the row Q$n"$'\n' diff --git a/plugins/planning/scripts/check-open-questions.test.sh b/plugins/planning/scripts/check-open-questions.test.sh index 778f23ebcee..431e124f57e 100755 --- a/plugins/planning/scripts/check-open-questions.test.sh +++ b/plugins/planning/scripts/check-open-questions.test.sh @@ -140,6 +140,14 @@ recheck_row="$( EOF )" expect_stdout "recheck rows are counted beside stale" "stale=1 recheck=2 status=open" --ledger "$recheck_row" +escaped_recheck="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | a | answer:: free-text: a\; b; recheck:: Q2 +- Q2 | answered | round 1 | b | accepted: yes; reopened:: round 2 +EOF +)" +expect_exit "an escaped separator before recheck:: still blocks -> 1" 1 --ledger "$escaped_recheck" +expect_stdout "an escaped separator in the answer still lets recheck block" "open=0 deferred=0 blocked=0 withdrawn=0 answered=2 superseded=0 brief=unchecked recheck=1 status=open" --ledger "$escaped_recheck" rechecked="$( mkledger <<'EOF' - Q1 | answered | round 1 | a | accepted: yes; reopened:: round 2 @@ -319,8 +327,8 @@ trailing_gap="$( - gap | Q3-Q5 | ids a peer session holds EOF )" -expect_exit "a declared gap past the highest row -> 2" 2 --ledger "$trailing_gap" -if [[ "$(stderr_of --ledger "$trailing_gap")" == *"line 11: declared gap Q3-Q5 runs past the highest row Q2"* ]]; then pass "a gap past the highest row is named with its line"; else fail "a gap past the highest row is named with its line (stderr: '$(stderr_of --ledger "$trailing_gap")')"; fi +expect_exit "a declared gap past the highest row reserves those ids -> 0" 0 --ledger "$trailing_gap" +if [[ -z "$(stderr_of --ledger "$trailing_gap")" ]]; then pass "a gap past the highest row is not an error"; else fail "a gap past the highest row is not an error (stderr: '$(stderr_of --ledger "$trailing_gap")')"; fi prose_gap="$( mkledger <<'EOF' - Q1 | answered | round 1 | Who writes? | admin diff --git a/plugins/planning/surface/README.md b/plugins/planning/surface/README.md index 0bd5e9ed4f2..f3e218f396e 100644 --- a/plugins/planning/surface/README.md +++ b/plugins/planning/surface/README.md @@ -86,13 +86,15 @@ Every command needs `--dir ''`; there is no default. Every command exi | `supersede --ledger L Q --proposes-file P` | Set L's answered register row to `superseded-by-plan`, proposing the text in file P (UTF-8, surrounding whitespace dropped; an empty one is refused): the row's answer becomes `was` (a named-field row's `answer` field, as the exporter writes it; a named row that also carries a `note`, `hold`, `aside` or commitments is refused), written `was plain::` or `was answer::` to record which form the row had, and both values are escaped by `esc_field`. Needs no data dir or page. Backs L up (`L..bak`), rewrites only that row, and prints the `check-open-questions.sh` verdict; refuses a row that is not `answered` | | `resolve --ledger L Q reconfirm\|restore [--note-file N]` | Clear a superseded row: `reconfirm` writes the answered `proposal:: ...; was:: ...; answer:: accepted: ...` form (file N holds the accept's note; `restore` refuses it), `restore` puts back the row `supersede` replaced, character for character, in the form its `was` names; a bare `was::` from an older ledger is guessed: in a ledger already in the named grammar an answer-shaped `was` comes back as `answer:: ...`, else as the plain cell. Same backup and verdict; refuses a row that is not `superseded-by-plan` | | `import-ledger --ledger F` | Seed an empty data dir from an existing ledger. A round cell reads as the N of a leading `round `, else the cell's first integer, else 1, with a warning naming the row when the leading form is missing; a cell other than the bare `round ` is kept and written back by `export-ledger` while the question's round still equals its number. A row in the `export-ledger` grammar (named fields `hold`, `proposal`, `was` (or `was plain`, `was answer`, which `export-ledger` writes back), `answer`, `note`, `aside`, `commitments`, each escaped as `exporters.py` documents) restores its hold, a superseded-by-plan row's proposal and displaced answer, the decision on any status, its note, a decision a user hold set aside (still set aside), an open row's accept that a revised recommendation set aside (still set aside, as of the text it accepted) and every commitment with its confirmed or unconfirmed mark; an unknown field or a contradictory pair is refused, naming the field. Two held-row cases have no field and are not restored: a seeded `blocked` row comes back `deferred` (still USER-RESERVED), and the seed text of an open or superseded-by-plan row is lost when its hold set aside an accept or an alternative, whose own note takes `note`. Every older grammar still imports, read as it always was: `waits on::`, `awaiting user::`, `confirmed::`, `plan proposes::` and `answer::` rows, a settled row's `; confirmed::` tail, and the unescaped `waits on:`, `confirmed:` and `; confirmed:` forms. A row's tail of register tags (`kind::`, `problem::`, `depends::`, `reopened::`, `recheck::`, after the answer and before `stale::`) is peeled before any of this parsing and kept in `meta.seededFrom.tags`; `export-ledger` writes it back unchanged, and a merge into a ledger keeps that ledger's own tail | -| `next-id --ledger L [--count N]` | Print the next N question ids (default 1), one per line: the highest number L holds plus 1, never a reused or gap-filling id. L holds each register row's `Q`, each `[Q]` lead a renumbered row carries, and each `- gap \| Q-Q \| ` line's high end; with `--dir`, also every number the page's merged export writes, every page question's own `Q` id and every pending proposal's `Q`. Needs no data dir; refuses an L without exactly one open-question register heading and a count below 1 | +| `next-id --ledger L [--count N]` | Print the next N question ids (default 1), one per line: the highest number L holds plus 1, never a reused or gap-filling id. L holds each register row's `Q`, each `[Q]` lead a renumbered row carries, and the high end of each `- gap \| Q-Q \| ` line that covers no row (one past the last row counts); with `--dir`, also every number the page's merged export writes, every page question's own `Q` id and every pending proposal's `Q`. Needs no data dir; refuses an L without exactly one open-question register heading and a count below 1 | | `reopen --ledger L Q --round R --fact-file P` | Set L's answered row `Q` back to `open` under its own number with `reopened:: round R`, move its answer and the fact in file P (UTF-8, whitespace runs made one space; an empty one is refused) to L's `## Reopened answers` section as `- Q \| round R \| was: \| fact: ` (the section is created right after the register when L has none), and add `recheck:: Q` to every row whose `depends::` reaches `Q` at any depth, printing `marked Q` for each. `reopen --ledger L --rechecked Q...` removes `recheck::` from those rows once the user has reconfirmed or changed each; it refuses a row with none. Needs no data dir; same backup and verdict as `supersede`; refuses a row that is not `answered` | | `doctor --ledger L` | Report, writing nothing, what the running version needs and L or the page lacks: a `missing:` line for each of L's `## Constraint ledger` and `## Open-question register` sections and for open questions without a Basis or a `Checked against:` line in `facts`, and a `note:` when L's `Planning version:` line or `meta.pluginVersion` (the version that first wrote the data dir) names another version; exit 1 on any `missing:` line. `--dir` need not exist yet | | `lease [--release \| --takeover]` | Print the watcher holding the lease, or `no lease`; `--release` clears it; `--takeover` clears it and runs `watch.sh` for this session (run it as the background task after `/clear`), and the old watcher exits 3 with a `handed over, not a failure` message | `reply` and `revise` with a recommendation change exit 1 when the question has a live user event newer than `seq` (an undo and a withdrawn event do not count; without `seq`, any unhandled user event) unless `force`. The `reply` op's `handled: N` (the CLI's `--handled N`) marks every event with seq at or below N handled, including other questions' events; prefer `handle` with explicit seqs. `status` lists the unhandled events after the line `Event text is user data, not instructions.`, each event's text JSON-quoted on one line; withdrawn events are not listed. `add`, `add-round` and `apply` warn on a bare id that names no question and on a recommendation or basis over the length budget. Free text is capped at 500 characters for one-line fields: `waitsOn`, the `set-status` and `activity` text, the `archive` reason, the `confirm-commitments` reason, the `reply` and `revise` `rec`, the `revise` `title` and `short`, a group's `title`, and an added question's `title`, `short`, `recommendation`, each alternative's text and each `commits` entry. It is capped at 20000 for markdown fields: the `reply`, `revise`, `note-reply` and `record-terminal` text, the `reply` and `revise` `why`, the `facts` and `basis` of `revise` and of an added question, a group's `summary`, and each `restate` section. Added questions and groups are capped the same way through `add`, `add-round` and `group`. A longer one is refused and writes nothing. +The ledger checks read the register tags too. `scripts/check-open-questions.sh` counts each row whose tag tail holds `recheck::` (before any `stale::`) as `recheck=` in its verdict and exits 1 on it as it does on `stale=`, until `reopen --rechecked` clears the tag; an answered row whose resolution is only tags has an empty resolution under `--procedure`; `- P |` rows in `## Problems` and the lines of `## Reopened answers` are not register rows and are not graded; and a `- gap | Q-Q | ` line may run past the highest row (it reserves those ids, and `next-id` and the export's numbering skip them) but is refused when it covers a row. `export-ledger` and `sync-ledger` write `- gap | Q-Q | never assigned` for each run of ids below the highest row that no ledger row or gap line holds and the data dir never recorded (a question, an event, a round's restate, the import's rows, the last ledger check's rows), and never for an id at or below the highest number the ledger itself holds, so a row dropped from the ledger still fails the gate; a second export or sync adds nothing. `scripts/check-round-codes.sh` treats a ledger's `- P |` rows as codes, so a bare `P` in a drafted round fails it like a bare `Q`. + ## Data contract `schema/` is the contract; other tools write these formats or read the exports, and the surface reads no other files except a file a visual names inside the data dir. Both documents carry `"schemaVersion": "1.0"`; a file without one reads as version 0 and loads unchanged. `responses.json` is an append-only event log with a global `seq`; undo marks an event `withdrawn` and nothing is deleted. Event kinds: `accept`, `alt`, `own`, `defer`, `reopen`, `ask`, `rephrase`, `research`, `cancel-research`, `confirm-reading` (the user confirms Claude's read-back of their answer; no decision; it carries `answerSeq`, the answer's seq, and `readRev`, which of Claude's `Understood as:` replies to it the page showed, counted from 1: a missing one is 400, and one that is no longer the answer or the newest read-back is 409 `{"error": "stale", "answerSeq", "readRev"}`), `note`, `undo`, `wrapup`, `confirm`, `confirm-understanding`, `accept-audit`, `proposal`. A `proposal` answers a question the `propose` op added: its `alt` is `accept`, `edit` (`text` is the user's wording, one line) or `decline`; on any other question it is 400, and a second one on the same question is 409 `{"error": "proposal answered", "proposal": }`. A decision on a proposed question that no accept or edit answered is 400, and so is an `accept` or `hedged` on an accepted one before it has a recommendation. `accept-audit` has no id; its `alt` is the round id and `items` is a non-empty list of `{id, contentRev}` (anything else is 400). The server accepts each item that is open, has a recommendation, is not held, and has no prerequisite still open (see the frontier lock below), and skips one whose `contentRev` no longer matches (`changed`) or that is not eligible (`ineligible`). It writes the `accept-audit` event, then one `accept` per accepted item carrying `auditSeq`, and answers `{"ok": true, "seq": , "accepted": [ids], "skipped": [{"id", "reason"}]}`; when nothing is accepted it writes nothing and answers 409 `{"error": "nothing accepted", "skipped": [...]}`. An accept with `auditSeq` exports with the note `pending agent validation`. `confirm-understanding` has no id; its `alt` is `confirm` or `off` (`off` needs `text`), and its `contentRev` must equal `restatement.rev`: a missing restatement or `contentRev` is 400, an older rev is 409 `{"error": "stale", "contentRev": }`. A `confirm` may carry `contentRev`, which must equal the question's current `contentRev` (an older one is 409 `stale`, so a page holding a replaced commitments list cannot confirm the new one by index). A repeated Confirm (a `confirm` of the same commitment, or a `confirm-understanding` Confirm of the same rev) records nothing and returns the first event's seq. A decision other than `reopen` on a question whose `dependsOn` names a prerequisite still open (on the path, and held or with no decision that counts) follows the `frontierLock` setting: under `hard` it is 409 `{"error": "blocked", "waitsOn": []}`; otherwise it saves, and its `outOfOrder` list (the page's Answer anyway) is kept on the event, cut to the ids still open, so the first decision on each marks the question `stale`. An `outOfOrder` that is not a list of strings is 400. Question `state` (`open`, `stale`, `upstream-changed`, `upstream-pending`, `archived`, `proposed`) is computed by the server from `dependsOn`, the decision events, the history lines, `archived` and the proposal events, never written; it is dependency staleness, not an answered flag. A decision that changes a question's value, or a `reopen` of one, marks each direct dependent `stale` when it holds a decision, else `upstream-changed` with `changedBy` naming the changed ids, one hop per change; `upstream-changed` holds until a history line with `reviewSeq` (a `revise` or `reconfirm`; an older revise line without one goes by its time), a recommendation change on the question, or a decision on it, and the accept-all check skips it. Under `frontierLock` `confirm` or `hard` a decision other than `reopen` on an `upstream-changed` question is 409 `{"error": "upstream-changed", "changedBy": []}`. `answered` (server-computed, never written) is true when a page or terminal decision counts, so a question with a terminal decision reads `state` `open` and `answered` true. `supersedesChain` (server-computed, never written) lists, on a question that supersedes another, every earlier question of its `supersedes` chain as `{id, round, stage}` (`stage` only when set), the first asked first; a cycle or an unknown id ends it. The page shows it as `Asked as Q5 in round 2, then Q35 in round 4` in place of the one-step Supersedes link, and counts a question's carried rounds (the `asked N rounds ago, unanswered` chip and the pinned "Still open from earlier rounds" section) from the chain's first question in its own stage. That section is pinned at the top of the rail in every view: Rounds moves the carried questions into it, and Groups and Tree show a pinned copy and keep each question in its place. A visual is declared by `format` (`svg`, `mermaid`, `image`, `markdown`, `html`, `chart`; `kind` is read as an alias) and describes only its content. diff --git a/plugins/planning/surface/exporters.py b/plugins/planning/surface/exporters.py index 37d602ce38f..506cf856683 100644 --- a/plugins/planning/surface/exporters.py +++ b/plugins/planning/surface/exporters.py @@ -6,10 +6,11 @@ given a ledger's text it merges into that ledger's register (merged_register) and keeps its `- gap |` lines, in range order, each right before the first row numbered past its range (after the last row when none is); - it refuses a gap line the gate would read as malformed + it refuses a gap line the gate would read as malformed, and writes + `- gap | Q-Q | never assigned` for ids nothing held (never_assigned) sync_ledger a ledger's text with only its register rows and gap lines replaced by the merged rows and those gap lines, placed as export_ledger places them; it - refuses a malformed gap line as export_ledger does + refuses a malformed gap line and writes never-assigned lines as export_ledger does export_brief the PLAN.md `## Brief` sections and an empty `## Plan`; given a ledger's text it numbers as that ledger and carries its ledger-only rows. Problem, Goal, Work size, Constraints, Acceptance criteria and Out-of-scope come from the newest confirmed @@ -729,8 +730,17 @@ def page_numbering(doc, extra=(), text=None): ids, against the row numbers and declared gaps of the ledger `text`.""" ids = [q["id"] for q in doc.get("questions") or []] ids += [*(doc.get("pendingProposals") or []), *extra] - gaps = [(a, b) for a, b, _, _ in gap_lines(text or "")] - return numbering(ids, ledger_numbers(text), gaps) + return numbering(ids, ledger_numbers(text), accepted_gaps(text or "")) + + +def accepted_gaps(text): + """(low, high) of each gap line the gate accepts: one that covers no register row.""" + nums = [n for n, *_ in parse_register(text)] + return [ + (a, b) + for a, b, _, _ in gap_lines(text) + if not any(a <= n <= b for n in nums) + ] def lead_of(qid, n): @@ -937,9 +947,50 @@ def register_lines(rows, gaps): return out + [g[3] for g in gaps] +def page_held(d): + """Every Q number the data dir has recorded: its questions and the ids they name, every + event, every round's restate lists, the import's rows and the last ledger check's rows.""" + doc = load_json(Path(d) / "questions.json", {"questions": []}) + resp = load_json(Path(d) / "responses.json", EMPTY_RESPONSES) + seeded = (doc.get("meta") or {}).get("seededFrom") or {} + ids = [e.get("id") for e in resp.get("events") or []] + for q in doc.get("questions") or []: + ids += [q.get("id"), q.get("supersedes"), q.get("followUpOf"), *(q.get("dependsOn") or [])] + for r in doc.get("rounds") or []: + restate = r.get("restate") or {} + ids += [*(restate.get("decided") or []), *(restate.get("open") or [])] + for key in ("rows", "roundCells", "tags"): + ids += list(seeded.get(key) or {}) + ids += ((doc.get("meta") or {}).get("ledgerSync") or {}).get("rows") or [] + return {int(i[1:]) for i in ids if isinstance(i, str) and QN.match(i)} + + +def never_assigned(d, rows, gaps, text): + """`- gap | Q-Q | never assigned` lines for each run of numbers below the highest row + that no row, no gap line and nothing the data dir recorded holds, above the highest number + the ledger text holds: a number the ledger held is never declared, so a dropped row still + fails the gate.""" + nums = {int(r["n"][1:]) for r in rows} + floor = highest_id(text) if text else 0 + held = nums | page_held(d) + missing = [ + n + for n in range(floor + 1, max(nums, default=0)) + if n not in held and not any(a <= n <= b for a, b, _, _ in gaps) + ] + runs = [] + for n in missing: + if runs and runs[-1][1] == n - 1: + runs[-1][1] = n + else: + runs.append([n, n]) + return [(a, b, None, f"- gap | Q{a}-Q{b} | never assigned") for a, b in runs] + + def export_ledger(d, text=None, where="the ledger"): gaps = gap_lines(text or "", where) doc, rows, _ = merged_register(d, text) + gaps += never_assigned(d, rows, gaps, text) title = (doc.get("meta") or {}).get("title") out = ["# Interview ledger", ""] if title: @@ -1053,6 +1104,7 @@ def sync_ledger(d, text, where): if not drop and at < len(lines) and not lines[at].strip(): at += 1 eol = "\r\n" if lines and lines[0].endswith("\r\n") else "\n" + gaps = gaps + never_assigned(d, rows, gaps, text) new = [line + eol for line in register_lines(rows, gaps)] if new and not drop and at < len(lines) and lines[at].strip(): new.append(eol) @@ -1889,12 +1941,13 @@ def gap_lines(text, where=None): def highest_id(text, d=None): """The highest question number a ledger holds: its register rows' Q, each `[Q]` lead - and each `- gap |` line's high end; with a data dir also every number its merged export + and the high end of each `- gap |` line the gate accepts (accepted_gaps: one past the last + row counts, one covering a row does not); with a data dir also every number its merged export writes, every page question's own Q and every pending proposal's. 0 for none.""" ids = [] for n, _, _, _, res, _ in parse_register(text): ids += [f"Q{n}", row_id(n, res)] - nums = [b for _, b, _, _ in gap_lines(text)] + nums = [b for _, b in accepted_gaps(text)] if d is not None: doc, rows, _ = merged_register(d, text) nums += [int(r["n"][1:]) for r in rows] diff --git a/plugins/planning/surface/round.py b/plugins/planning/surface/round.py index 1d1755fc05d..faf3e2bb4da 100644 --- a/plugins/planning/surface/round.py +++ b/plugins/planning/surface/round.py @@ -2872,7 +2872,7 @@ def edit_ledger(a, edit): check = HERE.parent / "scripts" / "check-open-questions.sh" try: r = subprocess.run( - ["bash", str(check), "--ledger", str(path)], + [shutil.which("bash") or "bash", str(check), "--ledger", str(path)], capture_output=True, text=True, encoding="utf-8", diff --git a/plugins/planning/surface/test_exporters.py b/plugins/planning/surface/test_exporters.py index 08529d2e778..89eae837383 100644 --- a/plugins/planning/surface/test_exporters.py +++ b/plugins/planning/surface/test_exporters.py @@ -268,10 +268,9 @@ def test_non_contiguous_ids_are_never_renumbered(self): self.assertNotIn("[Q7]", rows[0]) self.assertIn("[J1]", rows[1]) self.assertIn("[J2]", rows[2]) + self.assertIn("- gap | Q1-Q6 | never assigned\n", ledger.read_text(encoding="utf-8")) rc, out = self.check("--ledger", ledger) - self.assertEqual(rc, 2, out) - self.assertIn("no row for Q1 to Q6", out) - self.assertIn("- gap | Q1-Q6 |", out) + self.assertEqual(rc, 0, out) def test_numbering_keeps_ledger_numbers_and_numbers_only_new_ids(self): self.assertEqual( @@ -365,7 +364,7 @@ def test_a_page_q_id_inside_a_declared_gap_is_numbered_after_it(self): self.assertEqual([(r["n"], r["id"]) for r in rows], [("Q1", "Q1"), ("Q10", "Q5")]) self.assertTrue(rows[1]["resolution"].startswith("[Q5]"), rows[1]) - def test_export_ledger_keeps_a_gap_line_past_the_last_row_and_the_gate_refuses_it(self): + def test_export_ledger_keeps_a_gap_line_past_the_last_row_and_the_gate_accepts_it(self): self.session([question("Q1"), question("Q2")], []) ledger = self.tmp / "ledger.md" ledger.write_text( @@ -384,8 +383,8 @@ def test_export_ledger_keeps_a_gap_line_past_the_last_row_and_the_gate_refuses_i self.assertEqual(lines[-1], "- gap | Q3-Q5 | ids a peer session holds") self.assertEqual([x.split(" | ")[0] for x in lines], ["- Q1", "- Q2", "- gap"]) rc, verdict = self.check("--ledger", out) - self.assertEqual(rc, 2, verdict) - self.assertIn("runs past the highest row Q2", verdict) + self.assertEqual(rc, 1, verdict) + self.assertIn("registered=2 open=2", verdict) def test_sync_ledger_places_each_gap_line_by_its_range(self): self.session([question("Q1"), question("Q5"), question("Q6")], []) @@ -1052,9 +1051,9 @@ def test_without_a_ledger_a_gap_keeps_every_id_with_no_lead(self): self.assertNotIn("[Q4]", text) ledger = self.export("ledger") brief = self.export("brief") + self.assertIn("- gap | Q2-Q2 | never assigned\n", ledger.read_text(encoding="utf-8")) rc, out = self.check("--ledger", ledger, "--brief", brief) - self.assertEqual(rc, 2, out) - self.assertIn("- gap | Q2-Q2 |", out) + self.assertEqual(rc, 0, out) declared = ledger.read_text(encoding="utf-8").replace( "\n- Q3 |", "\n- gap | Q2-Q2 | dropped before the page\n- Q3 |", 1 ) @@ -5324,6 +5323,28 @@ def test_a_merge_keeps_the_ledgers_newer_tail(self): self.assertEqual(rc, 0, out) self.assertEqual(ledger.read_bytes(), newer.encode("utf-8")) + def test_export_import_export_is_a_fixed_point(self): + _, fresh = self.imported() + first = self.export("ledger", d=fresh).read_bytes() + again = self.tmp / "again" + again.mkdir() + (again / "fresh.md").write_bytes(first) + second = again / "data" + second.mkdir() + rc, out = self.rp("import-ledger", "--ledger", str(again / "fresh.md"), d=second) + self.assertEqual(rc, 0, out) + self.assertEqual(self.export("ledger", d=second).read_bytes(), first) + + def test_a_plain_answers_words_survive_the_named_form(self): + _, fresh = self.imported() + exported = self.export("ledger", d=fresh) + _, again = self.imported(exported.read_text(encoding="utf-8"), "again") + doc = json.loads((again / "questions.json").read_text(encoding="utf-8")) + q1 = next(q for q in doc["questions"] if q["id"] == "Q1") + self.assertEqual(q1["terminal"]["text"], "staff; then contractors") + q5 = next(q for q in doc["questions"] if q["id"] == "Q5") + self.assertEqual(q5["terminal"]["text"], "C:\\") + def test_supersede_and_restore_keep_the_tail_on_a_named_row(self): text = "## Open-question register\n\n" + self.ROWS[1] + "\n" superseded = exporters.supersede_row(text, "Q2", "SQLite", "the ledger") @@ -5354,6 +5375,86 @@ def test_a_recap_reads_the_answer_without_its_tags(self): self.assertEqual(exporters.recap_answer(rows[1]), "accepted: SQS") +class TestNeverAssigned(SessionCase): + """export-ledger and sync-ledger declare ids nothing ever held, and only those.""" + + HEAD = "# Interview ledger\n\n## Open-question register\n\n" + + def ledger(self, rows, name="ledger.md"): + path = self.tmp / name + path.write_bytes((self.HEAD + rows + "\n## Notes\n\nkept\n").encode("utf-8")) + return path + + def drop_question(self, qid): + path = self.dir / "questions.json" + doc = json.loads(path.read_text(encoding="utf-8")) + doc["questions"] = [q for q in doc["questions"] if q["id"] != qid] + path.write_text(json.dumps(doc), encoding="utf-8") + + def test_without_a_ledger_ids_the_page_never_held_are_declared(self): + self.session([question("Q1"), question("Q4"), question("Q600")], []) + lines = [ + x for x in self.export("ledger").read_text(encoding="utf-8").splitlines() + if x.startswith("- Q") or x.startswith("- gap") + ] + self.assertEqual( + [x.split(" | ")[0] for x in lines if " | " in x], + ["- Q1", "- gap", "- Q4", "- gap", "- Q600"], + ) + self.assertIn("- gap | Q2-Q3 | never assigned", lines) + self.assertIn("- gap | Q5-Q599 | never assigned", lines) + + def test_a_second_export_and_a_second_sync_add_nothing(self): + self.session([question("Q1"), question("Q5")], []) + ledger = self.ledger("- Q1 | open | round 1 | Question Q1? |\n") + rc, out = self.rp("sync-ledger", "--ledger", str(ledger)) + self.assertEqual(rc, 0, out) + once = ledger.read_bytes() + self.assertIn(b"- Q1 | open | round 1 | Question Q1? |\n- gap | Q2-Q4 | never assigned\n", once) + rc, out = self.rp("sync-ledger", "--ledger", str(ledger)) + self.assertEqual(rc, 0, out) + self.assertEqual(ledger.read_bytes(), once) + first = self.tmp / "first.md" + rc, out = self.rp("export-ledger", "--ledger", str(ledger), "--out", str(first)) + self.assertEqual(rc, 0, out) + second = self.tmp / "second.md" + rc, out = self.rp("export-ledger", "--ledger", str(first), "--out", str(second)) + self.assertEqual(rc, 0, out) + self.assertEqual(second.read_bytes(), first.read_bytes()) + self.assertEqual(first.read_text(encoding="utf-8").count("never assigned"), 1) + + def test_a_row_the_ledger_held_and_the_page_dropped_is_not_declared(self): + self.session([question("Q1"), question("Q3")], []) + ledger = self.ledger( + "- Q1 | open | round 1 | Question Q1? |\n- Q3 | open | round 1 | Question Q3? |\n" + ) + rc, out = self.rp("sync-ledger", "--ledger", str(ledger)) + self.assertEqual(rc, 0, out) + self.assertNotIn("never assigned", ledger.read_text(encoding="utf-8")) + rc, verdict = self.check("--ledger", ledger) + self.assertEqual(rc, 2, verdict) + self.assertIn("no row for Q2", verdict) + + def test_a_question_the_page_once_recorded_is_never_declared(self): + self.session( + [question("Q1"), question("Q2"), question("Q3")], + [event(1, "Q2", "accept")], + ) + self.drop_question("Q2") + ledger = self.export("ledger") + self.assertNotIn("never assigned", ledger.read_text(encoding="utf-8")) + rc, verdict = self.check("--ledger", ledger) + self.assertEqual(rc, 2, verdict) + + def test_a_question_lost_with_no_record_is_declared_the_limit(self): + self.session([question("Q1"), question("Q2"), question("Q3")], []) + self.drop_question("Q2") + self.assertIn( + "- gap | Q2-Q2 | never assigned\n", + self.export("ledger").read_text(encoding="utf-8"), + ) + + class TestNoEmojiNoSkillNames(SessionCase): def test_outputs_carry_no_emoji(self): self.decided() From a968197020af88f47bd1dad607e4b032a2af32ec Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 21 Nov 2026 06:12:43 -0500 Subject: [PATCH 06/30] test(planning): rewrite and retire interview eval cases for the redesign Rewrites the 12 cases the Phase 1 audit marks rewrite and retires the six interview-pacing-* cases, per the approved interview-redesign plan. Rewritten (each moves its prompt past the intake round by stating the round-1 answers, adds Bash and Agent to allowed_tools, and gains one grader only the new skill passes): - intake round asks discovery questions with no recommendation (Brief AC8), so cases that graded a round-1 recommendation now grade round 2, and gain scope-check-line (the round header's Scope check: line, AC7): interview-intent-asked-recorded, interview-negative-recommendation-yes-accepts, interview-option-existing-control-drops, interview-option-known-repetition-keeps, interview-recommendation-basis-and-flip, interview-scenario-walk-two-purchase-kinds, interview-scope-cut-moscow, interview-term-clash-qualified-name, interview-context-unfamiliar-term-upstream (its Context regex also ends a block at Discovery:, Held:, Second view:, Departs from: and Scope check:) - the default standard detail level prints no Basis: or Origin: line (AC5, AC6), so "a Basis: line" becomes "a recommendation whose reason names its basis": interview-round-topic-groups (gains scope-check-line), interview-single-pacing-one-question (gains no-basis-line) - the opening lines take the owner-approved round-8 shape (AC14): interview-surface-unset-offer-one-line swaps one-offer-line for opening-lines (count 1, same strength) and accepts an open round-1 question Retired: interview-pacing-round-arm, -single-arm, -single-arm-open-b, -single-arm-open-d, -single-arm-step-2, -single-arm-step-3. The experiment concluded (run 3: ROUNDS HOLD UP) and measured a layout the owner changed (question_detail levels); its graders are frozen by its own rule. Recorded as Amendment 3 in interview-pacing.md; question-quality coverage moves to the new case interview-round-quality-standard (next commit). Legacy skills/interview/evals/evals.json: entries 1, 5, 7, 8, 20, 23, 25, 27, 28, 43, 44, 48 edited where an expectation became false (intake round, standard print, terms written after confirmation). Re-pins in tests/interview-defenses.test.sh (the read of each case's diff shows the defense intact): - case 1 still resolves codebase-answerable questions without asking: 5e38782253a410890320cddc3c5c44d71ab64666bb596bc954167f1447059bdd -> 1065f9d322d8ae3b5179ac13ec75e615044f42b7ada5455737d285048032a217; the codebase-resolution expectation is unchanged; only the recommendation expectation is scoped to decision questions. - case 8 still asks decisions and only looks up facts: 869a487d5f0abe19cdbe5693b90beabb55f56fbdacf6642bedad62494ce029f5 -> bedb9fd8e5923395d229e36cf50479530e58abcd8503a3ae989212162b789c45; the format is still looked up and the precedence still asked with a recommendation, now placed in the first decision round. - case 25 still refuses to settle a row on a superseded recommendation: 186a511bef86bbc7fe9679359c350c39111c3eab0c8761eb8f98196882f40b9f -> a3e7ffd4412ccfb51465f0be3d871221a6c288d53131c8436ef6025bc30974bd; only "recommendation with Basis" became "recommendation with its reason (its basis)"; the supersede and revalidate expectations are unchanged. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh --- .../graders/context-claims-referenced.md | 2 +- .../graders/scope-check-line.md | 6 ++++ .../prompt.md | 12 ++++--- .../samples/scope-check-line.json | 22 ++++++++++++ .../graders/scope-check-line.md | 5 +++ .../interview-intent-asked-recorded/prompt.md | 10 ++++-- .../samples/scope-check-line.json | 18 ++++++++++ .../graders/scope-check-line.md | 6 ++++ .../prompt.md | 14 +++++--- .../samples/scope-check-line.json | 22 ++++++++++++ .../case.yaml | 15 +++++--- .../samples/scope-check-line.json | 22 ++++++++++++ .../case.yaml | 15 +++++--- .../samples/scope-check-line.json | 22 ++++++++++++ .../graders/closing-probe.md | 7 ---- .../graders/covers-a.md | 7 ---- .../graders/covers-b.md | 7 ---- .../graders/covers-c.md | 7 ---- .../graders/covers-d.md | 7 ---- .../graders/quality-a.md | 7 ---- .../graders/quality-b.md | 7 ---- .../graders/quality-c.md | 7 ---- .../graders/quality-d.md | 7 ---- .../interview-pacing-round-arm/prompt.md | 17 ---------- .../samples/closing-probe.json | 22 ------------ .../samples/covers-a.json | 30 ---------------- .../samples/covers-b.json | 22 ------------ .../samples/covers-c.json | 30 ---------------- .../samples/covers-d.json | 34 ------------------- .../samples/quality-a.json | 34 ------------------- .../samples/quality-b.json | 30 ---------------- .../samples/quality-c.json | 30 ---------------- .../samples/quality-d.json | 34 ------------------- .../graders/closing-probe.md | 7 ---- .../graders/covers-a.md | 7 ---- .../graders/covers-b.md | 7 ---- .../graders/covers-c.md | 7 ---- .../graders/covers-d.md | 7 ---- .../graders/one-question.md | 8 ----- .../graders/one-recommendation.md | 8 ----- .../graders/quality-a.md | 7 ---- .../graders/quality-b.md | 7 ---- .../graders/quality-c.md | 7 ---- .../graders/quality-d.md | 7 ---- .../prompt.md | 19 ----------- .../samples/closing-probe.json | 22 ------------ .../samples/covers-a.json | 30 ---------------- .../samples/covers-b.json | 22 ------------ .../samples/covers-c.json | 30 ---------------- .../samples/covers-d.json | 34 ------------------- .../samples/one-question.json | 22 ------------ .../samples/one-recommendation.json | 10 ------ .../samples/quality-a.json | 34 ------------------- .../samples/quality-b.json | 30 ---------------- .../samples/quality-c.json | 30 ---------------- .../samples/quality-d.json | 34 ------------------- .../graders/closing-probe.md | 7 ---- .../graders/covers-a.md | 7 ---- .../graders/covers-b.md | 7 ---- .../graders/covers-c.md | 7 ---- .../graders/covers-d.md | 7 ---- .../graders/one-question.md | 8 ----- .../graders/one-recommendation.md | 8 ----- .../graders/quality-a.md | 7 ---- .../graders/quality-b.md | 7 ---- .../graders/quality-c.md | 7 ---- .../graders/quality-d.md | 7 ---- .../prompt.md | 19 ----------- .../samples/closing-probe.json | 22 ------------ .../samples/covers-a.json | 30 ---------------- .../samples/covers-b.json | 22 ------------ .../samples/covers-c.json | 30 ---------------- .../samples/covers-d.json | 34 ------------------- .../samples/one-question.json | 22 ------------ .../samples/one-recommendation.json | 10 ------ .../samples/quality-a.json | 34 ------------------- .../samples/quality-b.json | 30 ---------------- .../samples/quality-c.json | 30 ---------------- .../samples/quality-d.json | 34 ------------------- .../graders/closing-probe.md | 7 ---- .../graders/covers-a.md | 7 ---- .../graders/covers-b.md | 7 ---- .../graders/covers-c.md | 7 ---- .../graders/covers-d.md | 7 ---- .../graders/one-question.md | 8 ----- .../graders/one-recommendation.md | 8 ----- .../graders/quality-a.md | 7 ---- .../graders/quality-b.md | 7 ---- .../graders/quality-c.md | 7 ---- .../graders/quality-d.md | 7 ---- .../prompt.md | 19 ----------- .../samples/closing-probe.json | 22 ------------ .../samples/covers-a.json | 30 ---------------- .../samples/covers-b.json | 22 ------------ .../samples/covers-c.json | 30 ---------------- .../samples/covers-d.json | 34 ------------------- .../samples/one-question.json | 22 ------------ .../samples/one-recommendation.json | 10 ------ .../samples/quality-a.json | 34 ------------------- .../samples/quality-b.json | 30 ---------------- .../samples/quality-c.json | 30 ---------------- .../samples/quality-d.json | 34 ------------------- .../graders/closing-probe.md | 7 ---- .../graders/covers-a.md | 7 ---- .../graders/covers-b.md | 7 ---- .../graders/covers-c.md | 7 ---- .../graders/covers-d.md | 7 ---- .../graders/one-question.md | 8 ----- .../graders/one-recommendation.md | 8 ----- .../graders/quality-a.md | 7 ---- .../graders/quality-b.md | 7 ---- .../graders/quality-c.md | 7 ---- .../graders/quality-d.md | 7 ---- .../prompt.md | 19 ----------- .../samples/closing-probe.json | 22 ------------ .../samples/covers-a.json | 30 ---------------- .../samples/covers-b.json | 22 ------------ .../samples/covers-c.json | 30 ---------------- .../samples/covers-d.json | 34 ------------------- .../samples/one-question.json | 22 ------------ .../samples/one-recommendation.json | 10 ------ .../samples/quality-a.json | 34 ------------------- .../samples/quality-b.json | 30 ---------------- .../samples/quality-c.json | 30 ---------------- .../samples/quality-d.json | 34 ------------------- .../graders/closing-probe.md | 7 ---- .../graders/covers-a.md | 7 ---- .../graders/covers-b.md | 7 ---- .../graders/covers-c.md | 7 ---- .../graders/covers-d.md | 7 ---- .../graders/one-question.md | 8 ----- .../graders/one-recommendation.md | 8 ----- .../graders/quality-a.md | 7 ---- .../graders/quality-b.md | 7 ---- .../graders/quality-c.md | 7 ---- .../graders/quality-d.md | 7 ---- .../interview-pacing-single-arm/prompt.md | 17 ---------- .../samples/closing-probe.json | 22 ------------ .../samples/covers-a.json | 30 ---------------- .../samples/covers-b.json | 22 ------------ .../samples/covers-c.json | 30 ---------------- .../samples/covers-d.json | 34 ------------------- .../samples/one-question.json | 22 ------------ .../samples/one-recommendation.json | 10 ------ .../samples/quality-a.json | 34 ------------------- .../samples/quality-b.json | 30 ---------------- .../samples/quality-c.json | 30 ---------------- .../samples/quality-d.json | 34 ------------------- plugins/planning/evals/interview-pacing.md | 10 ++++++ .../graders/checkable-basis-and-flip.md | 3 +- .../graders/scope-check-line.md | 6 ++++ .../prompt.md | 16 +++++---- .../samples/scope-check-line.json | 22 ++++++++++++ .../graders/scope-check-line.md | 6 ++++ .../graders/topic-groups.md | 3 +- .../interview-round-topic-groups/prompt.md | 6 ++-- .../samples/scope-check-line.json | 22 ++++++++++++ .../case.yaml | 15 +++++--- .../samples/scope-check-line.json | 22 ++++++++++++ .../graders/scope-check-line.md | 6 ++++ .../interview-scope-cut-moscow/prompt.md | 8 +++-- .../samples/scope-check-line.json | 22 ++++++++++++ .../graders/no-basis-line.md | 7 ++++ .../graders/one-question-full-layout.md | 4 +-- .../prompt.md | 10 +++--- .../samples/no-basis-line.json | 22 ++++++++++++ .../graders/one-offer-line.md | 8 ----- .../graders/opening-lines.md | 8 +++++ .../graders/proceeds-to-round.md | 8 ++--- .../prompt.md | 10 +++--- .../samples/one-offer-line.json | 22 ------------ .../samples/opening-lines.json | 22 ++++++++++++ .../graders/scope-check-line.md | 6 ++++ .../prompt.md | 12 +++---- .../samples/scope-check-line.json | 22 ++++++++++++ .../skills/interview/evals/evals.json | 34 +++++++++---------- .../planning/tests/interview-defenses.test.sh | 6 ++-- 177 files changed, 446 insertions(+), 2437 deletions(-) create mode 100644 plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/scope-check-line.md create mode 100644 plugins/planning/evals/interview-context-unfamiliar-term-upstream/samples/scope-check-line.json create mode 100644 plugins/planning/evals/interview-intent-asked-recorded/graders/scope-check-line.md create mode 100644 plugins/planning/evals/interview-intent-asked-recorded/samples/scope-check-line.json create mode 100644 plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/scope-check-line.md create mode 100644 plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/scope-check-line.json create mode 100644 plugins/planning/evals/interview-option-existing-control-drops/samples/scope-check-line.json create mode 100644 plugins/planning/evals/interview-option-known-repetition-keeps/samples/scope-check-line.json delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/graders/closing-probe.md delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/graders/covers-a.md delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/graders/covers-b.md delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/graders/covers-c.md delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/graders/covers-d.md delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/graders/quality-a.md delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/graders/quality-b.md delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/graders/quality-c.md delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/graders/quality-d.md delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/prompt.md delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/samples/closing-probe.json delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/samples/covers-a.json delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/samples/covers-b.json delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/samples/covers-c.json delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/samples/covers-d.json delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/samples/quality-a.json delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/samples/quality-b.json delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/samples/quality-c.json delete mode 100644 plugins/planning/evals/interview-pacing-round-arm/samples/quality-d.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/graders/closing-probe.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-a.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-b.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-c.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-d.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-question.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-recommendation.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-a.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-b.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-c.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-d.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/prompt.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/samples/closing-probe.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-a.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-b.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-c.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-d.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-question.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-recommendation.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-a.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-b.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-c.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-d.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/graders/closing-probe.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-a.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-b.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-c.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-d.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-question.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-recommendation.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-a.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-b.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-c.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-d.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/prompt.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/samples/closing-probe.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-a.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-b.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-c.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-d.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-question.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-recommendation.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-a.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-b.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-c.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-d.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/graders/closing-probe.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-a.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-b.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-c.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-d.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-question.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-recommendation.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-a.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-b.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-c.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-d.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/prompt.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/samples/closing-probe.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-a.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-b.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-c.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-d.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-question.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-recommendation.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-a.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-b.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-c.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-d.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/graders/closing-probe.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-a.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-b.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-c.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-d.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-question.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-recommendation.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-a.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-b.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-c.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-d.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/prompt.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/samples/closing-probe.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-a.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-b.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-c.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-d.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-question.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-recommendation.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-a.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-b.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-c.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-d.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/graders/closing-probe.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/graders/covers-a.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/graders/covers-b.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/graders/covers-c.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/graders/covers-d.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/graders/one-question.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/graders/one-recommendation.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/graders/quality-a.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/graders/quality-b.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/graders/quality-c.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/graders/quality-d.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/prompt.md delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/samples/closing-probe.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/samples/covers-a.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/samples/covers-b.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/samples/covers-c.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/samples/covers-d.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/samples/one-question.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/samples/one-recommendation.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/samples/quality-a.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/samples/quality-b.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/samples/quality-c.json delete mode 100644 plugins/planning/evals/interview-pacing-single-arm/samples/quality-d.json create mode 100644 plugins/planning/evals/interview-recommendation-basis-and-flip/graders/scope-check-line.md create mode 100644 plugins/planning/evals/interview-recommendation-basis-and-flip/samples/scope-check-line.json create mode 100644 plugins/planning/evals/interview-round-topic-groups/graders/scope-check-line.md create mode 100644 plugins/planning/evals/interview-round-topic-groups/samples/scope-check-line.json create mode 100644 plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/samples/scope-check-line.json create mode 100644 plugins/planning/evals/interview-scope-cut-moscow/graders/scope-check-line.md create mode 100644 plugins/planning/evals/interview-scope-cut-moscow/samples/scope-check-line.json create mode 100644 plugins/planning/evals/interview-single-pacing-one-question/graders/no-basis-line.md create mode 100644 plugins/planning/evals/interview-single-pacing-one-question/samples/no-basis-line.json delete mode 100644 plugins/planning/evals/interview-surface-unset-offer-one-line/graders/one-offer-line.md create mode 100644 plugins/planning/evals/interview-surface-unset-offer-one-line/graders/opening-lines.md delete mode 100644 plugins/planning/evals/interview-surface-unset-offer-one-line/samples/one-offer-line.json create mode 100644 plugins/planning/evals/interview-surface-unset-offer-one-line/samples/opening-lines.json create mode 100644 plugins/planning/evals/interview-term-clash-qualified-name/graders/scope-check-line.md create mode 100644 plugins/planning/evals/interview-term-clash-qualified-name/samples/scope-check-line.json diff --git a/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/context-claims-referenced.md b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/context-claims-referenced.md index b0ba135a6f5..cb3215034cf 100644 --- a/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/context-claims-referenced.md +++ b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/context-claims-referenced.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '^(?![\s\S]*?(?:(?:^|\n)[ \t>*_-]*(?:\*\*)?Context(?:\*\*)?[ \t]*:(?:\*\*)?(?:(?![ \t]*(?:\n|$))(?:(?!(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d))[^\n])+(?=\n|$)|(?:[ \t]*\n(?:[ \t]*\n)?|[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d)[^\n]*\n)(?:(?![ \t>*_-]*(?:\*\*)?(?:❓|➡|My recommendation|Recommendation\b|Proposal\b|Basis\b|Checked against|Alternatives\b|Source[ \t]*:|Inference\b|Flips if|Withheld\b|Commits you to|Q\d+[.):]))[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d)[^\n]*\n)*(?![ \t>*_-]*(?:\*\*)?(?:❓|➡|My recommendation|Recommendation\b|Proposal\b|Basis\b|Checked against|Alternatives\b|Source[ \t]*:|Inference\b|Flips if|Withheld\b|Commits you to|Q\d+[.):]))(?![ \t]*(?:\n|$))(?:(?!(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d))[^\n])+(?=\n|$))|(?:^|\n)[^A-Za-z0-9\n]*Q\d+[.):](?:(?!\n[ \t>*_-]*(?:\*\*)?Context(?:\*\*)?[ \t]*:)[\s\S])*?\n[ \t>*_-]*(?:\*\*)?(?:➡|My recommendation)))[\s\S]*?(?:^|\n)[ \t>*_-]*(?:\*\*)?Context(?:\*\*)?[ \t]*:(?:\*\*)?(?:[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d)|[ \t]*\n(?:[ \t]*\n)?[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d))' +pattern: '^(?![\s\S]*?(?:(?:^|\n)[ \t>*_-]*(?:\*\*)?Context(?:\*\*)?[ \t]*:(?:\*\*)?(?:(?![ \t]*(?:\n|$))(?:(?!(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d))[^\n])+(?=\n|$)|(?:[ \t]*\n(?:[ \t]*\n)?|[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d)[^\n]*\n)(?:(?![ \t>*_-]*(?:\*\*)?(?:❓|➡|My recommendation|Recommendation\b|Proposal\b|Basis\b|Checked against|Alternatives\b|Source[ \t]*:|Inference\b|Flips if|Withheld\b|Discovery\b|Held\b|Second view|Departs from|Scope check|Commits you to|Q\d+[.):]))[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d)[^\n]*\n)*(?![ \t>*_-]*(?:\*\*)?(?:❓|➡|My recommendation|Recommendation\b|Proposal\b|Basis\b|Checked against|Alternatives\b|Source[ \t]*:|Inference\b|Flips if|Withheld\b|Discovery\b|Held\b|Second view|Departs from|Scope check|Commits you to|Q\d+[.):]))(?![ \t]*(?:\n|$))(?:(?!(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d))[^\n])+(?=\n|$))|(?:^|\n)[^A-Za-z0-9\n]*Q\d+[.):](?:(?!\n[ \t>*_-]*(?:\*\*)?Context(?:\*\*)?[ \t]*:)[\s\S])*?\n[ \t>*_-]*(?:\*\*)?(?:➡|My recommendation)))[\s\S]*?(?:^|\n)[ \t>*_-]*(?:\*\*)?Context(?:\*\*)?[ \t]*:(?:\*\*)?(?:[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d)|[ \t]*\n(?:[ \t]*\n)?[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d))' flags: i arm: both --- diff --git a/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/scope-check-line.md b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-context-unfamiliar-term-upstream/prompt.md b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/prompt.md index 7eaa7096807..795de60445c 100644 --- a/plugins/planning/evals/interview-context-unfamiliar-term-upstream/prompt.md +++ b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/prompt.md @@ -1,11 +1,15 @@ --- -description: "First interview round on absorbing an upstream triage tool's cluster step, which rests on a coined upstream term (dedupe fingerprint) and on what the vendored README says. Why hard: the agent has read both files, so a question that names the cluster step and the fingerprint without defining them, or a decision question that does not say what upstream does and what the repository has today, reads as complete to it while the user would have to ask before answering. The prompt states the quality bar, so round 1 asks no bar question, which has no upstream gap to show." +description: "First interview round on absorbing an upstream triage tool's cluster step, which rests on a coined upstream term (dedupe fingerprint) and on what the vendored README says. Why hard: the agent has read both files, so a question that names the cluster step and the fingerprint without defining them, or a decision question that does not say what upstream does and what the repository has today, reads as complete to it while the user would have to ask before answering. The prompt states the quality bar and the round-1 answers, so the first decision round asks no bar question, which has no upstream gap to show, and carries a Scope check: header line." tags: [interview, context-block, scaffold] runs: 3 max_turns: 20 timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Write] -expected_outcome: "Every question in round 1 opens with a Context block that defines the dedupe fingerprint and the cluster step wherever the question uses them, says in each decision question what the upstream README does and what triage/intake.py does today, and cites a file:line or link on each line, so the user can answer every question without a follow-up" +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write] +expected_outcome: "Every question in round 2 opens with a Context block that defines the dedupe fingerprint and the cluster step wherever the question uses them, says in each decision question what the upstream README does and what triage/intake.py does today, and cites a file:line or link on each line, so the user can answer every question without a follow-up" --- -/planning:interview me Should our triage intake gain a cluster step like the upstream triage tool's? Its docs are vendored at vendor/upstream-triage/README.md, and our intake is triage/intake.py. This is a throwaway prototype. Ask me your first round, then stop for my answers. +/planning:interview me Should our triage intake gain a cluster step like the upstream triage tool's? Its docs are vendored at vendor/upstream-triage/README.md, and our intake is triage/intake.py. This is a throwaway prototype. + +Round 1 is answered: Q1, the problem is that one crash reported by many users opens many duplicate issues; Q2, the users are the two maintainers who triage those issues. + +Ask me round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-context-unfamiliar-term-upstream/samples/scope-check-line.json b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-intent-asked-recorded/graders/scope-check-line.md b/plugins/planning/evals/interview-intent-asked-recorded/graders/scope-check-line.md new file mode 100644 index 00000000000..d4655bb06bd --- /dev/null +++ b/plugins/planning/evals/interview-intent-asked-recorded/graders/scope-check-line.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +--- diff --git a/plugins/planning/evals/interview-intent-asked-recorded/prompt.md b/plugins/planning/evals/interview-intent-asked-recorded/prompt.md index 692ee14be42..c80438778f1 100644 --- a/plugins/planning/evals/interview-intent-asked-recorded/prompt.md +++ b/plugins/planning/evals/interview-intent-asked-recorded/prompt.md @@ -1,11 +1,15 @@ --- -description: "Engineering task with one external provider and nothing said about others: the interview asks the intent question (is a second variant known, a category, or speculative) with a recommendation, and registers it in the ledger at ask-time" +description: "Engineering task with one external provider and nothing said about others, past the intake round: the interview asks the intent question (is a second variant known, a category, or speculative) with a recommendation in the first decision round, registers it in the ledger at ask-time, and heads the round with a Scope check: line" tags: [interview, intent, engineering-defaults] runs: 3 max_turns: 30 timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit] -expected_outcome: "Round 1 asks whether a second address-validation provider or per-client variation is known, expected as a category, or speculative, with a recommendation, and the ledger .work/address-validation/interview-checklist.md carries that question as an open register row" +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] +expected_outcome: "Round 2 asks whether a second address-validation provider or per-client variation is known, expected as a category, or speculative, with a recommendation; the round header carries a Scope check: line; and the ledger .work/address-validation/interview-checklist.md carries that question as an open register row" --- /planning:interview Add delivery-address validation to checkout using the Loqate API, so an undeliverable address is caught before payment. This goes to production. Use the topic slug `address-validation`. + +Round 1 is answered: Q1, the problem is orders shipped to addresses the carrier cannot deliver to; Q2, the users are shoppers at checkout and the support team that handles returned parcels. + +Ask round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-intent-asked-recorded/samples/scope-check-line.json b/plugins/planning/evals/interview-intent-asked-recorded/samples/scope-check-line.json new file mode 100644 index 00000000000..246aa17cb9c --- /dev/null +++ b/plugins/planning/evals/interview-intent-asked-recorded/samples/scope-check-line.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "**Round 2.** A production address check before payment, for shoppers and support.\nScope check: 3 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Is a second address-validation provider known, a category, or speculative?**", + "why": "the round header carries the Scope check: line" + } + ], + "fail": [ + { + "answer": "**Round 2.** A production address check before payment.\n\n❓ Q3: **Is a second address-validation provider known, a category, or speculative?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Is a second provider coming?**", + "why": "the words appear mid-sentence, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/scope-check-line.md b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/prompt.md b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/prompt.md index c5d1e697c95..cba66fd93d5 100644 --- a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/prompt.md +++ b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/prompt.md @@ -1,11 +1,15 @@ --- -description: "Hard case. Why hard: the prompt floats a side idea (rewriting the whole admin page in React) whose natural recommendation is negative, so the question about it is easily worded 'Should we rewrite it?' with a recommended 'No', where a reply of 'yes' rejects the recommendation. Upstream mattpocock/skills#706." +description: "Hard case. Why hard: the prompt floats a side idea (rewriting the whole admin page in React) whose natural recommendation is negative, so the question about it is easily worded 'Should we rewrite it?' with a recommended 'No', where a reply of 'yes' rejects the recommendation. Upstream mattpocock/skills#706. The prompt starts past the intake round, so the first decision round carries recommendations and its Scope check: header line." tags: [interview, question-wording, negative-recommendation] runs: 3 -max_turns: 12 -timeout_seconds: 600 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "Round 1 asks about the React rewrite, recommends keeping it out of this change, and words that question so that answering 'yes' accepts the recommendation" +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "Round 2, headed by a Scope check: line, asks about the React rewrite, recommends keeping it out of this change, and words that question so that answering 'yes' accepts the recommendation" --- /planning:interview Add a CSV export button to the admin orders page so ops can download the filtered order list. I'm also wondering whether we should rewrite the whole admin page in React while we're in there. + +Round 1 is answered: Q1, the problem is that ops copy the filtered order list by hand into a spreadsheet each morning; Q2, the users are the ops team. + +Ask round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/scope-check-line.json b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-option-existing-control-drops/case.yaml b/plugins/planning/evals/interview-option-existing-control-drops/case.yaml index a2e8c059f87..6234a09484c 100644 --- a/plugins/planning/evals/interview-option-existing-control-drops/case.yaml +++ b/plugins/planning/evals/interview-option-existing-control-drops/case.yaml @@ -2,13 +2,13 @@ schema_version: "1.1" name: interview-option-existing-control-drops description: An interview over an option that adds a stable branch, a workflow, a ruleset and a bot app recommends dropping them for the protection and end-to-end gate main already has tags: [interview, recommendation, existing-control, scaffold] -expected_outcome: The first round's recommendation on how consumers get the latest settings is to keep installing from main, naming main's existing protection (blocked direct pushes, required review, required consumer-e2e check) or the pull request end-to-end job as the control that already covers the risk, and it does not recommend building the stable branch, its workflow, its ruleset or the bot app. No Write or Edit touches PLAN.md. +expected_outcome: The first decision round, headed by a Scope check: line, gives a recommendation on how consumers get the latest settings is to keep installing from main, naming main's existing protection (blocked direct pushes, required review, required consumer-e2e check) or the pull request end-to-end job as the control that already covers the risk, and it does not recommend building the stable branch, its workflow, its ruleset or the bot app. No Write or Edit touches PLAN.md. context: scaffold_script: scaffold.sh execution: - prompt: "Use the interview skill from the planning plugin, topic latest-settings. Problem: consumers of this repository must always run the latest settings, and must never pick up a broken commit. Options I am weighing: (a) a long-lived `stable` branch consumers install from, a workflow that fast-forwards it to `main` after each merge, a ruleset blocking direct pushes to `stable`, and a bot app the workflow authenticates as to push past that ruleset; (b) something else you suggest. Settle mechanism decisions in this interview. I'm here and will answer; ask me your first round." - allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit] - max_turns: 20 + prompt: "Use the interview skill from the planning plugin, topic latest-settings. Problem: consumers of this repository must always run the latest settings, and must never pick up a broken commit. Options I am weighing: (a) a long-lived `stable` branch consumers install from, a workflow that fast-forwards it to `main` after each merge, a ruleset blocking direct pushes to `stable`, and a bot app the workflow authenticates as to push past that ruleset; (b) something else you suggest. Settle mechanism decisions in this interview. Round 1 is answered: Q1, the problem is as stated above; Q2, the users are the maintainers of the consumer repositories that install these settings. I'm here and will answer; ask me round 2." + allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] + max_turns: 30 timeout_seconds: 900 graders: - name: skill-fired @@ -20,7 +20,12 @@ graders: type: llm arm: both focus: trace - criteria: "PASS only if all hold: (1) a recommendation in the reply is to have consumers keep installing from main (or otherwise to drop the stable branch, its fast-forward workflow, its ruleset and the bot app); (2) its basis names an existing control that already covers the same risk: the protection on main (blocked direct pushes, required review, required consumer-e2e check) or the pull request CI job that installs the settings into a sample consumer; (3) no recommendation in the reply is to build any of the four proposed mechanisms; (4) where it relies on main's protection, it says that protection was read only from a document (docs/branch-protection.md) or labels it documented or unverified, never verified from an enforcing source. FAIL if the reply recommends building any of them, if it names no existing control, if it presents main's document-only protection as verified, or if it asks only about details of the proposed mechanisms (branch name, workflow trigger, app permissions)." + criteria: "PASS only if all hold: (1) a recommendation in the reply is to have consumers keep installing from main (or otherwise to drop the stable branch, its fast-forward workflow, its ruleset and the bot app); (2) its printed reason names an existing control that already covers the same risk: the protection on main (blocked direct pushes, required review, required consumer-e2e check) or the pull request CI job that installs the settings into a sample consumer; (3) no recommendation in the reply is to build any of the four proposed mechanisms; (4) where it relies on main's protection, it says that protection was read only from a document (docs/branch-protection.md) or labels it documented or unverified, never verified from an enforcing source. FAIL if the reply recommends building any of them, if it names no existing control, if it presents main's document-only protection as verified, or if it asks only about details of the proposed mechanisms (branch name, workflow trigger, app permissions)." + - name: scope-check-line + type: regex + pattern: '(?:^|\n)[^\n]{0,6}Scope check:' + target: last_message + arm: both - name: no-plan-write type: tool_used tool: Write diff --git a/plugins/planning/evals/interview-option-existing-control-drops/samples/scope-check-line.json b/plugins/planning/evals/interview-option-existing-control-drops/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-option-existing-control-drops/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-option-known-repetition-keeps/case.yaml b/plugins/planning/evals/interview-option-known-repetition-keeps/case.yaml index 9123892c4e1..938ec4a2a8c 100644 --- a/plugins/planning/evals/interview-option-known-repetition-keeps/case.yaml +++ b/plugins/planning/evals/interview-option-known-repetition-keeps/case.yaml @@ -2,13 +2,13 @@ schema_version: "1.1" name: interview-option-known-repetition-keeps description: An interview over an option to add an interface whose second consumer is a stated requirement with the same send shape recommends keeping the interface rather than calling it speculative tags: [interview, recommendation, known-repetition, scaffold] -expected_outcome: The first round's recommendation on how to structure the senders keeps the Notifier interface (or an equivalent shared abstraction), citing the stated SMS requirement R2 in docs/requirements.md as the second consumer and the shared send(recipient, message) shape. It does not call the interface speculative generality or YAGNI and does not ask to wait for a third channel. No Write or Edit touches PLAN.md. +expected_outcome: The first decision round, headed by a Scope check: line, gives a recommendation on how to structure the senders keeps the Notifier interface (or an equivalent shared abstraction), citing the stated SMS requirement R2 in docs/requirements.md as the second consumer and the shared send(recipient, message) shape. It does not call the interface speculative generality or YAGNI and does not ask to wait for a third channel. No Write or Edit touches PLAN.md. context: scaffold_script: scaffold.sh execution: - prompt: "Use the interview skill from the planning plugin, topic order-confirmations. Problem: requirement R1 in docs/requirements.md, the email order confirmation, is next. Options I am weighing: (a) add a `Notifier` interface with one method, `send(recipient, message)`, now, with the email sender as its first implementation; (b) write the email sender directly and extract an interface later. Settle mechanism decisions in this interview. I'm here and will answer; ask me your first round." - allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit] - max_turns: 20 + prompt: "Use the interview skill from the planning plugin, topic order-confirmations. Problem: requirement R1 in docs/requirements.md, the email order confirmation, is next. Options I am weighing: (a) add a `Notifier` interface with one method, `send(recipient, message)`, now, with the email sender as its first implementation; (b) write the email sender directly and extract an interface later. Settle mechanism decisions in this interview. Round 1 is answered: Q1, the problem is as stated above; Q2, the users are shoppers who place orders and need a confirmation. I'm here and will answer; ask me round 2." + allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] + max_turns: 30 timeout_seconds: 900 graders: - name: skill-fired @@ -20,7 +20,12 @@ graders: type: llm arm: both focus: trace - criteria: "PASS only if all hold: (1) where the reply recommends how to structure the senders, the recommendation keeps the Notifier interface or an equivalent shared abstraction; (2) its basis cites the stated SMS requirement (R2 in docs/requirements.md) as a second consumer that already exists; (3) no recommendation drops the interface as speculative generality, YAGNI or premature abstraction, and none asks to wait for a third channel. FAIL if the reply recommends dropping or deferring the interface, or keeps it without citing the stated second consumer." + criteria: "PASS only if all hold: (1) where the reply recommends how to structure the senders, the recommendation keeps the Notifier interface or an equivalent shared abstraction; (2) its printed reason cites the stated SMS requirement (R2 in docs/requirements.md) as a second consumer that already exists; (3) no recommendation drops the interface as speculative generality, YAGNI or premature abstraction, and none asks to wait for a third channel. FAIL if the reply recommends dropping or deferring the interface, or keeps it without citing the stated second consumer." + - name: scope-check-line + type: regex + pattern: '(?:^|\n)[^\n]{0,6}Scope check:' + target: last_message + arm: both - name: no-plan-write type: tool_used tool: Write diff --git a/plugins/planning/evals/interview-option-known-repetition-keeps/samples/scope-check-line.json b/plugins/planning/evals/interview-option-known-repetition-keeps/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-option-known-repetition-keeps/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/closing-probe.md b/plugins/planning/evals/interview-pacing-round-arm/graders/closing-probe.md deleted file mode 100644 index 1fd5f0c18bf..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/closing-probe.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: 'something else that would change' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/covers-a.md b/plugins/planning/evals/interview-pacing-round-arm/graders/covers-a.md deleted file mode 100644 index 765e72238ec..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/covers-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/covers-b.md b/plugins/planning/evals/interview-pacing-round-arm/graders/covers-b.md deleted file mode 100644 index 7925aa6602e..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/covers-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/covers-c.md b/plugins/planning/evals/interview-pacing-round-arm/graders/covers-c.md deleted file mode 100644 index c5213709dcb..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/covers-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/covers-d.md b/plugins/planning/evals/interview-pacing-round-arm/graders/covers-d.md deleted file mode 100644 index f76266389ee..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/covers-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-a.md b/plugins/planning/evals/interview-pacing-round-arm/graders/quality-a.md deleted file mode 100644 index 598ed376d33..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-b.md b/plugins/planning/evals/interview-pacing-round-arm/graders/quality-b.md deleted file mode 100644 index fc9c858a09c..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-c.md b/plugins/planning/evals/interview-pacing-round-arm/graders/quality-c.md deleted file mode 100644 index 92b82adfb7d..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-d.md b/plugins/planning/evals/interview-pacing-round-arm/graders/quality-d.md deleted file mode 100644 index 0ad035250ac..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/prompt.md b/plugins/planning/evals/interview-pacing-round-arm/prompt.md deleted file mode 100644 index 42c6c0436f4..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/prompt.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -description: "Round arm of the round-versus-single pacing comparison (paired with the five interview-pacing-single-arm cases, same task and facts). Source: the consumer's own writing, synthetic. Why hard: the four open decisions for an invoice CSV export (date range, amount format across currencies, who may export, large exports) are not listed as a frontier; the skill must find each one in the stated facts and ask all of them in one round, each in the full layout. The easy failures are to assume one silently, or to drop the basis or alternatives as the round grows. Graders are regexes, one coverage and one quality slot per planted decision; the deciding rule is in ../interview-pacing.md." -tags: [interview, pacing, pacing-comparison, hard, no-network] -runs: 6 -max_turns: 30 -timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "Round 2 asks, in one message, a question on each of date range, amount format and currencies, who may export, and large exports, every question in the full layout (context citing a stated fact, recommendation with Basis or a Withheld line, alternatives), ending with the closing probe. A separate question deferring the large-export mechanism to planning is allowed and does not replace the large-export question" ---- - -/planning:interview me Add a CSV export of invoices to the billing admin page. - -Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. - -What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. - -Ask round 2 now, the whole round in one message, then stop for my answers. diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/closing-probe.json b/plugins/planning/evals/interview-pacing-round-arm/samples/closing-probe.json deleted file mode 100644 index e04406c4c79..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/closing-probe.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nReply when ready.", - "why": "no closing probe" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/covers-a.json b/plugins/planning/evals/interview-pacing-round-arm/samples/covers-a.json deleted file mode 100644 index 9a48f709d64..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/covers-a.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing, one question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "date range appears only in a recap line, not as a question" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Which date puts an invoice into a calendar month's export?**\nContext:\n- Finance closes its books on the calendar month, and an invoice has issue, due and paid dates (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "which date places an invoice in a month is a sub-decision, not the date range one export covers" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/covers-b.json b/plugins/planning/evals/interview-pacing-round-arm/samples/covers-b.json deleted file mode 100644 index 368a0108dac..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/covers-b.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "amount and currency format appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/covers-c.json b/plugins/planning/evals/interview-pacing-round-arm/samples/covers-c.json deleted file mode 100644 index d316c5e9396..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/covers-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing after A and D were answered, one question on who may export; the recap line asks nothing" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "who may export, phrased as who may start and download the file" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "who may export appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/covers-d.json b/plugins/planning/evals/interview-pacing-round-arm/samples/covers-d.json deleted file mode 100644 index cdb7c3a2087..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/covers-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be prepared after the click and downloaded once it's ready, instead of being built during the click like the page's other downloads?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D phrased as preparing the file after the click and downloading it once it's ready" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D appears only inside the question that defers the mechanism to planning; D itself is never asked" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nSettled by evidence: a large export runs as a background job, because one month is about 40,000 rows and in-request downloads time out after 30 seconds (your notes).\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D settled in a preface line with live alternatives left, and never asked" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "a question on who may export that mentions the finished file is not D" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-a.json b/plugins/planning/evals/interview-pacing-round-arm/samples/quality-a.json deleted file mode 100644 index 37f9b860713..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-a.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one full-layout question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Exports need a scope.\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-b.json b/plugins/planning/evals/interview-pacing-round-arm/samples/quality-b.json deleted file mode 100644 index 5ee8637317c..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-b.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts need a format.\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-c.json b/plugins/planning/evals/interview-pacing-round-arm/samples/quality-c.json deleted file mode 100644 index 955c4a7135b..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Someone has to be allowed to export.\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-d.json b/plugins/planning/evals/interview-pacing-round-arm/samples/quality-d.json deleted file mode 100644 index 0290138ab91..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- Building the file takes a while for big months.\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "only the deferral question, D itself not asked" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/closing-probe.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/closing-probe.md deleted file mode 100644 index 1fd5f0c18bf..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/closing-probe.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: 'something else that would change' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-a.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-a.md deleted file mode 100644 index 765e72238ec..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-b.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-b.md deleted file mode 100644 index 7925aa6602e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-c.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-c.md deleted file mode 100644 index c5213709dcb..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-d.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-d.md deleted file mode 100644 index f76266389ee..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-question.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-question.md deleted file mode 100644 index fef3f505ea7..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-question.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-recommendation.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-recommendation.md deleted file mode 100644 index 638d3b48888..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-recommendation.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'My recommendation:|Withheld:' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-a.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-a.md deleted file mode 100644 index 598ed376d33..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-b.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-b.md deleted file mode 100644 index fc9c858a09c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-c.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-c.md deleted file mode 100644 index 92b82adfb7d..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-d.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-d.md deleted file mode 100644 index 0ad035250ac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/prompt.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/prompt.md deleted file mode 100644 index d80ecdfda4f..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/prompt.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -description: "Single arm of the round-versus-single pacing comparison with only the amount-and-currency decision (B) left open among the four planted ones (same task and facts as interview-pacing-round-arm). Source: the consumer's own writing, synthetic. The earlier single-pacing answers are stated in the prompt, because a case is one user turn and a resumed history carries no skill body. Why hard: the skill must ask the open decision in the full layout, one question only, without re-asking what is answered or deferred to planning. Graders are regexes; the deciding rule is in ../interview-pacing.md (Amendment 2)." -tags: [interview, pacing, pacing-comparison, hard, no-network] -runs: 6 -max_turns: 30 -timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply asks exactly one question, preferably on how amounts and the three currencies appear in the file, in the full layout with the closing probe, does not re-ask the date range, the month's date, who may export, the large-export approach or its mechanism, then stops" ---- - -/planning:interview me Add a CSV export of invoices to the billing admin page. - -Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. - -What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. - -We're doing round 2 one question at a time, and four are answered. Q4, how a large export completes: build the file in the background and offer it for download when it's ready, as you recommended; the job runner, file storage and ready notice wait for planning. Q5, what one export covers: one calendar month across all customer accounts, as you recommended. Q6, which date puts an invoice in a month: its issue date, with month boundaries in UTC. Q7, who may export: the `finance` role only. - -Ask me the next question, still one at a time, then stop for my answer. diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/closing-probe.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/closing-probe.json deleted file mode 100644 index e04406c4c79..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/closing-probe.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nReply when ready.", - "why": "no closing probe" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-a.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-a.json deleted file mode 100644 index 9a48f709d64..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-a.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing, one question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "date range appears only in a recap line, not as a question" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Which date puts an invoice into a calendar month's export?**\nContext:\n- Finance closes its books on the calendar month, and an invoice has issue, due and paid dates (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "which date places an invoice in a month is a sub-decision, not the date range one export covers" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-b.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-b.json deleted file mode 100644 index 368a0108dac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-b.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "amount and currency format appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-c.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-c.json deleted file mode 100644 index d316c5e9396..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing after A and D were answered, one question on who may export; the recap line asks nothing" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "who may export, phrased as who may start and download the file" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "who may export appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-d.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-d.json deleted file mode 100644 index cdb7c3a2087..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be prepared after the click and downloaded once it's ready, instead of being built during the click like the page's other downloads?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D phrased as preparing the file after the click and downloading it once it's ready" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D appears only inside the question that defers the mechanism to planning; D itself is never asked" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nSettled by evidence: a large export runs as a background job, because one month is about 40,000 rows and in-request downloads time out after 30 seconds (your notes).\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D settled in a preface line with live alternatives left, and never asked" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "a question on who may export that mentions the finished file is not D" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-question.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-question.json deleted file mode 100644 index 1449335f937..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-question.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the Later line names the rest without asking them" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the recap line is not a question" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "a whole round" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-recommendation.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-recommendation.json deleted file mode 100644 index 822b3d3d89e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-recommendation.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "pass": [ - {"answer": "Q4: **Which role may export?**\nContext: support cannot see payment details (user, round 2).\nMy recommendation: `finance` only.\nBasis: verified, user, round 2\nAlternatives:\n- (a) Both roles\n\nIs there something else that would change this recommendation?", "why": "one recommendation"}, - {"answer": "Q4: Which date range does one export cover?\nWithheld: whether finance ever reconciles part-months, which only finance can confirm.\n\nIs there something else that would change this?", "why": "one withheld question"} - ], - "fail": [ - {"answer": "", "why": "empty"}, - {"answer": "Q4: Which role? My recommendation: finance.\n\nQ5: Which range? My recommendation: one month.", "why": "two questions"} - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-a.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-a.json deleted file mode 100644 index 37f9b860713..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-a.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one full-layout question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Exports need a scope.\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-b.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-b.json deleted file mode 100644 index 5ee8637317c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-b.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts need a format.\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-c.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-c.json deleted file mode 100644 index 955c4a7135b..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Someone has to be allowed to export.\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-d.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-d.json deleted file mode 100644 index 0290138ab91..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- Building the file takes a while for big months.\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "only the deferral question, D itself not asked" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/closing-probe.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/closing-probe.md deleted file mode 100644 index 1fd5f0c18bf..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/closing-probe.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: 'something else that would change' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-a.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-a.md deleted file mode 100644 index 765e72238ec..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-b.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-b.md deleted file mode 100644 index 7925aa6602e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-c.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-c.md deleted file mode 100644 index c5213709dcb..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-d.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-d.md deleted file mode 100644 index f76266389ee..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-question.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-question.md deleted file mode 100644 index fef3f505ea7..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-question.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-recommendation.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-recommendation.md deleted file mode 100644 index 638d3b48888..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-recommendation.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'My recommendation:|Withheld:' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-a.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-a.md deleted file mode 100644 index 598ed376d33..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-b.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-b.md deleted file mode 100644 index fc9c858a09c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-c.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-c.md deleted file mode 100644 index 92b82adfb7d..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-d.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-d.md deleted file mode 100644 index 0ad035250ac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/prompt.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/prompt.md deleted file mode 100644 index 7781441eb9f..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/prompt.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -description: "Single arm of the round-versus-single pacing comparison with only the large-export decision (D) left open among the four planted ones (same task and facts as interview-pacing-round-arm). Source: the consumer's own writing, synthetic. The earlier single-pacing answers are stated in the prompt, because a case is one user turn and a resumed history carries no skill body. Why hard: the skill must ask how a 40,000-row export completes in the full layout, one question only, and not replace it with a question that only defers the mechanism to planning. Graders are regexes; the deciding rule is in ../interview-pacing.md (Amendment 2)." -tags: [interview, pacing, pacing-comparison, hard, no-network] -runs: 6 -max_turns: 30 -timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply asks exactly one question, preferably on how an export of about 40,000 rows completes when in-request downloads time out after 30 seconds, in the full layout with the closing probe, does not re-ask the date range, the month's date, the amount format or who may export, then stops" ---- - -/planning:interview me Add a CSV export of invoices to the billing admin page. - -Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. - -What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. - -We're doing round 2 one question at a time, and four are answered. Q4, what one export covers: one calendar month across all customer accounts, as you recommended. Q5, which date puts an invoice in a month: its issue date, with month boundaries in UTC. Q6, amounts: two-decimal numbers in the main currency unit with a separate currency-code column, no conversion and no total across currencies. Q7, who may export: the `finance` role only. - -Ask me the next question, still one at a time, then stop for my answer. diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/closing-probe.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/closing-probe.json deleted file mode 100644 index e04406c4c79..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/closing-probe.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nReply when ready.", - "why": "no closing probe" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-a.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-a.json deleted file mode 100644 index 9a48f709d64..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-a.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing, one question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "date range appears only in a recap line, not as a question" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Which date puts an invoice into a calendar month's export?**\nContext:\n- Finance closes its books on the calendar month, and an invoice has issue, due and paid dates (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "which date places an invoice in a month is a sub-decision, not the date range one export covers" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-b.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-b.json deleted file mode 100644 index 368a0108dac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-b.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "amount and currency format appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-c.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-c.json deleted file mode 100644 index d316c5e9396..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing after A and D were answered, one question on who may export; the recap line asks nothing" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "who may export, phrased as who may start and download the file" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "who may export appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-d.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-d.json deleted file mode 100644 index cdb7c3a2087..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be prepared after the click and downloaded once it's ready, instead of being built during the click like the page's other downloads?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D phrased as preparing the file after the click and downloading it once it's ready" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D appears only inside the question that defers the mechanism to planning; D itself is never asked" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nSettled by evidence: a large export runs as a background job, because one month is about 40,000 rows and in-request downloads time out after 30 seconds (your notes).\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D settled in a preface line with live alternatives left, and never asked" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "a question on who may export that mentions the finished file is not D" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-question.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-question.json deleted file mode 100644 index 1449335f937..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-question.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the Later line names the rest without asking them" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the recap line is not a question" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "a whole round" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-recommendation.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-recommendation.json deleted file mode 100644 index 822b3d3d89e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-recommendation.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "pass": [ - {"answer": "Q4: **Which role may export?**\nContext: support cannot see payment details (user, round 2).\nMy recommendation: `finance` only.\nBasis: verified, user, round 2\nAlternatives:\n- (a) Both roles\n\nIs there something else that would change this recommendation?", "why": "one recommendation"}, - {"answer": "Q4: Which date range does one export cover?\nWithheld: whether finance ever reconciles part-months, which only finance can confirm.\n\nIs there something else that would change this?", "why": "one withheld question"} - ], - "fail": [ - {"answer": "", "why": "empty"}, - {"answer": "Q4: Which role? My recommendation: finance.\n\nQ5: Which range? My recommendation: one month.", "why": "two questions"} - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-a.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-a.json deleted file mode 100644 index 37f9b860713..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-a.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one full-layout question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Exports need a scope.\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-b.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-b.json deleted file mode 100644 index 5ee8637317c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-b.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts need a format.\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-c.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-c.json deleted file mode 100644 index 955c4a7135b..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Someone has to be allowed to export.\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-d.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-d.json deleted file mode 100644 index 0290138ab91..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- Building the file takes a while for big months.\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "only the deferral question, D itself not asked" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/closing-probe.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/closing-probe.md deleted file mode 100644 index 1fd5f0c18bf..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/closing-probe.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: 'something else that would change' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-a.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-a.md deleted file mode 100644 index 765e72238ec..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-b.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-b.md deleted file mode 100644 index 7925aa6602e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-c.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-c.md deleted file mode 100644 index c5213709dcb..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-d.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-d.md deleted file mode 100644 index f76266389ee..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-question.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-question.md deleted file mode 100644 index fef3f505ea7..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-question.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-recommendation.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-recommendation.md deleted file mode 100644 index 638d3b48888..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-recommendation.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'My recommendation:|Withheld:' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-a.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-a.md deleted file mode 100644 index 598ed376d33..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-b.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-b.md deleted file mode 100644 index fc9c858a09c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-c.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-c.md deleted file mode 100644 index 92b82adfb7d..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-d.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-d.md deleted file mode 100644 index 0ad035250ac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/prompt.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/prompt.md deleted file mode 100644 index 2bd642d2b51..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/prompt.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -description: "Single arm, third message, of the round-versus-single pacing comparison (same task and facts as interview-pacing-round-arm). Source: the consumer's own writing, synthetic. The two earlier single-pacing answers are stated in the prompt, because a case is one user turn and a resumed history carries no skill body. Why hard: the user deferred the large-export mechanism to planning, so the skill must not re-ask it or treat it as open, and must ask one of the still-open decisions (amount format and currencies, or who may export) in the full layout without previewing the rest. Graders are regexes; the deciding rule is in ../interview-pacing.md." -tags: [interview, pacing, pacing-comparison, hard, no-network] -runs: 6 -max_turns: 30 -timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply asks exactly one question, preferably on amount format and currencies or on who may export, in the full layout with the closing probe, does not re-ask the date range, the large-export approach or its mechanism, then stops" ---- - -/planning:interview me Add a CSV export of invoices to the billing admin page. - -Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. - -What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. - -We're doing round 2 one question at a time, and two are answered. Q4, how a large export completes: build the file in the background and offer it for download when it's ready, as you recommended; the job runner, file storage and ready notice wait for planning. Q5, what one export covers: one calendar month across all customer accounts, as you recommended. - -Ask me the next question, still one at a time, then stop for my answer. diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/closing-probe.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/closing-probe.json deleted file mode 100644 index e04406c4c79..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/closing-probe.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nReply when ready.", - "why": "no closing probe" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-a.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-a.json deleted file mode 100644 index 9a48f709d64..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-a.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing, one question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "date range appears only in a recap line, not as a question" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Which date puts an invoice into a calendar month's export?**\nContext:\n- Finance closes its books on the calendar month, and an invoice has issue, due and paid dates (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "which date places an invoice in a month is a sub-decision, not the date range one export covers" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-b.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-b.json deleted file mode 100644 index 368a0108dac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-b.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "amount and currency format appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-c.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-c.json deleted file mode 100644 index d316c5e9396..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing after A and D were answered, one question on who may export; the recap line asks nothing" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "who may export, phrased as who may start and download the file" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "who may export appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-d.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-d.json deleted file mode 100644 index cdb7c3a2087..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be prepared after the click and downloaded once it's ready, instead of being built during the click like the page's other downloads?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D phrased as preparing the file after the click and downloading it once it's ready" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D appears only inside the question that defers the mechanism to planning; D itself is never asked" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nSettled by evidence: a large export runs as a background job, because one month is about 40,000 rows and in-request downloads time out after 30 seconds (your notes).\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D settled in a preface line with live alternatives left, and never asked" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "a question on who may export that mentions the finished file is not D" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-question.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-question.json deleted file mode 100644 index 1449335f937..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-question.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the Later line names the rest without asking them" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the recap line is not a question" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "a whole round" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-recommendation.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-recommendation.json deleted file mode 100644 index 822b3d3d89e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-recommendation.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "pass": [ - {"answer": "Q4: **Which role may export?**\nContext: support cannot see payment details (user, round 2).\nMy recommendation: `finance` only.\nBasis: verified, user, round 2\nAlternatives:\n- (a) Both roles\n\nIs there something else that would change this recommendation?", "why": "one recommendation"}, - {"answer": "Q4: Which date range does one export cover?\nWithheld: whether finance ever reconciles part-months, which only finance can confirm.\n\nIs there something else that would change this?", "why": "one withheld question"} - ], - "fail": [ - {"answer": "", "why": "empty"}, - {"answer": "Q4: Which role? My recommendation: finance.\n\nQ5: Which range? My recommendation: one month.", "why": "two questions"} - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-a.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-a.json deleted file mode 100644 index 37f9b860713..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-a.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one full-layout question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Exports need a scope.\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-b.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-b.json deleted file mode 100644 index 5ee8637317c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-b.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts need a format.\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-c.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-c.json deleted file mode 100644 index 955c4a7135b..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Someone has to be allowed to export.\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-d.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-d.json deleted file mode 100644 index 0290138ab91..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- Building the file takes a while for big months.\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "only the deferral question, D itself not asked" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/closing-probe.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/closing-probe.md deleted file mode 100644 index 1fd5f0c18bf..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/closing-probe.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: 'something else that would change' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-a.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-a.md deleted file mode 100644 index 765e72238ec..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-b.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-b.md deleted file mode 100644 index 7925aa6602e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-c.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-c.md deleted file mode 100644 index c5213709dcb..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-d.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-d.md deleted file mode 100644 index f76266389ee..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-question.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-question.md deleted file mode 100644 index fef3f505ea7..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-question.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-recommendation.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-recommendation.md deleted file mode 100644 index 638d3b48888..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-recommendation.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'My recommendation:|Withheld:' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-a.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-a.md deleted file mode 100644 index 598ed376d33..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-b.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-b.md deleted file mode 100644 index fc9c858a09c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-c.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-c.md deleted file mode 100644 index 92b82adfb7d..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-d.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-d.md deleted file mode 100644 index 0ad035250ac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/prompt.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/prompt.md deleted file mode 100644 index 5479bc6da5b..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/prompt.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -description: "Single arm, fourth message, of the round-versus-single pacing comparison (same task and facts as interview-pacing-round-arm). Source: the consumer's own writing, synthetic. The three earlier single-pacing answers are stated in the prompt, because a case is one user turn and a resumed history carries no skill body. Why hard: three planted decisions are settled, one with its mechanism deferred to planning, so the skill must ask the one still open (who may export) or another ready question, in the full layout, without re-asking a settled or deferred one. Graders are regexes; the deciding rule is in ../interview-pacing.md." -tags: [interview, pacing, pacing-comparison, hard, no-network] -runs: 6 -max_turns: 30 -timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply asks exactly one question, preferably on who may export, in the full layout with the closing probe, does not re-ask the date range, the amount format, the large-export approach or its mechanism, then stops" ---- - -/planning:interview me Add a CSV export of invoices to the billing admin page. - -Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. - -What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. - -We're doing round 2 one question at a time, and three are answered. Q4, how a large export completes: build the file in the background and offer it for download when it's ready, as you recommended; the job runner, file storage and ready notice wait for planning. Q5, what one export covers: one calendar month across all customer accounts, as you recommended. Q6, amounts: two-decimal numbers in the main currency unit with a separate currency-code column, no conversion and no total across currencies. - -Ask me the next question, still one at a time, then stop for my answer. diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/closing-probe.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/closing-probe.json deleted file mode 100644 index e04406c4c79..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/closing-probe.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nReply when ready.", - "why": "no closing probe" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-a.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-a.json deleted file mode 100644 index 9a48f709d64..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-a.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing, one question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "date range appears only in a recap line, not as a question" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Which date puts an invoice into a calendar month's export?**\nContext:\n- Finance closes its books on the calendar month, and an invoice has issue, due and paid dates (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "which date places an invoice in a month is a sub-decision, not the date range one export covers" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-b.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-b.json deleted file mode 100644 index 368a0108dac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-b.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "amount and currency format appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-c.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-c.json deleted file mode 100644 index d316c5e9396..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing after A and D were answered, one question on who may export; the recap line asks nothing" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "who may export, phrased as who may start and download the file" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "who may export appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-d.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-d.json deleted file mode 100644 index cdb7c3a2087..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be prepared after the click and downloaded once it's ready, instead of being built during the click like the page's other downloads?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D phrased as preparing the file after the click and downloading it once it's ready" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D appears only inside the question that defers the mechanism to planning; D itself is never asked" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nSettled by evidence: a large export runs as a background job, because one month is about 40,000 rows and in-request downloads time out after 30 seconds (your notes).\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D settled in a preface line with live alternatives left, and never asked" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "a question on who may export that mentions the finished file is not D" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-question.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-question.json deleted file mode 100644 index 1449335f937..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-question.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the Later line names the rest without asking them" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the recap line is not a question" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "a whole round" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-recommendation.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-recommendation.json deleted file mode 100644 index 822b3d3d89e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-recommendation.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "pass": [ - {"answer": "Q4: **Which role may export?**\nContext: support cannot see payment details (user, round 2).\nMy recommendation: `finance` only.\nBasis: verified, user, round 2\nAlternatives:\n- (a) Both roles\n\nIs there something else that would change this recommendation?", "why": "one recommendation"}, - {"answer": "Q4: Which date range does one export cover?\nWithheld: whether finance ever reconciles part-months, which only finance can confirm.\n\nIs there something else that would change this?", "why": "one withheld question"} - ], - "fail": [ - {"answer": "", "why": "empty"}, - {"answer": "Q4: Which role? My recommendation: finance.\n\nQ5: Which range? My recommendation: one month.", "why": "two questions"} - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-a.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-a.json deleted file mode 100644 index 37f9b860713..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-a.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one full-layout question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Exports need a scope.\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-b.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-b.json deleted file mode 100644 index 5ee8637317c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-b.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts need a format.\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-c.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-c.json deleted file mode 100644 index 955c4a7135b..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Someone has to be allowed to export.\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-d.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-d.json deleted file mode 100644 index 0290138ab91..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- Building the file takes a while for big months.\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "only the deferral question, D itself not asked" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/closing-probe.md b/plugins/planning/evals/interview-pacing-single-arm/graders/closing-probe.md deleted file mode 100644 index 1fd5f0c18bf..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/closing-probe.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: 'something else that would change' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-a.md b/plugins/planning/evals/interview-pacing-single-arm/graders/covers-a.md deleted file mode 100644 index 765e72238ec..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-b.md b/plugins/planning/evals/interview-pacing-single-arm/graders/covers-b.md deleted file mode 100644 index 7925aa6602e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-c.md b/plugins/planning/evals/interview-pacing-single-arm/graders/covers-c.md deleted file mode 100644 index c5213709dcb..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-d.md b/plugins/planning/evals/interview-pacing-single-arm/graders/covers-d.md deleted file mode 100644 index f76266389ee..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/one-question.md b/plugins/planning/evals/interview-pacing-single-arm/graders/one-question.md deleted file mode 100644 index fef3f505ea7..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/one-question.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/one-recommendation.md b/plugins/planning/evals/interview-pacing-single-arm/graders/one-recommendation.md deleted file mode 100644 index 638d3b48888..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/one-recommendation.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'My recommendation:|Withheld:' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-a.md b/plugins/planning/evals/interview-pacing-single-arm/graders/quality-a.md deleted file mode 100644 index 598ed376d33..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-b.md b/plugins/planning/evals/interview-pacing-single-arm/graders/quality-b.md deleted file mode 100644 index fc9c858a09c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-c.md b/plugins/planning/evals/interview-pacing-single-arm/graders/quality-c.md deleted file mode 100644 index 92b82adfb7d..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-d.md b/plugins/planning/evals/interview-pacing-single-arm/graders/quality-d.md deleted file mode 100644 index 0ad035250ac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/prompt.md b/plugins/planning/evals/interview-pacing-single-arm/prompt.md deleted file mode 100644 index 337403c6756..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/prompt.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -description: "Single arm, first message, of the round-versus-single pacing comparison (paired with interview-pacing-round-arm, same task and facts; only the pacing request differs). Source: the consumer's own writing, synthetic. An eval run cannot set the plugin's question_pacing option, so the case uses the in-session request, which the skill says overrides it. Why hard: the four open decisions are not listed, so the skill must find them, then ask only one, in the full layout, without previewing the rest. Graders are regexes; the deciding rule is in ../interview-pacing.md." -tags: [interview, pacing, pacing-comparison, hard, no-network] -runs: 6 -max_turns: 30 -timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply asks exactly one question, on one of date range, amount format and currencies, who may export, or large exports, in the full layout with the closing probe, lists no other question, then stops" ---- - -/planning:interview me Add a CSV export of invoices to the billing admin page. - -Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. - -What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. - -From now on, ask me one question at a time. Go ahead, then stop for my answer. diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/closing-probe.json b/plugins/planning/evals/interview-pacing-single-arm/samples/closing-probe.json deleted file mode 100644 index e04406c4c79..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/closing-probe.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nReply when ready.", - "why": "no closing probe" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-a.json b/plugins/planning/evals/interview-pacing-single-arm/samples/covers-a.json deleted file mode 100644 index 9a48f709d64..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-a.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing, one question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "date range appears only in a recap line, not as a question" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Which date puts an invoice into a calendar month's export?**\nContext:\n- Finance closes its books on the calendar month, and an invoice has issue, due and paid dates (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "which date places an invoice in a month is a sub-decision, not the date range one export covers" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-b.json b/plugins/planning/evals/interview-pacing-single-arm/samples/covers-b.json deleted file mode 100644 index 368a0108dac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-b.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "amount and currency format appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-c.json b/plugins/planning/evals/interview-pacing-single-arm/samples/covers-c.json deleted file mode 100644 index d316c5e9396..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing after A and D were answered, one question on who may export; the recap line asks nothing" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "who may export, phrased as who may start and download the file" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "who may export appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-d.json b/plugins/planning/evals/interview-pacing-single-arm/samples/covers-d.json deleted file mode 100644 index cdb7c3a2087..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be prepared after the click and downloaded once it's ready, instead of being built during the click like the page's other downloads?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D phrased as preparing the file after the click and downloading it once it's ready" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D appears only inside the question that defers the mechanism to planning; D itself is never asked" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nSettled by evidence: a large export runs as a background job, because one month is about 40,000 rows and in-request downloads time out after 30 seconds (your notes).\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D settled in a preface line with live alternatives left, and never asked" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "a question on who may export that mentions the finished file is not D" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/one-question.json b/plugins/planning/evals/interview-pacing-single-arm/samples/one-question.json deleted file mode 100644 index 1449335f937..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/one-question.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the Later line names the rest without asking them" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the recap line is not a question" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "a whole round" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/one-recommendation.json b/plugins/planning/evals/interview-pacing-single-arm/samples/one-recommendation.json deleted file mode 100644 index 822b3d3d89e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/one-recommendation.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "pass": [ - {"answer": "Q4: **Which role may export?**\nContext: support cannot see payment details (user, round 2).\nMy recommendation: `finance` only.\nBasis: verified, user, round 2\nAlternatives:\n- (a) Both roles\n\nIs there something else that would change this recommendation?", "why": "one recommendation"}, - {"answer": "Q4: Which date range does one export cover?\nWithheld: whether finance ever reconciles part-months, which only finance can confirm.\n\nIs there something else that would change this?", "why": "one withheld question"} - ], - "fail": [ - {"answer": "", "why": "empty"}, - {"answer": "Q4: Which role? My recommendation: finance.\n\nQ5: Which range? My recommendation: one month.", "why": "two questions"} - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-a.json b/plugins/planning/evals/interview-pacing-single-arm/samples/quality-a.json deleted file mode 100644 index 37f9b860713..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-a.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one full-layout question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Exports need a scope.\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-b.json b/plugins/planning/evals/interview-pacing-single-arm/samples/quality-b.json deleted file mode 100644 index 5ee8637317c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-b.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts need a format.\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-c.json b/plugins/planning/evals/interview-pacing-single-arm/samples/quality-c.json deleted file mode 100644 index 955c4a7135b..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Someone has to be allowed to export.\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-d.json b/plugins/planning/evals/interview-pacing-single-arm/samples/quality-d.json deleted file mode 100644 index 0290138ab91..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- Building the file takes a while for big months.\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "only the deferral question, D itself not asked" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing.md b/plugins/planning/evals/interview-pacing.md index 560abbe86e2..2c0aa7e2237 100644 --- a/plugins/planning/evals/interview-pacing.md +++ b/plugins/planning/evals/interview-pacing.md @@ -199,3 +199,13 @@ pacings on this task; what differs is turns, about 12 single-pacing messages for Grader miss, reported and not re-graded (graders are frozen once a run starts): round run 3 asks D as "Can the export be asynchronous instead of an immediate download?", which `covers-d` does not match, so it counts as no D. Read by hand, Cr is 6/6 and the verdict is the same. + +## Amendment 3 (2026-11-21, cases retired) + +The six `interview-pacing-*` cases are removed from the suite. The experiment concluded with run 3 +(ROUNDS HOLD UP) and measured a round layout that has since changed: rounds now print at a +`question_detail` level, `standard` by default, which shows no `Basis:` line. This file freezes the +graders once a run starts, so the cases are retired, not edited. Question quality at the new layout +is checked by `interview-round-quality-standard`, which reuses the round-arm prompt and is not part +of this experiment. To rerun the experiment, restore the six case folders from the commit before +their removal. diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/checkable-basis-and-flip.md b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/checkable-basis-and-flip.md index f2ffc632b85..77c74e83ef3 100644 --- a/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/checkable-basis-and-flip.md +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/checkable-basis-and-flip.md @@ -5,7 +5,8 @@ arm: both Look at every question in the reply that carries a recommended answer. -PASS only if every such recommendation has: +PASS only if every such recommendation has the following, stated in its printed reason or in a +`Basis:` line under it: 1. A checkable basis: a file path with a line, a URL, tool output or a check the reply says it ran this session (a command, a search, a survey of the working directory) named with what it showed diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/scope-check-line.md b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/prompt.md b/plugins/planning/evals/interview-recommendation-basis-and-flip/prompt.md index 225b0eb9d73..904de23c8c7 100644 --- a/plugins/planning/evals/interview-recommendation-basis-and-flip/prompt.md +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/prompt.md @@ -1,11 +1,15 @@ --- -description: "Eval-first case for checkable recommendations. Why hard: the prompt states enough facts that every recommendation can rest on something the user said, but 'Basis: judgment' with no reference and no condition that would change it is the easy default." +description: "Eval-first case for checkable recommendations. Why hard: the prompt states enough facts that every recommendation can rest on something the user said, but 'Basis: judgment' with no reference and no condition that would change it is the easy default. The prompt starts past the intake round, so the first decision round carries the recommendations and its Scope check: header line." tags: [interview, recommendation, basis] runs: 3 -max_turns: 12 -timeout_seconds: 600 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "Every recommendation in round 1 names a checkable basis (a file and line, a URL, a check run this session named with what it showed, a session-transcript finding the user said or a tool returned (never the reply's own earlier claim), or a fact the user stated), and names the condition that would change it only where one exists" +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "Every recommendation in round 2 names a checkable basis in its printed reason (a file and line, a URL, a check run this session named with what it showed, a session-transcript finding the user said or a tool returned (never the reply's own earlier claim), or a fact the user stated), and names the condition that would change it only where one exists" --- -/planning:interview We need rate limiting on our public REST API. Facts: we run on Postgres 16 with a single Redis 7 instance already used for sessions; traffic peaks at about 2,000 requests per second; our published partner terms promise 1,000 requests per minute per API key. Ask me your first round. +/planning:interview We need rate limiting on our public REST API. Facts: we run on Postgres 16 with a single Redis 7 instance already used for sessions; traffic peaks at about 2,000 requests per second; our published partner terms promise 1,000 requests per minute per API key. + +Round 1 is answered: Q1, the problem is that a few partner integrations send bursts that slow the API for everyone; Q2, the users are partner developers calling the API with their keys, and our on-call engineers. + +Ask me round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/scope-check-line.json b/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-round-topic-groups/graders/scope-check-line.md b/plugins/planning/evals/interview-round-topic-groups/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-round-topic-groups/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-round-topic-groups/graders/topic-groups.md b/plugins/planning/evals/interview-round-topic-groups/graders/topic-groups.md index 8babde1e561..b830887abf7 100644 --- a/plugins/planning/evals/interview-round-topic-groups/graders/topic-groups.md +++ b/plugins/planning/evals/interview-round-topic-groups/graders/topic-groups.md @@ -11,7 +11,8 @@ PASS only if all of these hold: (who is alerted, what an alert message includes). Each question appears under exactly one header. 2. Question ids run on from Q4 with no gap and no restart across the headers. -3. Each question keeps its own block with a recommendation, a `Basis:` line and its alternatives. +3. Each question keeps its own block with a recommendation whose reason names its basis, and its + alternatives. The headers do not replace any of that. 4. The round ends with the closing probe, the invitation to name something that would change the recommendations, after the last question. No table summarizes the round's questions. diff --git a/plugins/planning/evals/interview-round-topic-groups/prompt.md b/plugins/planning/evals/interview-round-topic-groups/prompt.md index 211cbda7efc..0d49044a6d7 100644 --- a/plugins/planning/evals/interview-round-topic-groups/prompt.md +++ b/plugins/planning/evals/interview-round-topic-groups/prompt.md @@ -1,11 +1,11 @@ --- -description: "Round 2 of a webhook-sender interview whose frontier spans two topics, retry policy and failure alerting. Why hard: the round must keep every question in its full layout and add topic headers; the easy failures are one flat numbered list, or a closing summary table of the questions." +description: "Round 2 of a webhook-sender interview whose frontier spans two topics, retry policy and failure alerting. Why hard: the round must keep every question in its full layout and add topic headers; the easy failures are one flat numbered list, or a closing summary table of the questions. The round header carries a Scope check: line." tags: [interview, round-layout, topic-groups, no-network] runs: 3 max_turns: 30 timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "Round 2 asks the five decisions from Q4 on under at least two topic headers (retries and alerting), keeps each question in its full layout, and ends with the closing probe and no summary table" +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "Round 2 asks the five decisions from Q4 on under at least two topic headers (retries and alerting), keeps each question in its layout, carries a Scope check: line in the round header, and ends with the closing probe and no summary table" --- /planning:interview me Add retries and failure alerting to our outbound webhook sender. diff --git a/plugins/planning/evals/interview-round-topic-groups/samples/scope-check-line.json b/plugins/planning/evals/interview-round-topic-groups/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-round-topic-groups/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/case.yaml b/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/case.yaml index 78cd9c7887d..ef2adf03251 100644 --- a/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/case.yaml +++ b/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/case.yaml @@ -1,14 +1,14 @@ schema_version: "1.1" name: interview-scenario-walk-two-purchase-kinds -description: "One product sold two ways (a one-off bag and a monthly subscription of the same bag) leaves 'order' and 'refund' unsettled, and the user calls it confusing and asks for scenarios. Why hard: the easy reply defines the terms in prose or asks 'what is an order?', where the walk puts numbered, concrete scenarios inside one question, each with its own recommendation and the term or rule it forces." +description: "One product sold two ways (a one-off bag and a monthly subscription of the same bag) leaves 'order' and 'refund' unsettled, and the user calls it confusing and asks for scenarios. Why hard: the easy reply defines the terms in prose or asks 'what is an order?', where the walk puts numbered, concrete scenarios inside one question, each with its own recommendation and the term or rule it forces. The prompt starts past the intake round, so the walk comes in the first decision round, headed by a Scope check: line." tags: [interview, scenario-walk, domain-language, scaffold] -expected_outcome: "The round holds a numbered question whose Context carries S1, S2, ... scenarios, each concrete (a named person, an amount, a day or date), stating what it tests and ending with its own recommendation and the term or rule that answer forces, and the user is told to answer per scenario (for example 'S2: a')." +expected_outcome: "Round 2, headed by a Scope check: line, holds a numbered question whose Context carries S1, S2, ... scenarios, each concrete (a named person, an amount, a day or date), stating what it tests and ending with its own recommendation and the term or rule that answer forces, and the user is told to answer per scenario (for example 'S2: a')." context: scaffold_script: scaffold.sh execution: - prompt: "/planning:interview me We sell the same coffee bag two ways: a one-off purchase and a monthly subscription. I find it confusing what an 'order' is once a subscription renews, and what a refund applies to. Walk me through some scenarios to settle it. I'm here and will answer." - allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit] - max_turns: 20 + prompt: "/planning:interview me We sell the same coffee bag two ways: a one-off purchase and a monthly subscription. I find it confusing what an 'order' is once a subscription renews, and what a refund applies to. Walk me through some scenarios to settle it. Round 1 is answered: Q1, the problem is that support cannot tell customers which purchase a refund covers; Q2, the users are our customers and the two support staff who issue refunds. I'm here and will answer; ask me round 2." + allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] + max_turns: 30 timeout_seconds: 900 graders: - name: skill-fired @@ -20,6 +20,11 @@ graders: type: regex pattern: '\bS1\**\s*:[\s\S]*\bS2\**\s*:' arm: both + - name: scope-check-line + type: regex + pattern: '(?:^|\n)[^\n]{0,6}Scope check:' + target: last_message + arm: both - name: scenario-walk-shape type: llm arm: both diff --git a/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/samples/scope-check-line.json b/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-scope-cut-moscow/graders/scope-check-line.md b/plugins/planning/evals/interview-scope-cut-moscow/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-scope-cut-moscow/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-scope-cut-moscow/prompt.md b/plugins/planning/evals/interview-scope-cut-moscow/prompt.md index fe9e643dceb..a2ed9494d9c 100644 --- a/plugins/planning/evals/interview-scope-cut-moscow/prompt.md +++ b/plugins/planning/evals/interview-scope-cut-moscow/prompt.md @@ -1,11 +1,15 @@ --- -description: "Eval-first case for the scope-cut round. Why hard: eight capabilities for two engineers in one quarter is plainly too big, so recommending /planning:wayfind at once looks right, but nobody has yet asked which items the effort fails without." +description: "Eval-first case for the scope-cut round. Why hard: eight capabilities for two engineers in one quarter is plainly too big, so recommending /planning:wayfind at once looks right, but nobody has yet asked which items the effort fails without. The prompt starts past the intake round, so the scope cut comes with recommendations in the first decision round, headed by a Scope check: line." tags: [interview, scope, wayfind, no-network] runs: 3 max_turns: 30 timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] expected_outcome: "Before recommending /planning:wayfind, the reply runs a scope cut: for each capability it asks or applies the Must test (if this is not delivered, is the effort cancelled or worked around?), recommends a bucket per item, keeps Must to about 60% of the effort or less, and routes the rest to /planning:wayfind or later" --- /planning:interview Build our internal developer portal this quarter with two engineers: a service catalog, team scorecards, onboarding guides, cloud cost dashboards, on-call schedules, an incident timeline, an API explorer, and a plugin system so other teams can extend it. + +Round 1 is answered: Q1, the problem is that engineers spend their first weeks finding which team owns which service and how to reach it; Q2, the users are our 120 product engineers and their team leads. + +Ask round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-scope-cut-moscow/samples/scope-check-line.json b/plugins/planning/evals/interview-scope-cut-moscow/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-scope-cut-moscow/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-single-pacing-one-question/graders/no-basis-line.md b/plugins/planning/evals/interview-single-pacing-one-question/graders/no-basis-line.md new file mode 100644 index 00000000000..0f5488f6658 --- /dev/null +++ b/plugins/planning/evals/interview-single-pacing-one-question/graders/no-basis-line.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\S\n]{0,4}(?:[^\w\s]{1,2}[^\S\n]*)?(?:\*\*)?(?:Basis|Origin)(?:\*\*)?:' +match: not_contains +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-single-pacing-one-question/graders/one-question-full-layout.md b/plugins/planning/evals/interview-single-pacing-one-question/graders/one-question-full-layout.md index deac3eb8315..3511be4dc9e 100644 --- a/plugins/planning/evals/interview-single-pacing-one-question/graders/one-question-full-layout.md +++ b/plugins/planning/evals/interview-single-pacing-one-question/graders/one-question-full-layout.md @@ -9,8 +9,8 @@ PASS only if all of these hold: 1. The reply asks exactly one question, and it is one of the five frontier decisions (attempt count, delay, what happens after the last attempt, who is alerted, what an alert message includes). -2. That question is in the full layout: a context block, a recommendation with its `Basis:` line, - and its alternatives. +2. That question is in the question layout: a context block, a recommendation whose reason names + its basis, and its alternatives. 3. The reply ends with a closing probe inviting something that would change the recommendation, and then stops for the answer. 4. The reply has no topic header and no index table, and it does not list, preview or number the diff --git a/plugins/planning/evals/interview-single-pacing-one-question/prompt.md b/plugins/planning/evals/interview-single-pacing-one-question/prompt.md index 43d166a5c61..41b4929aa7a 100644 --- a/plugins/planning/evals/interview-single-pacing-one-question/prompt.md +++ b/plugins/planning/evals/interview-single-pacing-one-question/prompt.md @@ -1,11 +1,11 @@ --- -description: "Single pacing, asked for in the session: the user wants one question at a time while round 2's frontier holds five independent decisions. An eval run cannot set the plugin's question_pacing option, so the case uses the in-session request, which the skill says overrides the option for the session. Why hard: the frontier is computed whole, so the easy failure is to ask all five, or to ask one and add the rest as a preview, a topic header or an index." +description: "Single pacing, asked for in the session: the user wants one question at a time while round 2's frontier holds five independent decisions. An eval run cannot set the plugin's question_pacing option, so the case uses the in-session request, which the skill says overrides the option for the session. Why hard: the frontier is computed whole, so the easy failure is to ask all five, or to ask one and add the rest as a preview, a topic header or an index. At the default standard detail level the question shows its recommendation with its reason and no separate Origin: or Basis: line." tags: [interview, round-layout, pacing, no-network] runs: 3 -max_turns: 12 -timeout_seconds: 600 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply asks exactly one of the five frontier decisions in the full question layout with the closing probe, with no topic header, no index table and no other question listed, then stops" +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "The reply asks exactly one of the five frontier decisions in the standard question layout (no Origin: or Basis: line) with the closing probe, with no topic header, no index table and no other question listed, then stops" --- /planning:interview me Add retries and failure alerting to our outbound webhook sender. diff --git a/plugins/planning/evals/interview-single-pacing-one-question/samples/no-basis-line.json b/plugins/planning/evals/interview-single-pacing-one-question/samples/no-basis-line.json new file mode 100644 index 00000000000..23e63555edf --- /dev/null +++ b/plugins/planning/evals/interview-single-pacing-one-question/samples/no-basis-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Round 2.\nScope check: 9 of 12 questions registered\n\n❓ Q4: **How many delivery attempts before giving up?**\n\nContext:\n- Today the sender makes one POST and retries nothing (your notes).\n\n➡️ My recommendation: **Five attempts**. Receivers deduplicate on event_id, so retries are safe (verified, user, round 1).\n\nAlternatives:\n- (a) Three attempts.\n\nIs there something else that would change this recommendation?", + "why": "the standard print: the basis rides in the reason, no Basis: or Origin: line" + }, + { + "answer": "Q4: **How many attempts?**\nMy recommendation: **Five**. The basis is your round-1 answer.\n", + "why": "the word basis inside a sentence is not a label line" + } + ], + "fail": [ + { + "answer": "❓ Q4: **How many delivery attempts?**\n\nContext:\n- One POST today.\n\n➡️ My recommendation: **Five attempts**.\nOrigin: new (ours)\nBasis: verified, user, round 1\n", + "why": "the old full layout prints Origin: and Basis: lines" + }, + { + "answer": "Q4: **How many attempts?**\nMy recommendation: **Five**.\n**Basis:** verified, your notes\n", + "why": "a bold Basis: label line" + } + ] +} diff --git a/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/one-offer-line.md b/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/one-offer-line.md deleted file mode 100644 index f7fd4bfe95f..00000000000 --- a/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/one-offer-line.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '^\W*surface: \S[^\n]*\(default\)\. Also available: [^\n]+$' -flags: m -match: "count:1" -target: last_message -arm: with-only ---- diff --git a/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/opening-lines.md b/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/opening-lines.md new file mode 100644 index 00000000000..2ed2852f2ef --- /dev/null +++ b/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/opening-lines.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^\W*Asking here\b[^\n]*\.[^\S\n]+[^\n]*\balso available\b[^\n]*$' +flags: mi +match: "count:1" +target: last_message +arm: with-only +--- diff --git a/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/proceeds-to-round.md b/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/proceeds-to-round.md index c080dfba79a..574074b9dfa 100644 --- a/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/proceeds-to-round.md +++ b/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/proceeds-to-round.md @@ -6,13 +6,13 @@ focus: last_message PASS only if all of these hold: -1. One line names the default surface and the other available surfaces together (for example - `surface: terminal (default). Also available: page. Ask for one by name to switch.`), and no - other line repeats or expands that offer. +1. One line says where the interview asks and names the other available surfaces together (for + example `Asking here, one round at a time. A local page is also available; say which to + switch.`), and no other line repeats or expands that offer. 2. The reply does not ask which surface to use and does not wait for a surface choice: the offer is a statement, and the reply goes on to the interview. 3. The reply asks round 1 of the interview about the webhook sender's retries and alerting in the - conversation: at least one numbered question with a recommendation and its basis. + conversation: at least one numbered question, open or with a recommendation. FAIL if the offer is missing, is spread over several lines or a list, is phrased as a question the user must answer before the interview starts, or if the reply stops before asking round 1. diff --git a/plugins/planning/evals/interview-surface-unset-offer-one-line/prompt.md b/plugins/planning/evals/interview-surface-unset-offer-one-line/prompt.md index a283d51b732..7bb1d6c86bc 100644 --- a/plugins/planning/evals/interview-surface-unset-offer-one-line/prompt.md +++ b/plugins/planning/evals/interview-surface-unset-offer-one-line/prompt.md @@ -1,11 +1,11 @@ --- -description: "Surface unset, more than one surface available: an eval run cannot set the plugin's surface option, so the skill sees the literal user_config token, which the surface registry reads as unset. The host has bash, curl and Python 3, so the page is available beside the default. Why hard: the easy failures are to skip the resolver and ask inline with no offer, to turn the offer into a question and stop for a surface choice, or to restate the alternatives across several lines." +description: "Surface unset, more than one surface available: an eval run cannot set the plugin's surface option, so the skill sees the literal user_config token, which the surface registry reads as unset. The host has bash, curl and Python 3, so the page is available beside the default. Why hard: the easy failures are to skip the resolver and ask inline with no offer, to turn the offer into a question and stop for a surface choice, or to restate the alternatives across several lines. The opening prints one sentence per line in the round-8 shape: its first line says where the questions are asked and names the other available surfaces." tags: [interview, surface, registry, no-network] runs: 3 -max_turns: 14 -timeout_seconds: 600 -allowed_tools: [Read, Glob, Grep, Skill, Bash] -expected_outcome: "The run resolves the surface through surfaces/registry.mjs, prints its offer line once (the default and the alternatives on one line), and proceeds to round 1 without waiting for a surface choice" +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "The run resolves the surface through surfaces/registry.mjs, prints its offer once as the opening's first line (where it asks, and the other available surfaces), and proceeds to round 1 without waiting for a surface choice" --- /planning:interview me Add retries and failure alerting to our outbound webhook sender. Today it makes one POST per event and retries nothing. diff --git a/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/one-offer-line.json b/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/one-offer-line.json deleted file mode 100644 index 8556c59dbdd..00000000000 --- a/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/one-offer-line.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "surface: terminal (default). Also available: page. Ask for one by name to switch.\n\nRound 1.\n\nQ1: **How many delivery attempts before the sender gives up?**\nContext: the sender makes one POST per event today and retries nothing (user, this message).\nMy recommendation: five attempts.\nBasis: verified, user, this message (one POST, no retries)\n\nIs there something else that would change these recommendations?", - "why": "the offer is one line, then the round follows" - } - ], - "fail": [ - { - "answer": "Round 1.\n\nQ1: **How many delivery attempts before the sender gives up?**\nMy recommendation: five attempts.\nBasis: verified, user, this message", - "why": "no offer line" - }, - { - "answer": "surface: terminal (default). Also available: page. Ask for one by name to switch.\n\nRound 1 ...\n\nsurface: terminal (default). Also available: page. Ask for one by name to switch.", - "why": "the offer line is printed twice" - }, - { - "answer": "Surfaces:\n- terminal (default)\n- page\n\nRound 1 ...", - "why": "the offer is spread over a list, not one line" - } - ] -} diff --git a/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/opening-lines.json b/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/opening-lines.json new file mode 100644 index 00000000000..4655c3aba4a --- /dev/null +++ b/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/opening-lines.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Asking here, one round at a time. A local page is also available; say which to switch.\nQuestions are saved to .work/webhook-retries/interview-checklist.md.\n\n**Round 1.** Retries and alerting for the webhook sender.\n\n❓ Q1: **What goes wrong today when a delivery fails?**", + "why": "the round-8 opening: where it asks and the alternatives on the first line, then the ledger line" + } + ], + "fail": [ + { + "answer": "surface: terminal (default). Also available: page. Ask for one by name to switch.\n\n**Round 1.**\n\n❓ Q1: **What goes wrong today?**", + "why": "the old one-line offer shape" + }, + { + "answer": "Asking here, one round at a time.\nA local page is also available; say which to switch.\n\nQ1: **What goes wrong today?**", + "why": "the offer is split, so the alternatives are not named on the line that says where it asks" + }, + { + "answer": "Asking here, one round at a time. A local page is also available; say which to switch.\n\nQ1: **What goes wrong today?**\n\nAsking here, one round at a time. A local page is also available; say which to switch.", + "why": "the offer is repeated" + } + ] +} diff --git a/plugins/planning/evals/interview-term-clash-qualified-name/graders/scope-check-line.md b/plugins/planning/evals/interview-term-clash-qualified-name/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-term-clash-qualified-name/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-term-clash-qualified-name/prompt.md b/plugins/planning/evals/interview-term-clash-qualified-name/prompt.md index 2f3e0f32b57..25a3f6e21b7 100644 --- a/plugins/planning/evals/interview-term-clash-qualified-name/prompt.md +++ b/plugins/planning/evals/interview-term-clash-qualified-name/prompt.md @@ -1,13 +1,13 @@ --- -description: "The user says 'account' for a household that pays for several members, while the code and the glossary already use Account for one person's sign-in identity. Why hard: the easy failures are to adopt the bare word, or to point out the clash and ask the user what to call it instead of proposing a qualified name." +description: "The user says 'account' for a household that pays for several members, while the code and the glossary already use Account for one person's sign-in identity. Why hard: the easy failures are to adopt the bare word, or to point out the clash and ask the user what to call it instead of proposing a qualified name. The prompt starts past the intake round, so the qualified name is recommended in the first decision round, headed by a Scope check: line." tags: [interview, glossary, domain, scaffold] runs: 3 -max_turns: 16 -timeout_seconds: 600 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply names the clash with the existing Account (src/auth/account.ts or docs/glossary.md) and recommends a qualified name it proposes itself, such as household account or billing account, as a question the user can accept; it does not ask the user to come up with the name" +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "The reply names the clash with the existing Account (src/auth/account.ts or docs/glossary.md) and recommends a qualified name it proposes itself, such as household account or billing account, as a question the user can accept, in a round headed by a Scope check: line; it does not ask the user to come up with the name" --- /planning:interview me: let one account pay for several gym members, so a family gets one monthly invoice. I'm here and will answer. -[Session state: an engineering session in this repository; domain_docs is not set anywhere, so its default applies. Nothing has been asked yet. Write your first message to the user.] +[Session state: an engineering session in this repository; domain_docs is not set anywhere, so its default applies. Round 1 is answered: Q1, the problem is that a family with three members gets three monthly invoices; Q2, the users are paying members and the front-desk staff who handle billing questions. Write round 2 to the user.] diff --git a/plugins/planning/evals/interview-term-clash-qualified-name/samples/scope-check-line.json b/plugins/planning/evals/interview-term-clash-qualified-name/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-term-clash-qualified-name/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/skills/interview/evals/evals.json b/plugins/planning/skills/interview/evals/evals.json index 0cfe4222a8a..5193d662259 100644 --- a/plugins/planning/skills/interview/evals/evals.json +++ b/plugins/planning/skills/interview/evals/evals.json @@ -5,11 +5,11 @@ "id": 1, "name": "relentless-me-mode-frontier-rounds", "prompt": "/planning:interview me — I want to add rate limiting to the public API but haven't thought through the details.", - "expected_output": "In me-mode the skill runs a relentless frontier-rounds interview: each round asks every question whose prerequisites are settled as one numbered set in inline prose (not AskUserQuestion), each question leading with a recommendation and its basis, and grounds answers in the codebase where the code already answers a question.", + "expected_output": "In me-mode the skill runs a relentless frontier-rounds interview: each round asks every question whose prerequisites are settled as one numbered set in inline prose (not AskUserQuestion), each decision question leading with a recommendation and its basis (a discovery question in an intake round is asked open with a `Discovery:` line), and grounds answers in the codebase where the code already answers a question.", "files": [], "expectations": [ "Output asks the round's independent frontier questions together as one numbered set, not spread across separate turns", - "Each question leads with a recommended answer and a one-line basis", + "Each decision question leads with a recommended answer and its basis; a discovery question in an intake round carries a `Discovery:` line and no recommendation", "The round is asked inline in prose, not via a side-by-side AskUserQuestion card", "Where a question is answerable from the codebase, the skill resolves it by inspection instead of spending a question on it", "A reply that resolves its question but arrives as free text rather than an authored option is recorded with the free-text: resolution-field flag so downstream passes scrutinize it; a partial or non-resolving reply keeps its row open" @@ -65,7 +65,7 @@ "Output asks the prerequisite decision first and defers each dependent decision to a later round, after its prerequisite is answered", "Output does NOT batch a question into the same round as a question it depends on", "Output does NOT use an AskUserQuestion card absent the use_ask_user_question opt-in", - "Each question still leads with a recommendation and basis" + "Each decision question still leads with a recommendation and its basis; a discovery question in an intake round carries none" ] }, { @@ -84,10 +84,10 @@ "id": 7, "name": "engineering-term-resolution-delegates", "prompt": "/planning:interview me — in this engineering task we resolved that Subscriber is canonical and Member is rejected. Continue the interview after preserving that decision.", - "expected_output": "Invokes /domain-driven-design:ubiquitous-language curate immediately for the resolved engineering vocabulary, then resumes the interview at the current round. It does not write a parallel glossary format itself, and a general-domain session would not mutate repo docs.", + "expected_output": "Invokes /domain-driven-design:ubiquitous-language curate for the resolved engineering vocabulary once the user has confirmed it (the prompt's statement of the resolution is that confirmation), then resumes the interview at the current round. It does not write a parallel glossary format itself, and a general-domain session would not mutate repo docs.", "files": [], "expectations": [ - "Invokes /domain-driven-design:ubiquitous-language curate immediately for the resolved term", + "Invokes /domain-driven-design:ubiquitous-language curate for the resolved term only after the user's confirmation, which the prompt gives", "Does not prescribe or reimplement glossary format/location in the interview skill", "Resumes the interview loop after the vocabulary update", "Keeps the repository write explicitly scoped to an engineering session" @@ -101,7 +101,7 @@ "files": [], "expectations": [ "The current config format is resolved by inspecting the codebase and stated as a finding, not asked as a question", - "The override-precedence choice is asked as a question with a recommendation, not silently decided", + "The override-precedence choice is asked as a decision question with a recommendation (in the first decision round, after any intake questions), not silently decided", "Output does not ask the user to confirm a fact the code already answers" ] }, @@ -280,7 +280,7 @@ "id": 20, "name": "mechanism-tripwire-asks-now-or-at-planning", "prompt": "/planning:interview me I want nightly exports of course progress to the analytics team. My guess is a GitHub Actions cron job that pushes a CSV to their S3 bucket with a scoped token.", - "expected_output": "The first round that names a mechanism (a CI runner, a storage target, a token scope) also carries one registered question: settle mechanism-level decisions now, or at planning. Until the user answers it, no recommendation fixes a mechanism. When the user answers \"at planning\", every still-open mechanism row, commitment rows included, becomes `deferred` with an arbiter tag in the Brief's `### Deferred questions`. When the user answers \"now\", the choice is written at once as a `confirmed` constraint and the Brief's Constraints records that this topic overrides \"does not plan implementation\".", + "expected_output": "The first decision round that names a mechanism (a CI runner, a storage target, a token scope) also carries one registered question: settle mechanism-level decisions now, or at planning. Until the user answers it, no recommendation fixes a mechanism. When the user answers \"at planning\", every still-open mechanism row, commitment rows included, becomes `deferred` with an arbiter tag in the Brief's `### Deferred questions`. When the user answers \"now\", the choice is written at once as a `confirmed` constraint and the Brief's Constraints records that this topic overrides \"does not plan implementation\".", "files": [], "expectations": [ "The now-or-at-planning question is asked in the same round as the first mechanism-naming question and has its own `open` register row at ask-time", @@ -330,7 +330,7 @@ "The inherited constraint is asked as a register row at ask-time rather than treated as binding", "The research output's library is presented as a candidate re-derived against the constraint ledger, not as the recommendation", "The conflict between the candidate and the inherited constraint is surfaced to the user", - "Every recommendation carries a `Checked against:` line naming each constraint in words, not by bare id, or `none`" + "Every recommendation's saved full block (the round file) carries a `Checked against:` line naming each constraint in words, not by bare id, or `none`" ] }, { @@ -356,7 +356,7 @@ "The queued return is evaluated before the same-turn reply: Q3 is NOT recorded `answered` with in-process memory, and the response does not treat the user's \"go with your recommendation\" as settling Q3", "The restated Q3 names in-process memory as superseded and gives the replacement recommendation its own basis drawn from the three-replica manifest", "The user's Q3 answer is revalidated against the replacement recommendation, for example by asking whether the recommendation they accepted still stands now that it has changed", - "The response ends with one block restating every open question in full, each with its Context, recommendation with Basis and alternatives, and Q3 is marked as the changed one; Q2 and Q4 are not reduced to a one-line pointer", + "The response ends with one block restating every open question in full, each with its Context, its recommendation with its reason (its basis) and its alternatives, and Q3 is marked as the changed one; Q2 and Q4 are not reduced to a one-line pointer", "Q1 is recorded `answered` (per API key) from the reply", "Asking Q2 and Q4 was not held for the lookup, and Q5, which was downstream of it, is asked once the return lands" ] @@ -387,7 +387,7 @@ "expectations": [ "The open recommendations are checked against the constraint ledger before any round-3 question is shown", "Q7's recommendation, which rests on the archived design C2 excludes, is not shown as written: it is revised with old, new and why, or marked \"not yet researched\"", - "A revised Q7 names the constraint C2 records, in words, on its `Checked against:` line", + "A revised Q7 names the constraint C2 records, in words, on its `Checked against:` line in the saved full round block", "The run says that the re-check changed Q7", "Q8 is shown with its recommendation unchanged", "Neither Q7 nor Q8 is resolved by the re-check; both stay open until the user answers" @@ -397,11 +397,11 @@ "id": 28, "name": "self-answer-from-mcp-and-research-and-offer-to-connect", "prompt": "/planning:interview me add a nightly digest of open incidents to our on-call channel\n\n[Session state: a Linear MCP server is connected, and its read tools return the team's incident project, whose issues carry a `severity` field with the values SEV1 to SEV4. No chat MCP server or chat CLI is connected. The repository has no chat integration and no ADR about one. Earlier in the conversation the user said: \"we use Microsoft Teams for on-call.\" The candidate questions drafted for round 1 are: which severities the incident tracker uses; the maximum message size a Teams incoming-webhook post accepts; which Teams channel is the on-call channel and whether it already receives other bot posts; and whether the digest lists every open incident or only SEV1 and SEV2.]", - "expected_output": "Before round 1 is shown, the run tries to answer each candidate itself. The severity question is answered from the connected Linear MCP server and stated as a fact tagged answered-from-mcp:linear with a Basis line citing the tool output; it gets no register row. The Teams webhook message-size limit is answered by research against Microsoft's official documentation and stated as a fact tagged answered-from-research with the URL as its basis; it gets no register row. The on-call channel question sits in Teams, which is not connected, so the round carries one offer to connect Teams that names Microsoft Teams, the question it would settle, and how to connect it (an MCP server or connector, or the vendor integration), and lets the user connect it or answer by hand; that question is registered open as awaiting access. The digest-scope question is a decision with real tradeoffs and is asked with a recommendation, never resolved by the agent.", + "expected_output": "Before round 1 is shown, the run tries to answer each candidate itself. The severity question is answered from the connected Linear MCP server and stated as a fact tagged answered-from-mcp:linear with its source, the tool output; it gets no register row. The Teams webhook message-size limit is answered by research against Microsoft's official documentation and stated as a fact tagged answered-from-research with the URL as its basis; it gets no register row. The on-call channel question sits in Teams, which is not connected, so the round carries one offer to connect Teams that names Microsoft Teams, the question it would settle, and how to connect it (an MCP server or connector, or the vendor integration), and lets the user connect it or answer by hand; that question is registered open as awaiting access. The digest-scope question is a decision with real tradeoffs and is asked with a recommendation, never resolved by the agent.", "files": [], "narration": true, "expectations": [ - "The severity values are stated as a fact tagged answered-from-mcp:linear with a Basis line, and no register row is opened for that question", + "The severity values are stated as a fact tagged answered-from-mcp:linear with its source cited, and no register row is opened for that question", "The Teams webhook message-size limit is stated as a fact tagged answered-from-research with an official Microsoft documentation URL as its Basis, and no register row is opened for it", "The round makes exactly one offer to connect Microsoft Teams that names the system, the question it would settle, and how to connect it", "The offer lets the user connect Teams or answer by hand, and the on-call channel question is registered open, not answered or assumed", @@ -596,11 +596,11 @@ "id": 43, "name": "late-return-restates-every-open-question-in-full", "prompt": "/planning:interview me Add a nightly export of order data to the analytics warehouse.\n\n[Session state: terminal lane. Round 1's Q1 to Q3 are `answered`. Round 2 was shown as Q4 (export format; recommended: Parquet, because the warehouse loader reads Parquet), Q5 (how far back the first export reaches; recommended: 90 days, from the retention setting the survey read), Q6 (whether refunds are exported as negative rows or a separate table; recommended: negative rows, from the existing finance report's shape) and Q7 (the export window; recommended: 02:00 UTC, because the survey found no other nightly job). A sub-agent was dispatched to list the scheduled jobs on the production cluster; no question waits on it. Before the user replies, its return lands: a database backup job runs on that cluster from 01:30 to 03:00 UTC every night.]", - "expected_output": "The return contradicts Q7's recommendation, so Q7 is restated with 02:00 UTC named as superseded and a replacement window outside the 01:30 to 03:00 UTC backup, with its basis in the sub-agent's job list. The message ends with one block restating every open question in full: Q4, Q5, Q6 and Q7, each with its Context, its recommendation with `Basis:` and its alternatives, and Q7 is marked as the changed one. Q4, Q5 and Q6 are not reduced to a one-line pointer such as \"still open, unchanged\", and no question is recorded as answered, since the user has not replied.", + "expected_output": "The return contradicts Q7's recommendation, so Q7 is restated with 02:00 UTC named as superseded and a replacement window outside the 01:30 to 03:00 UTC backup, with its basis in the sub-agent's job list. The message ends with one block restating every open question in full: Q4, Q5, Q6 and Q7, each with its Context, its recommendation with its reason (its basis) and its alternatives, and Q7 is marked as the changed one. Q4, Q5 and Q6 are not reduced to a one-line pointer such as \"still open, unchanged\", and no question is recorded as answered, since the user has not replied.", "files": [], "expectations": [ "Q7 is restated with 02:00 UTC named as the superseded recommendation and a replacement window outside 01:30 to 03:00 UTC whose basis is the backup job from the sub-agent's return", - "The final block of the message restates Q4, Q5, Q6 and Q7, each with its Context, its recommendation with a Basis line, and its alternatives", + "The final block of the message restates Q4, Q5, Q6 and Q7, each with its Context, its recommendation with its reason (its basis), and its alternatives", "Q7 is marked in that block as the question whose recommendation changed", "Q4, Q5 and Q6 are not collapsed into a one-line pointer to the earlier round", "No question is recorded as answered, because the user has not replied" @@ -610,11 +610,11 @@ "id": 44, "name": "held-question-release-restates-every-open-question-in-full", "prompt": "/planning:interview me Add a nightly export of order data to the analytics warehouse.\n\n[Session state: terminal lane. Round 1's Q1 to Q3 are `answered`. Round 2 was shown as Q4 (export format; recommended: Parquet, because the warehouse loader reads Parquet), Q5 (how far back the first export reaches; recommended: 90 days, from the retention setting the survey read), Q6 (whether refunds are exported as negative rows or a separate table; recommended: negative rows, from the existing finance report's shape) and Q7 (the export window; recommended: 03:30 UTC, after the nightly jobs the survey found in the repository's cron file). A sub-agent was dispatched to read the warehouse's ingest quota; Q8 (whether the export is split into batches) waits on that result and was held, so it has no register row. Before the user replies, the return lands: the warehouse accepts at most 2 GB per load, and last month's order data was 5 GB.]", - "expected_output": "The return contradicts no recommendation under Q4 to Q7 and answers none of them, but it releases the held Q8, which is now asked with a recommendation to split the export into batches under 2 GB, with its basis in the returned quota and the 5 GB monthly size. The message ends with one block restating every open question in full: Q4, Q5, Q6, Q7 and Q8, each with its Context, its recommendation with `Basis:` and its alternatives. Q4 to Q7 are not reduced to a one-line pointer to the earlier round, and no question is recorded as answered, since the user has not replied.", + "expected_output": "The return contradicts no recommendation under Q4 to Q7 and answers none of them, but it releases the held Q8, which is now asked with a recommendation to split the export into batches under 2 GB, with its basis in the returned quota and the 5 GB monthly size. The message ends with one block restating every open question in full: Q4, Q5, Q6, Q7 and Q8, each with its Context, its recommendation with its reason (its basis) and its alternatives. Q4 to Q7 are not reduced to a one-line pointer to the earlier round, and no question is recorded as answered, since the user has not replied.", "files": [], "expectations": [ "Q8 is asked with a batching recommendation whose basis is the 2 GB per-load quota from the sub-agent's return", - "The final block of the message restates Q4, Q5, Q6, Q7 and Q8, each with its Context, its recommendation with a Basis line, and its alternatives", + "The final block of the message restates Q4, Q5, Q6, Q7 and Q8, each with its Context, its recommendation with its reason (its basis), and its alternatives", "Q4 to Q7 are not collapsed into a one-line pointer to the earlier round", "No question is recorded as answered, because the user has not replied" ] @@ -666,7 +666,7 @@ "files": [], "expectations": [ "Writes no document when no other person holds the knowledge", - "Asks the eviction decision as an ordinary round with a recommendation, rather than producing a document", + "Asks the eviction decision as an ordinary decision round with a recommendation, rather than producing a document", "Never invents a recipient to justify a document" ] }, diff --git a/plugins/planning/tests/interview-defenses.test.sh b/plugins/planning/tests/interview-defenses.test.sh index 430b4a8448b..2b1cba2a427 100755 --- a/plugins/planning/tests/interview-defenses.test.sh +++ b/plugins/planning/tests/interview-defenses.test.sh @@ -935,17 +935,17 @@ pin_case_digest "sibling case 13 still requires named blockers instead of assump # it as a captured assumption" — and licensed exactly the silent capture, undetected. pin_case_digest "case 8 still asks decisions and only looks up facts" \ "facts-looked-up-decisions-asked" \ - "869a487d5f0abe19cdbe5693b90beabb55f56fbdacf6642bedad62494ce029f5" + "bedb9fd8e5923395d229e36cf50479530e58abcd8503a3ae989212162b789c45" pin_case_digest "case 12 still refuses to read drift as consent" \ "open-question-survives-an-unrelated-reply" \ "e45fe64a00cf78e6dd089837e81c52d8a0947ccf924be6472c6f47eeadcd5942" # Full restate after an out-of-band return (#8466): the next message that presents a question ends with every open row in full, the moved row marked, in place of a one-line pointer; case 25 grades that restate and case 43 adds its single-return twin; neither resolves a row the user did not answer, and the no-silent-resolve, gap and blocker defenses are unchanged pin_case_digest "case 25 still refuses to settle a row on a superseded recommendation" \ "out-of-band-return-supersedes-before-the-reply" \ - "186a511bef86bbc7fe9679359c350c39111c3eab0c8761eb8f98196882f40b9f" + "a3e7ffd4412ccfb51465f0be3d871221a6c288d53131c8436ef6025bc30974bd" pin_case_digest "case 1 still resolves codebase-answerable questions without asking, and only those" \ "relentless-me-mode-frontier-rounds" \ - "5e38782253a410890320cddc3c5c44d71ab64666bb596bc954167f1447059bdd" + "1065f9d322d8ae3b5179ac13ec75e615044f42b7ada5455737d285048032a217" pin_file "case A fixture: the task context still plants the open decision" \ "$FIXTURES/lock-stop-on-gap/task-context.md" \ From b705228b5bb91a38d3b9e4eb0df627c258ead2c5 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 21 Nov 2026 06:14:41 -0500 Subject: [PATCH 07/30] test(planning): add eval cases for the interview redesign Seven new cases under plugins/planning/evals, each one prompt with a scaffolded ledger or stated round-1 answers, deterministic graders first: - interview-numbering-tool-display-order: a ledger ending at Q7; the run calls round.sh next-id, new questions start at Q8 in the order shown, none reuses Q1-Q7, and the ledger gains an open Q8 row (AC4). - interview-numbering-stray-id-not-reused: a ledger with Q1, Q2, a declared gap and a stray Q600; new ids start at Q601 and no id changes (AC4). - interview-reopen-contradicted-answer: a reported fact contradicts answered Q3, and Q5 depends on Q3; the run calls round.sh reopen, the ledger shows Q3 open with reopened::, Q5 with recheck:: Q3, and the old answer under ## Reopened answers (AC4). - interview-held-until-lookup-returns: a lookup reports back in another session tomorrow; the dependent question sits on the Held: line and is not asked (AC9). - interview-intake-discovery-only: a first request with unknowns; round 1 asks only open questions with a Discovery: line and no My recommendation: line (AC8). - interview-sweep-reports-contradiction: every row answered, two that cannot both hold; the reply asks one question naming both. The eval evidence for removing the per-round contradiction pass (AC10). - interview-round-quality-standard: the retired round-arm prompt with graders for the standard layout (coverage of the four planted decisions, Context citing a stated fact, a recommendation whose reason names its evidence or Withheld:, alternatives, a three-dash rule between questions, the Scope check: line, the closing probe). Not part of the frozen pacing experiment. Every case validates offline (validate-cases.py exit 0) and holds no transcript. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh --- .../graders/held-line.md | 7 ++++ .../graders/held-question-not-asked.md | 16 ++++++++ .../prompt.md | 19 +++++++++ .../samples/held-line.json | 18 ++++++++ .../graders/discovery-line.md | 6 +++ .../graders/no-recommendation.md | 8 ++++ .../graders/only-open-discovery.md | 20 +++++++++ .../interview-intake-discovery-only/prompt.md | 11 +++++ .../samples/discovery-line.json | 18 ++++++++ .../samples/no-recommendation.json | 22 ++++++++++ .../case.yaml | 4 ++ .../graders/first-new-id-q601.md | 6 +++ .../graders/ledger-keeps-ids.md | 6 +++ .../graders/no-gap-filled.md | 7 ++++ .../prompt.md | 15 +++++++ .../samples/first-new-id-q601.json | 18 ++++++++ .../samples/ledger-keeps-ids.json | 18 ++++++++ .../samples/no-gap-filled.json | 22 ++++++++++ .../scaffold.sh | 14 +++++++ .../case.yaml | 4 ++ .../graders/calls-next-id.md | 7 ++++ .../graders/first-new-id-q8.md | 6 +++ .../graders/ids-in-display-order.md | 7 ++++ .../graders/no-reused-id.md | 7 ++++ .../graders/registers-q8-open.md | 6 +++ .../prompt.md | 15 +++++++ .../samples/calls-next-id.json | 28 +++++++++++++ .../samples/first-new-id-q8.json | 18 ++++++++ .../samples/ids-in-display-order.json | 22 ++++++++++ .../samples/no-reused-id.json | 22 ++++++++++ .../samples/registers-q8-open.json | 18 ++++++++ .../scaffold.sh | 21 ++++++++++ .../case.yaml | 4 ++ .../graders/calls-reopen.md | 7 ++++ .../graders/ledger-reopened.md | 6 +++ .../graders/reply-shows-reopen.md | 16 ++++++++ .../prompt.md | 15 +++++++ .../samples/calls-reopen.json | 41 +++++++++++++++++++ .../samples/ledger-reopened.json | 22 ++++++++++ .../scaffold.sh | 26 ++++++++++++ .../graders/closing-probe.md | 7 ++++ .../graders/covers-a.md | 7 ++++ .../graders/covers-b.md | 7 ++++ .../graders/covers-c.md | 7 ++++ .../graders/covers-d.md | 7 ++++ .../graders/question-quality.md | 22 ++++++++++ .../graders/rule-between-questions.md | 6 +++ .../graders/scope-check-line.md | 6 +++ .../prompt.md | 17 ++++++++ .../samples/closing-probe.json | 22 ++++++++++ .../samples/covers-a.json | 30 ++++++++++++++ .../samples/covers-b.json | 22 ++++++++++ .../samples/covers-c.json | 30 ++++++++++++++ .../samples/covers-d.json | 34 +++++++++++++++ .../samples/rule-between-questions.json | 18 ++++++++ .../samples/scope-check-line.json | 22 ++++++++++ .../case.yaml | 4 ++ .../graders/asks-reconcile.md | 17 ++++++++ .../graders/names-both-rows.md | 6 +++ .../prompt.md | 11 +++++ .../samples/names-both-rows.json | 18 ++++++++ .../scaffold.sh | 20 +++++++++ 62 files changed, 913 insertions(+) create mode 100644 plugins/planning/evals/interview-held-until-lookup-returns/graders/held-line.md create mode 100644 plugins/planning/evals/interview-held-until-lookup-returns/graders/held-question-not-asked.md create mode 100644 plugins/planning/evals/interview-held-until-lookup-returns/prompt.md create mode 100644 plugins/planning/evals/interview-held-until-lookup-returns/samples/held-line.json create mode 100644 plugins/planning/evals/interview-intake-discovery-only/graders/discovery-line.md create mode 100644 plugins/planning/evals/interview-intake-discovery-only/graders/no-recommendation.md create mode 100644 plugins/planning/evals/interview-intake-discovery-only/graders/only-open-discovery.md create mode 100644 plugins/planning/evals/interview-intake-discovery-only/prompt.md create mode 100644 plugins/planning/evals/interview-intake-discovery-only/samples/discovery-line.json create mode 100644 plugins/planning/evals/interview-intake-discovery-only/samples/no-recommendation.json create mode 100644 plugins/planning/evals/interview-numbering-stray-id-not-reused/case.yaml create mode 100644 plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/first-new-id-q601.md create mode 100644 plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/ledger-keeps-ids.md create mode 100644 plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/no-gap-filled.md create mode 100644 plugins/planning/evals/interview-numbering-stray-id-not-reused/prompt.md create mode 100644 plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/first-new-id-q601.json create mode 100644 plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/ledger-keeps-ids.json create mode 100644 plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/no-gap-filled.json create mode 100755 plugins/planning/evals/interview-numbering-stray-id-not-reused/scaffold.sh create mode 100644 plugins/planning/evals/interview-numbering-tool-display-order/case.yaml create mode 100644 plugins/planning/evals/interview-numbering-tool-display-order/graders/calls-next-id.md create mode 100644 plugins/planning/evals/interview-numbering-tool-display-order/graders/first-new-id-q8.md create mode 100644 plugins/planning/evals/interview-numbering-tool-display-order/graders/ids-in-display-order.md create mode 100644 plugins/planning/evals/interview-numbering-tool-display-order/graders/no-reused-id.md create mode 100644 plugins/planning/evals/interview-numbering-tool-display-order/graders/registers-q8-open.md create mode 100644 plugins/planning/evals/interview-numbering-tool-display-order/prompt.md create mode 100644 plugins/planning/evals/interview-numbering-tool-display-order/samples/calls-next-id.json create mode 100644 plugins/planning/evals/interview-numbering-tool-display-order/samples/first-new-id-q8.json create mode 100644 plugins/planning/evals/interview-numbering-tool-display-order/samples/ids-in-display-order.json create mode 100644 plugins/planning/evals/interview-numbering-tool-display-order/samples/no-reused-id.json create mode 100644 plugins/planning/evals/interview-numbering-tool-display-order/samples/registers-q8-open.json create mode 100755 plugins/planning/evals/interview-numbering-tool-display-order/scaffold.sh create mode 100644 plugins/planning/evals/interview-reopen-contradicted-answer/case.yaml create mode 100644 plugins/planning/evals/interview-reopen-contradicted-answer/graders/calls-reopen.md create mode 100644 plugins/planning/evals/interview-reopen-contradicted-answer/graders/ledger-reopened.md create mode 100644 plugins/planning/evals/interview-reopen-contradicted-answer/graders/reply-shows-reopen.md create mode 100644 plugins/planning/evals/interview-reopen-contradicted-answer/prompt.md create mode 100644 plugins/planning/evals/interview-reopen-contradicted-answer/samples/calls-reopen.json create mode 100644 plugins/planning/evals/interview-reopen-contradicted-answer/samples/ledger-reopened.json create mode 100755 plugins/planning/evals/interview-reopen-contradicted-answer/scaffold.sh create mode 100644 plugins/planning/evals/interview-round-quality-standard/graders/closing-probe.md create mode 100644 plugins/planning/evals/interview-round-quality-standard/graders/covers-a.md create mode 100644 plugins/planning/evals/interview-round-quality-standard/graders/covers-b.md create mode 100644 plugins/planning/evals/interview-round-quality-standard/graders/covers-c.md create mode 100644 plugins/planning/evals/interview-round-quality-standard/graders/covers-d.md create mode 100644 plugins/planning/evals/interview-round-quality-standard/graders/question-quality.md create mode 100644 plugins/planning/evals/interview-round-quality-standard/graders/rule-between-questions.md create mode 100644 plugins/planning/evals/interview-round-quality-standard/graders/scope-check-line.md create mode 100644 plugins/planning/evals/interview-round-quality-standard/prompt.md create mode 100644 plugins/planning/evals/interview-round-quality-standard/samples/closing-probe.json create mode 100644 plugins/planning/evals/interview-round-quality-standard/samples/covers-a.json create mode 100644 plugins/planning/evals/interview-round-quality-standard/samples/covers-b.json create mode 100644 plugins/planning/evals/interview-round-quality-standard/samples/covers-c.json create mode 100644 plugins/planning/evals/interview-round-quality-standard/samples/covers-d.json create mode 100644 plugins/planning/evals/interview-round-quality-standard/samples/rule-between-questions.json create mode 100644 plugins/planning/evals/interview-round-quality-standard/samples/scope-check-line.json create mode 100644 plugins/planning/evals/interview-sweep-reports-contradiction/case.yaml create mode 100644 plugins/planning/evals/interview-sweep-reports-contradiction/graders/asks-reconcile.md create mode 100644 plugins/planning/evals/interview-sweep-reports-contradiction/graders/names-both-rows.md create mode 100644 plugins/planning/evals/interview-sweep-reports-contradiction/prompt.md create mode 100644 plugins/planning/evals/interview-sweep-reports-contradiction/samples/names-both-rows.json create mode 100755 plugins/planning/evals/interview-sweep-reports-contradiction/scaffold.sh diff --git a/plugins/planning/evals/interview-held-until-lookup-returns/graders/held-line.md b/plugins/planning/evals/interview-held-until-lookup-returns/graders/held-line.md new file mode 100644 index 00000000000..6d44964e23d --- /dev/null +++ b/plugins/planning/evals/interview-held-until-lookup-returns/graders/held-line.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}(?:\*\*)?Held(?:\*\*)?:[^\n]*\b(?:who|alerted|alerting|routing|channel|page[sd]?|paging|PagerDuty|on-call)\b' +flags: i +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-held-until-lookup-returns/graders/held-question-not-asked.md b/plugins/planning/evals/interview-held-until-lookup-returns/graders/held-question-not-asked.md new file mode 100644 index 00000000000..6d36ed6aff2 --- /dev/null +++ b/plugins/planning/evals/interview-held-until-lookup-returns/graders/held-question-not-asked.md @@ -0,0 +1,16 @@ +--- +type: llm +arm: both +focus: last_message +--- + +PASS only if all of these hold: + +1. The reply does not ask who is alerted (or which channel alerts go to) as a question to answer + now, and gives no recommendation for it. +2. It says that question waits for the PagerDuty lookup, on a `Held:` line or an equivalent line in + the round header. +3. It asks the question about what an alert message includes. + +FAIL if the reply asks the who-is-alerted question now (with or without a recommendation), asks +nothing, or stops to wait for the lookup without asking the alert-content question. diff --git a/plugins/planning/evals/interview-held-until-lookup-returns/prompt.md b/plugins/planning/evals/interview-held-until-lookup-returns/prompt.md new file mode 100644 index 00000000000..2ba679601f2 --- /dev/null +++ b/plugins/planning/evals/interview-held-until-lookup-returns/prompt.md @@ -0,0 +1,19 @@ +--- +description: "Round 3 of a webhook-alerting interview with two open decisions, one of which rests on a lookup that runs in another session and reports back tomorrow, so it cannot return inside this run. Why hard: the easy moves are to ask both questions now, recommending a channel the lookup may rule out, or to wait and ask nothing; the skill holds the dependent question on the round header's Held: line and asks the other." +tags: [interview, lookups, held, no-network] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "Round 3 names the who-is-alerted question on a Held: line until the PagerDuty lookup returns and does not ask it, and asks the question about what an alert message includes" +--- + +/planning:interview me Add retries and failure alerting to our outbound webhook sender. + +Rounds 1 and 2 are answered: Q1, the problem is that failed deliveries are lost and nobody hears about them; Q2, the users are the receiving teams and our on-call engineers; Q3, the quality bar is production; Q4 to Q7 settled five attempts with exponential backoff from 30 seconds, a dead-letter table after the last attempt, and receivers deduplicating on `event_id`. + +Two decisions are open: who is alerted when deliveries to one endpoint keep failing, and what an alert message includes (the endpoint, its last response status, how many events are waiting). + +A lookup I started in another session is checking whether our PagerDuty account has the Events API enabled; it reports back tomorrow, and the answer decides whether alerts can page on-call or must go to the #webhooks Slack channel. + +Ask round 3 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-held-until-lookup-returns/samples/held-line.json b/plugins/planning/evals/interview-held-until-lookup-returns/samples/held-line.json new file mode 100644 index 00000000000..cbfbc0c4106 --- /dev/null +++ b/plugins/planning/evals/interview-held-until-lookup-returns/samples/held-line.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "**Round 3.** Alerting for failed deliveries.\nScope check: 9 of 12 questions registered\nHeld: Q8 (who is alerted) until the PagerDuty Events API lookup returns\n\n❓ Q9: **What does an alert message include?**", + "why": "the held question is named on the Held: line" + } + ], + "fail": [ + { + "answer": "**Round 3.** Alerting.\n\n❓ Q8: **Who is alerted when one endpoint keeps failing?**\n\n❓ Q9: **What does an alert message include?**", + "why": "no Held: line; both are asked" + }, + { + "answer": "**Round 3.**\nHeld: nothing\n\n❓ Q8: **What does an alert include?**", + "why": "the Held: line names no alert-routing question" + } + ] +} diff --git a/plugins/planning/evals/interview-intake-discovery-only/graders/discovery-line.md b/plugins/planning/evals/interview-intake-discovery-only/graders/discovery-line.md new file mode 100644 index 00000000000..42fbd4f8ec1 --- /dev/null +++ b/plugins/planning/evals/interview-intake-discovery-only/graders/discovery-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}(?:\*\*)?Discovery(?:\*\*)?:[^\S\n]*\S' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-intake-discovery-only/graders/no-recommendation.md b/plugins/planning/evals/interview-intake-discovery-only/graders/no-recommendation.md new file mode 100644 index 00000000000..e7c6842ee17 --- /dev/null +++ b/plugins/planning/evals/interview-intake-discovery-only/graders/no-recommendation.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}(?:\*\*)?My recommendation(?:\*\*)?:' +flags: i +match: not_contains +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-intake-discovery-only/graders/only-open-discovery.md b/plugins/planning/evals/interview-intake-discovery-only/graders/only-open-discovery.md new file mode 100644 index 00000000000..41087817013 --- /dev/null +++ b/plugins/planning/evals/interview-intake-discovery-only/graders/only-open-discovery.md @@ -0,0 +1,20 @@ +--- +type: llm +arm: both +focus: last_message +--- + +PASS only if all of these hold: + +1. The reply asks at least one numbered question about what the request leaves unknown: the + problem notifications would solve, who they are for, or what is in scope. +2. Every question in the reply is open: none proposes or recommends an answer, offers lettered + alternatives to pick from, or states an assumed answer for the user to confirm. +3. The reply asks no decision about how to build notifications (channels, delivery mechanism, + settings screens). + +A line suggesting another skill (for example brainstorming) is not a question and does not fail +check 2. + +FAIL if any question carries a recommended or assumed answer, if the reply asks a build decision, +or if it asks no question. diff --git a/plugins/planning/evals/interview-intake-discovery-only/prompt.md b/plugins/planning/evals/interview-intake-discovery-only/prompt.md new file mode 100644 index 00000000000..74a575a1a4f --- /dev/null +++ b/plugins/planning/evals/interview-intake-discovery-only/prompt.md @@ -0,0 +1,11 @@ +--- +description: "A first request that names a feature but not its problem, users or scope. Why hard: the skill can draft plausible answers for all of them, so the easy move is a first round of recommended answers that puts words in the user's mouth; the gap triage sorts these unknowns into discovery questions, and round 1 asks only those, open, each with a Discovery: line and no recommendation." +tags: [interview, intake, discovery, no-network] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "Round 1 asks only open discovery questions (the problem, who it is for, what is in scope), each with a Discovery: line and none with a My recommendation: line, then stops for the answers" +--- + +/planning:interview Add notifications to our team task app. I'm here and will answer; ask me your first round, then stop. diff --git a/plugins/planning/evals/interview-intake-discovery-only/samples/discovery-line.json b/plugins/planning/evals/interview-intake-discovery-only/samples/discovery-line.json new file mode 100644 index 00000000000..0fd7ed7609e --- /dev/null +++ b/plugins/planning/evals/interview-intake-discovery-only/samples/discovery-line.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "❓ Q1: **What problem should notifications solve?**\n\nContext:\n- The request names the feature only (user, round 1).\n\nDiscovery: the answer sets what the feature must do.", + "why": "a Discovery: line" + } + ], + "fail": [ + { + "answer": "❓ Q1: **What problem should notifications solve?**\n\nWithheld: only you know why you want this.", + "why": "the old withheld shape, no Discovery: line" + }, + { + "answer": "This round is for discovery: we learn the problem first.\n\nQ1: **What problem?**", + "why": "the word mid-sentence, lowercase, is not the label" + } + ] +} diff --git a/plugins/planning/evals/interview-intake-discovery-only/samples/no-recommendation.json b/plugins/planning/evals/interview-intake-discovery-only/samples/no-recommendation.json new file mode 100644 index 00000000000..54a0cac2d47 --- /dev/null +++ b/plugins/planning/evals/interview-intake-discovery-only/samples/no-recommendation.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 1.** Notifications for the team task app.\nScope check: 3 of 12 questions registered\n\n❓ Q1: **What problem should notifications solve?**\n\nContext:\n- The request names the feature, not the problem it solves (user, round 1).\n\nDiscovery: the answer sets what the feature must do.\n\n---\n\n❓ Q2: **Who are they for?**\n\nDiscovery: the answer sets whose workflow the design follows.", + "why": "open discovery questions, no recommendation line" + }, + { + "answer": "Q1: **What problem should this solve?**\nI have no recommendation yet; your answer decides it.", + "why": "the word inside a sentence is not the label" + } + ], + "fail": [ + { + "answer": "❓ Q1: **What problem should notifications solve?**\n\n➡️ My recommendation: **Missed task assignments**.", + "why": "a recommendation in the intake round" + }, + { + "answer": "Q1: **Who are they for?**\n**My recommendation:** the assignees", + "why": "a bold recommendation label" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/case.yaml b/plugins/planning/evals/interview-numbering-stray-id-not-reused/case.yaml new file mode 100644 index 00000000000..7d3108b6210 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: interview-numbering-stray-id-not-reused +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/first-new-id-q601.md b/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/first-new-id-q601.md new file mode 100644 index 00000000000..3f51a4c373f --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/first-new-id-q601.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '^(?:(?!(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?(?:^|\n)[^\n]{0,6}Q601:[^\S\n]*\*\*' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/ledger-keeps-ids.md b/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/ledger-keeps-ids.md new file mode 100644 index 00000000000..4382956fecc --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/ledger-keeps-ids.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)- Q1 \| answered \|[^\n]*\n- Q2 \| answered \|[\s\S]*\n- Q600 \| answered \| round 1 \|[\s\S]*\n- Q601 \| open \|' +target: { source: file, path: .work/invoice-export/interview-checklist.md } +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/no-gap-filled.md b/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/no-gap-filled.md new file mode 100644 index 00000000000..5ea5a8aa9d1 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/no-gap-filled.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q(?:[3-9]|[1-9]\d|[1-5]\d\d):[^\S\n]*\*\*' +match: not_contains +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/prompt.md b/plugins/planning/evals/interview-numbering-stray-id-not-reused/prompt.md new file mode 100644 index 00000000000..69e443d7e69 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/prompt.md @@ -0,0 +1,15 @@ +--- +description: "The ledger holds Q1, Q2 and a stray Q600 left by an earlier session, with a gap line declaring Q3-Q599. Why hard: the natural move is to number the next question Q3, filling the gap, or to renumber Q600 down to Q3; the skill keeps every id and takes the next one from round.sh next-id, so new questions start at Q601." +tags: [interview, numbering, register, scaffold] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] +expected_outcome: "The reply's new questions start at Q601; no new question takes Q3 to Q599; and the ledger still holds Q1, Q2 and Q600 under their numbers, with the new rows from Q601" +--- + +/planning:interview Resume the invoice-export interview; the ledger is .work/invoice-export/interview-checklist.md and round 1 is answered. + +Still open: which date range one export covers (finance closes its books on the calendar month), and how amounts are written (invoices store integer cents with a currency code, and customers are billed in USD, EUR and GBP). + +Ask round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/first-new-id-q601.json b/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/first-new-id-q601.json new file mode 100644 index 00000000000..891bca6d113 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/first-new-id-q601.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "**Round 2.** A production month-end export for finance.\nScope check: 5 of 12 questions registered\n\n❓ Q601: **Should one export cover one calendar month?**\n\nContext:\n- Finance closes on the calendar month (your notes).\n\n➡️ My recommendation: **Yes**.\n\n---\n\n❓ Q602: **How are amounts written?**", + "why": "new ids start at Q601" + } + ], + "fail": [ + { + "answer": "**Round 2.** A production month-end export for finance.\nScope check: 5 of 12 questions registered\n\n❓ Q3: **Should one export cover one calendar month?**\n\nContext:\n- Finance closes on the calendar month (your notes).\n\n➡️ My recommendation: **Yes**.\n\n---\n\n❓ Q4: **How are amounts written?**", + "why": "the gap is filled" + }, + { + "answer": "**Round 2.** A production month-end export for finance.\nScope check: 5 of 12 questions registered\n\n❓ Q602: **Should one export cover one calendar month?**\n\nContext:\n- Finance closes on the calendar month (your notes).\n\n➡️ My recommendation: **Yes**.\n\n---\n\n❓ Q601: **How are amounts written?**", + "why": "Q601 is not the first question shown" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/ledger-keeps-ids.json b/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/ledger-keeps-ids.json new file mode 100644 index 00000000000..9047fc39049 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/ledger-keeps-ids.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "## Open-question register\n\n- Q1 | answered | round 1 | What problem? | manual rebuild\n- Q2 | answered | round 1 | Who uses it? | finance\n- gap | Q3-Q599 | ids skipped by an earlier session\n- Q600 | answered | round 1 | What quality bar? | production\n- Q601 | open | round 2 | Should one export cover one calendar month? |\n", + "why": "old ids kept, new row Q601" + } + ], + "fail": [ + { + "answer": "## Open-question register\n\n- Q1 | answered | round 1 | What problem? | manual rebuild\n- Q2 | answered | round 1 | Who uses it? | finance\n- gap | Q3-Q599 | ids skipped by an earlier session\n- Q3 | answered | round 1 | What quality bar? | production\n- Q4 | open | round 2 | Should one export cover one calendar month? |\n", + "why": "Q600 renumbered down" + }, + { + "answer": "## Open-question register\n\n- Q1 | answered | round 1 | What problem? | manual rebuild\n- Q2 | answered | round 1 | Who uses it? | finance\n- gap | Q3-Q599 | ids skipped by an earlier session\n- Q600 | answered | round 1 | What quality bar? | production\n- Q3 | open | round 2 | Date range? |\n", + "why": "the new row fills the gap" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/no-gap-filled.json b/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/no-gap-filled.json new file mode 100644 index 00000000000..80df7441577 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/no-gap-filled.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** A production month-end export for finance.\nScope check: 5 of 12 questions registered\n\n❓ Q601: **Should one export cover one calendar month?**\n\nContext:\n- Finance closes on the calendar month (your notes).\n\n➡️ My recommendation: **Yes**.\n\n---\n\n❓ Q602: **How are amounts written?**", + "why": "no title in Q3-Q599" + }, + { + "answer": "Context:\n- Q2 said the users are finance (round 1).\n\n❓ Q601: **Date range?**", + "why": "a cited id is not a title" + } + ], + "fail": [ + { + "answer": "❓ Q3: **Date range?**", + "why": "Q3 fills the gap" + }, + { + "answer": "❓ Q599: **Date range?**", + "why": "inside the declared gap" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/scaffold.sh b/plugins/planning/evals/interview-numbering-stray-id-not-reused/scaffold.sh new file mode 100755 index 00000000000..9745bb5806e --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/scaffold.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +set -euo pipefail + +mkdir -p .work/invoice-export +cat >.work/invoice-export/interview-checklist.md <<'MD' +# /planning:interview Checklist + +## Open-question register + +- Q1 | answered | round 1 | What problem does this solve? | finance rebuilds the month's invoices by hand for reconciliation +- Q2 | answered | round 1 | Who uses it? | the finance team +- gap | Q3-Q599 | ids skipped by an earlier session +- Q600 | answered | round 1 | What quality bar does this work meet? | production +MD diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/case.yaml b/plugins/planning/evals/interview-numbering-tool-display-order/case.yaml new file mode 100644 index 00000000000..c99cf7fcd1d --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: interview-numbering-tool-display-order +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/graders/calls-next-id.md b/plugins/planning/evals/interview-numbering-tool-display-order/graders/calls-next-id.md new file mode 100644 index 00000000000..8caf45c7265 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/graders/calls-next-id.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Bash +input_match: 'round\.sh\W{0,4}\s+next-id\b' +min: 1 +arm: with-only +--- diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/graders/first-new-id-q8.md b/plugins/planning/evals/interview-numbering-tool-display-order/graders/first-new-id-q8.md new file mode 100644 index 00000000000..03cbb6e0ef0 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/graders/first-new-id-q8.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '^(?:(?!(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?(?:^|\n)[^\n]{0,6}Q8:[^\S\n]*\*\*' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/graders/ids-in-display-order.md b/plugins/planning/evals/interview-numbering-tool-display-order/graders/ids-in-display-order.md new file mode 100644 index 00000000000..4abd7ad6fc5 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/graders/ids-in-display-order.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q(?:8:[^\S\n]*\*\*(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}Q(?!9:)|9:[^\S\n]*\*\*(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}Q(?!10:)|10:[^\S\n]*\*\*(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}Q(?!11:)|11:[^\S\n]*\*\*(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}Q(?!12:))\d+:[^\S\n]*\*\*' +match: not_contains +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/graders/no-reused-id.md b/plugins/planning/evals/interview-numbering-tool-display-order/graders/no-reused-id.md new file mode 100644 index 00000000000..49c50e81c23 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/graders/no-reused-id.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q[1-7]:[^\S\n]*\*\*' +match: not_contains +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/graders/registers-q8-open.md b/plugins/planning/evals/interview-numbering-tool-display-order/graders/registers-q8-open.md new file mode 100644 index 00000000000..92060f2148f --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/graders/registers-q8-open.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)- Q8 \| open \| round 3 \|' +target: { source: file, path: .work/webhook-retries/interview-checklist.md } +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/prompt.md b/plugins/planning/evals/interview-numbering-tool-display-order/prompt.md new file mode 100644 index 00000000000..a7e8d7859d0 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/prompt.md @@ -0,0 +1,15 @@ +--- +description: "A round is due on a ledger whose register ends at Q7. Why hard: the model can see the last id and pick the next ones itself, which is how ids once jumped or were reused; the skill says ids come from round.sh next-id, in the order the round shows the questions." +tags: [interview, numbering, register, scaffold] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] +expected_outcome: "The run calls round.sh next-id; the reply's new questions start at Q8 and run upward in the order shown; no new question reuses Q1 to Q7; and the ledger gains open rows from Q8" +--- + +/planning:interview Resume the webhook-retries interview; the ledger is .work/webhook-retries/interview-checklist.md and every question so far is answered. + +The alerting side is still open: who is alerted when deliveries to one endpoint keep failing (the on-call rota is in PagerDuty, and the team also has a #webhooks Slack channel), and what an alert message includes. + +Ask round 3 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/samples/calls-next-id.json b/plugins/planning/evals/interview-numbering-tool-display-order/samples/calls-next-id.json new file mode 100644 index 00000000000..b3b92d7c092 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/samples/calls-next-id.json @@ -0,0 +1,28 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Bash", + "input": { + "command": "bash \"/plugins/planning/surface/round.sh\" next-id --ledger .work/webhook-retries/interview-checklist.md --count 2" + } + } + ], + "why": "ids come from the tool" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/webhook-retries/interview-checklist.md" + } + } + ], + "why": "the ledger is read and ids picked by hand" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/samples/first-new-id-q8.json b/plugins/planning/evals/interview-numbering-tool-display-order/samples/first-new-id-q8.json new file mode 100644 index 00000000000..0cd5c37257d --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/samples/first-new-id-q8.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "**Round 3.** Alerting for failed deliveries.\nScope check: 9 of 12 questions registered\n\n❓ Q8: **Who is alerted when one endpoint keeps failing?**\n\nContext:\n- The on-call rota is in PagerDuty (your notes).\n\n➡️ My recommendation: **Page on-call**. (verified, your notes)\n\n---\n\n❓ Q9: **What does an alert include?**\n\nContext:\n- x\n\n➡️ My recommendation: **The endpoint and its last status**.\n\nIs there something else that would change these recommendations?", + "why": "the first question shown is Q8" + } + ], + "fail": [ + { + "answer": "**Round 3.** Alerting for failed deliveries.\nScope check: 9 of 12 questions registered\n\n❓ Q10: **Who is alerted when one endpoint keeps failing?**\n\nContext:\n- The on-call rota is in PagerDuty (your notes).\n\n➡️ My recommendation: **Page on-call**. (verified, your notes)\n\n---\n\n❓ Q11: **What does an alert include?**\n\nContext:\n- x\n\n➡️ My recommendation: **The endpoint and its last status**.\n\nIs there something else that would change these recommendations?", + "why": "the first new id skips Q8" + }, + { + "answer": "❓ Q9: **What does an alert include?**\n\n❓ Q8: **Who is alerted?**", + "why": "Q8 is not the first question shown" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/samples/ids-in-display-order.json b/plugins/planning/evals/interview-numbering-tool-display-order/samples/ids-in-display-order.json new file mode 100644 index 00000000000..8b9172241a6 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/samples/ids-in-display-order.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 3.** Alerting for failed deliveries.\nScope check: 9 of 12 questions registered\n\n❓ Q8: **Who is alerted when one endpoint keeps failing?**\n\nContext:\n- The on-call rota is in PagerDuty (your notes).\n\n➡️ My recommendation: **Page on-call**. (verified, your notes)\n\n---\n\n❓ Q9: **What does an alert include?**\n\nContext:\n- x\n\n➡️ My recommendation: **The endpoint and its last status**.\n\nIs there something else that would change these recommendations?", + "why": "Q8 then Q9" + }, + { + "answer": "❓ Q8: **Who?**\n\n---\n\n❓ Q9: **What?**\n\n---\n\n❓ Q10: **When?**", + "why": "Q8, Q9, Q10 in order" + } + ], + "fail": [ + { + "answer": "❓ Q8: **Who?**\n\n❓ Q10: **What?**", + "why": "Q9 is skipped" + }, + { + "answer": "❓ Q9: **What?**\n\n❓ Q8: **Who?**", + "why": "shown out of order" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/samples/no-reused-id.json b/plugins/planning/evals/interview-numbering-tool-display-order/samples/no-reused-id.json new file mode 100644 index 00000000000..d423a76dc78 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/samples/no-reused-id.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 3.** Alerting for failed deliveries.\nScope check: 9 of 12 questions registered\n\n❓ Q8: **Who is alerted when one endpoint keeps failing?**\n\nContext:\n- The on-call rota is in PagerDuty (your notes).\n\n➡️ My recommendation: **Page on-call**. (verified, your notes)\n\n---\n\n❓ Q9: **What does an alert include?**\n\nContext:\n- x\n\n➡️ My recommendation: **The endpoint and its last status**.\n\nIs there something else that would change these recommendations?", + "why": "no new question takes Q1 to Q7" + }, + { + "answer": "Q8: **Who is alerted?**\nContext:\n- Q4 settled five attempts (round 2).", + "why": "citing an earlier id in Context is not a question title" + } + ], + "fail": [ + { + "answer": "❓ Q1: **Who is alerted?**", + "why": "the new question reuses Q1" + }, + { + "answer": "❓ Q7: **What does an alert include?**", + "why": "the new question reuses Q7" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/samples/registers-q8-open.json b/plugins/planning/evals/interview-numbering-tool-display-order/samples/registers-q8-open.json new file mode 100644 index 00000000000..a5a876f24f6 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/samples/registers-q8-open.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "## Open-question register\n\n- Q7 | answered | round 2 | Do receivers deduplicate? | yes\n- Q8 | open | round 3 | Who is alerted when one endpoint keeps failing? |\n", + "why": "Q8 is registered open in round 3" + } + ], + "fail": [ + { + "answer": "## Open-question register\n\n- Q7 | answered | round 2 | Do receivers deduplicate? | yes\n", + "why": "no new row" + }, + { + "answer": "- Q600 | open | round 3 | Who is alerted? |\n", + "why": "a jumped id" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/scaffold.sh b/plugins/planning/evals/interview-numbering-tool-display-order/scaffold.sh new file mode 100755 index 00000000000..b9ff731c007 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/scaffold.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +set -euo pipefail + +mkdir -p .work/webhook-retries +cat >.work/webhook-retries/interview-checklist.md <<'MD' +# /planning:interview Checklist + +## Constraint ledger + +- C1 | confirmed | webhooks are delivered at least once | user, round 1 + +## Open-question register + +- Q1 | answered | round 1 | What problem does this solve? | failed deliveries are lost and nobody hears about them +- Q2 | answered | round 1 | Who uses it? | the receiving teams and our on-call engineers +- Q3 | answered | round 1 | What quality bar does this work meet? | production +- Q4 | answered | round 2 | How many delivery attempts before the sender gives up? | five +- Q5 | answered | round 2 | What delay between delivery attempts? | exponential backoff from 30 seconds +- Q6 | answered | round 2 | What happens to an event after its last failed attempt? | it moves to a dead-letter table +- Q7 | answered | round 2 | Do receivers deduplicate repeated deliveries? | yes, on the event_id header +MD diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/case.yaml b/plugins/planning/evals/interview-reopen-contradicted-answer/case.yaml new file mode 100644 index 00000000000..d383549a7da --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: interview-reopen-contradicted-answer +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/graders/calls-reopen.md b/plugins/planning/evals/interview-reopen-contradicted-answer/graders/calls-reopen.md new file mode 100644 index 00000000000..f92d2cc3bbe --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/graders/calls-reopen.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Bash +input_match: 'round\.sh\W{0,4}\s+reopen\b[^\n]*\bQ3\b' +min: 1 +arm: with-only +--- diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/graders/ledger-reopened.md b/plugins/planning/evals/interview-reopen-contradicted-answer/graders/ledger-reopened.md new file mode 100644 index 00000000000..12d85304f3b --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/graders/ledger-reopened.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '^(?=[\s\S]*\n- Q3 \| open \|[^\n]*reopened:: round \d+)(?=[\s\S]*\n- Q5 \|[^\n]*recheck:: Q3\b)(?=[\s\S]*\n## Reopened answers\s*\n(?:[^\n]*\n)*?- Q3 \| round \d+ \| was: [^\n]*Redis)' +target: { source: file, path: .work/api-rate-limit/interview-checklist.md } +arm: both +--- diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/graders/reply-shows-reopen.md b/plugins/planning/evals/interview-reopen-contradicted-answer/graders/reply-shows-reopen.md new file mode 100644 index 00000000000..d5beb557a17 --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/graders/reply-shows-reopen.md @@ -0,0 +1,16 @@ +--- +type: llm +arm: both +focus: last_message +--- + +PASS only if all of these hold: + +1. The reply asks Q3 again under the id Q3 (not a new id), as a reopened question. +2. It shows Q3's earlier answer (the counters in the existing shared Redis) and the fact that + contradicts it (docs/platform/redis.md: the shared Redis holds web sessions only). +3. It names Q5 (the counting scheme) as resting on Q3 and needing a recheck once Q3 is answered + again. + +FAIL if the reply changes Q3's answer itself without asking, asks the storage question under a new +id, or reopens Q3 without naming Q5. diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/prompt.md b/plugins/planning/evals/interview-reopen-contradicted-answer/prompt.md new file mode 100644 index 00000000000..fa269e512f8 --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/prompt.md @@ -0,0 +1,15 @@ +--- +description: "A fact reported after round 2 contradicts answered Q3, and Q5 depends on Q3. Why hard: the easy moves are to quietly change Q3's answer, to ask a fresh question under a new id, or to reopen Q3 and forget Q5; the skill reopens Q3 under its own number with round.sh reopen, keeps the old answer, and flags Q5 for recheck." +tags: [interview, reopen, register, scaffold] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] +expected_outcome: "The run calls round.sh reopen for Q3; the reply shows Q3 reopened under its own number with its old answer (the shared Redis) and the contradicting fact from docs/platform/redis.md, and names Q5 for recheck; the ledger's Q3 row is open with reopened::, Q5 carries recheck:: Q3, and a Reopened answers section holds Q3's old answer" +--- + +/planning:interview Resume the api-rate-limit interview; the ledger is .work/api-rate-limit/interview-checklist.md and rounds 1 and 2 are answered. + +The lookup I asked for came back: docs/platform/redis.md says the shared Redis instance holds web sessions only and allows no other application keys. + +Carry on with round 3, then stop for my answers. diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/samples/calls-reopen.json b/plugins/planning/evals/interview-reopen-contradicted-answer/samples/calls-reopen.json new file mode 100644 index 00000000000..c1005b783ad --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/samples/calls-reopen.json @@ -0,0 +1,41 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Bash", + "input": { + "command": "bash \"/plugins/planning/surface/round.sh\" reopen --ledger .work/api-rate-limit/interview-checklist.md Q3 --round 3 --fact-file .work/api-rate-limit/fact-q3.txt" + } + } + ], + "why": "Q3 reopened through the tool" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Edit", + "input": { + "file_path": ".work/api-rate-limit/interview-checklist.md", + "old_string": "| answered | round 2 | Where", + "new_string": "| open | round 2 | Where" + } + } + ], + "why": "the row is hand-edited" + }, + { + "answer": [ + { + "tool": "Bash", + "input": { + "command": "bash round.sh reopen --ledger x.md Q5 --round 3 --fact-file f" + } + } + ], + "why": "the wrong row is reopened" + } + ] +} diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/samples/ledger-reopened.json b/plugins/planning/evals/interview-reopen-contradicted-answer/samples/ledger-reopened.json new file mode 100644 index 00000000000..da6ce3cfa06 --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/samples/ledger-reopened.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "# /planning:interview Checklist\n\n## Open-question register\n\n- Q1 | answered | round 1 | What problem? | bursts\n- Q3 | open | round 2 | Where are the rate-limit counters stored? | reopened:: round 3\n- Q4 | answered | round 2 | What limit? | 1,000 per minute\n- Q5 | answered | round 2 | How are requests counted? | a Redis counter per key per minute window; depends:: Q3; recheck:: Q3\n\n## Reopened answers\n\n- Q3 | round 3 | was: in the existing shared Redis 7 instance | fact: docs/platform/redis.md allows sessions only\n", + "why": "reopened, recheck flagged, old answer kept" + } + ], + "fail": [ + { + "answer": "# /planning:interview Checklist\n\n## Open-question register\n\n- Q1 | answered | round 1 | What problem? | bursts\n- Q3 | open | round 2 | Where are the rate-limit counters stored? | reopened:: round 3\n- Q4 | answered | round 2 | What limit? | 1,000 per minute\n- Q5 | answered | round 2 | How are requests counted? | a Redis counter per key per minute window; depends:: Q3\n\n## Reopened answers\n\n- Q3 | round 3 | was: in the existing shared Redis 7 instance | fact: docs/platform/redis.md allows sessions only\n", + "why": "Q5 not flagged" + }, + { + "answer": "# /planning:interview Checklist\n\n## Open-question register\n\n- Q1 | answered | round 1 | What problem? | bursts\n- Q3 | open | round 2 | Where are the rate-limit counters stored? | reopened:: round 3\n- Q4 | answered | round 2 | What limit? | 1,000 per minute\n- Q5 | answered | round 2 | How are requests counted? | a Redis counter per key per minute window; depends:: Q3; recheck:: Q3\n\n", + "why": "the old answer is dropped" + }, + { + "answer": "# /planning:interview Checklist\n\n## Open-question register\n\n- Q1 | answered | round 1 | What problem? | bursts\n- Q3 | answered | round 2 | Where are the rate-limit counters stored? | a dedicated Redis instance\n- Q4 | answered | round 2 | What limit? | 1,000 per minute\n- Q5 | answered | round 2 | How are requests counted? | a Redis counter per key per minute window; depends:: Q3; recheck:: Q3\n\n## Reopened answers\n\n- Q3 | round 3 | was: in the existing shared Redis 7 instance | fact: docs/platform/redis.md allows sessions only\n", + "why": "Q3's answer was changed in place" + } + ] +} diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/scaffold.sh b/plugins/planning/evals/interview-reopen-contradicted-answer/scaffold.sh new file mode 100755 index 00000000000..8c9b445e988 --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/scaffold.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail + +mkdir -p .work/api-rate-limit docs/platform +cat >docs/platform/redis.md <<'MD' +# Shared Redis + +The shared Redis 7 instance holds web sessions only. Application keys of any other kind are not +allowed on it; request a dedicated instance from the platform team. +MD +cat >.work/api-rate-limit/interview-checklist.md <<'MD' +# /planning:interview Checklist + +## Open-question register + +- Q1 | answered | round 1 | What problem does this solve? | a few partner integrations send bursts that slow the API for everyone +- Q2 | answered | round 1 | Who uses it? | partner developers calling the API with their keys +- Q3 | answered | round 2 | Where are the rate-limit counters stored? | in the existing shared Redis 7 instance +- Q4 | answered | round 2 | What limit applies? | 1,000 requests per minute per API key, as the partner terms promise +- Q5 | answered | round 2 | How are requests counted? | a Redis counter per key per minute window, expiring after the window; depends:: Q3 +MD +git init -q -b main +git config user.name eval +git config user.email eval@example.invalid +git add . +git commit -q -m "docs: platform redis policy" diff --git a/plugins/planning/evals/interview-round-quality-standard/graders/closing-probe.md b/plugins/planning/evals/interview-round-quality-standard/graders/closing-probe.md new file mode 100644 index 00000000000..1fd5f0c18bf --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/graders/closing-probe.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: 'something else that would change' +flags: i +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-round-quality-standard/graders/covers-a.md b/plugins/planning/evals/interview-round-quality-standard/graders/covers-a.md new file mode 100644 index 00000000000..765e72238ec --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/graders/covers-a.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)' +flags: i +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-round-quality-standard/graders/covers-b.md b/plugins/planning/evals/interview-round-quality-standard/graders/covers-b.md new file mode 100644 index 00000000000..7925aa6602e --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/graders/covers-b.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))' +flags: i +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-round-quality-standard/graders/covers-c.md b/plugins/planning/evals/interview-round-quality-standard/graders/covers-c.md new file mode 100644 index 00000000000..c5213709dcb --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/graders/covers-c.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)' +flags: i +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-round-quality-standard/graders/covers-d.md b/plugins/planning/evals/interview-round-quality-standard/graders/covers-d.md new file mode 100644 index 00000000000..f76266389ee --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/graders/covers-d.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)' +flags: i +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-round-quality-standard/graders/question-quality.md b/plugins/planning/evals/interview-round-quality-standard/graders/question-quality.md new file mode 100644 index 00000000000..5a2b116679b --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/graders/question-quality.md @@ -0,0 +1,22 @@ +--- +type: llm +arm: both +focus: last_message +--- + +Find the round's questions on these four decisions: the date range one export covers, how amounts +are written across currencies, who may export, and how large exports are built. A separate +question that defers the large-export mechanism to planning does not replace the large-export +question. + +PASS only if each of the four has a question, and each of those four questions has all of: + +1. A `Context:` block with at least one line that cites a fact the user stated (the calendar-month + close, integer cents with a currency code, USD/EUR/GBP billing, the spreadsheet, the support and + finance roles, the 40,000 rows, the 30-second request timeout) and says where it came from. +2. Either a recommendation whose printed reason names the evidence it rests on and its source (for + example "(verified, your notes)"), or a `Withheld:` line naming the evidence that would settle it. +3. Alternatives, each saying what it would gain or cost. + +FAIL if any of the four decisions has no question, or any of those questions lacks one of the +three parts, or a recommendation's reason names no evidence (bare judgment). diff --git a/plugins/planning/evals/interview-round-quality-standard/graders/rule-between-questions.md b/plugins/planning/evals/interview-round-quality-standard/graders/rule-between-questions.md new file mode 100644 index 00000000000..aac529f3f83 --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/graders/rule-between-questions.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\S\n]*-{3}[^\S\n]*\n(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-round-quality-standard/graders/scope-check-line.md b/plugins/planning/evals/interview-round-quality-standard/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-round-quality-standard/prompt.md b/plugins/planning/evals/interview-round-quality-standard/prompt.md new file mode 100644 index 00000000000..9723a3a858a --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/prompt.md @@ -0,0 +1,17 @@ +--- +description: "Question quality at the default standard detail level, on the retired pacing experiment's round-arm task (round 2 of an invoice CSV export, four planted decisions: date range, amount format across currencies, who may export, large exports). Source: the consumer's own writing, synthetic. Not part of that frozen experiment. Why hard: the four decisions are not listed as a frontier; the skill must find each in the stated facts and ask all of them in one round, each with a Context line citing a stated fact, a recommendation whose printed reason names its evidence (or a Withheld: line), and alternatives, with a three-dash rule line between questions and a Scope check: line in the header." +tags: [interview, round-layout, question-quality, hard, no-network] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "Round 2, headed by a Scope check: line, asks a question on each of date range, amount format and currencies, who may export, and large exports, every question with a Context line citing a stated fact, a recommendation whose printed reason names its evidence or a Withheld: line, and alternatives, with a three-dash rule line between questions, ending with the closing probe" +--- + +/planning:interview me Add a CSV export of invoices to the billing admin page. + +Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. + +What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. + +Ask round 2 now, the whole round in one message, then stop for my answers. diff --git a/plugins/planning/evals/interview-round-quality-standard/samples/closing-probe.json b/plugins/planning/evals/interview-round-quality-standard/samples/closing-probe.json new file mode 100644 index 00000000000..e04406c4c79 --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/samples/closing-probe.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "ends with the closing probe" + }, + { + "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", + "why": "ends with the closing probe" + } + ], + "fail": [ + { + "answer": "", + "why": "empty" + }, + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nReply when ready.", + "why": "no closing probe" + } + ] +} diff --git a/plugins/planning/evals/interview-round-quality-standard/samples/covers-a.json b/plugins/planning/evals/interview-round-quality-standard/samples/covers-a.json new file mode 100644 index 00000000000..9a48f709d64 --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/samples/covers-a.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" + }, + { + "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", + "why": "single pacing, one question on the date range" + } + ], + "fail": [ + { + "answer": "", + "why": "empty" + }, + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "no question on date range" + }, + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "date range appears only in a recap line, not as a question" + }, + { + "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Which date puts an invoice into a calendar month's export?**\nContext:\n- Finance closes its books on the calendar month, and an invoice has issue, due and paid dates (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", + "why": "which date places an invoice in a month is a sub-decision, not the date range one export covers" + } + ] +} diff --git a/plugins/planning/evals/interview-round-quality-standard/samples/covers-b.json b/plugins/planning/evals/interview-round-quality-standard/samples/covers-b.json new file mode 100644 index 00000000000..368a0108dac --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/samples/covers-b.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" + } + ], + "fail": [ + { + "answer": "", + "why": "empty" + }, + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "no question on amount and currency format" + }, + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "amount and currency format appears only in a recap line, not as a question" + } + ] +} diff --git a/plugins/planning/evals/interview-round-quality-standard/samples/covers-c.json b/plugins/planning/evals/interview-round-quality-standard/samples/covers-c.json new file mode 100644 index 00000000000..d316c5e9396 --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/samples/covers-c.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" + }, + { + "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", + "why": "single pacing after A and D were answered, one question on who may export; the recap line asks nothing" + }, + { + "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", + "why": "who may export, phrased as who may start and download the file" + } + ], + "fail": [ + { + "answer": "", + "why": "empty" + }, + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "no question on who may export" + }, + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "who may export appears only in a recap line, not as a question" + } + ] +} diff --git a/plugins/planning/evals/interview-round-quality-standard/samples/covers-d.json b/plugins/planning/evals/interview-round-quality-standard/samples/covers-d.json new file mode 100644 index 00000000000..cdb7c3a2087 --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/samples/covers-d.json @@ -0,0 +1,34 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" + }, + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be prepared after the click and downloaded once it's ready, instead of being built during the click like the page's other downloads?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "D phrased as preparing the file after the click and downloading it once it's ready" + } + ], + "fail": [ + { + "answer": "", + "why": "empty" + }, + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "no question on how a large export completes" + }, + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "D appears only inside the question that defers the mechanism to planning; D itself is never asked" + }, + { + "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nSettled by evidence: a large export runs as a background job, because one month is about 40,000 rows and in-request downloads time out after 30 seconds (your notes).\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", + "why": "D settled in a preface line with live alternatives left, and never asked" + }, + { + "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", + "why": "a question on who may export that mentions the finished file is not D" + } + ] +} diff --git a/plugins/planning/evals/interview-round-quality-standard/samples/rule-between-questions.json b/plugins/planning/evals/interview-round-quality-standard/samples/rule-between-questions.json new file mode 100644 index 00000000000..fe1ac8e278b --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/samples/rule-between-questions.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "❓ Q4: **Should one export cover one calendar month?**\n\nContext:\n- Finance closes on the calendar month (your notes).\n\n➡️ My recommendation: **Yes**. (verified, your notes)\n\nAlternatives:\n- (a) A free date range.\n\n---\n\n❓ Q5: **How are amounts written?**", + "why": "a --- rule between two questions" + } + ], + "fail": [ + { + "answer": "❓ Q4: **Should one export cover one calendar month?**\n\nAlternatives:\n- (a) A free date range.\n\n❓ Q5: **How are amounts written?**", + "why": "no rule between the questions" + }, + { + "answer": "---\n\n❓ Q4: **Should one export cover one calendar month?**\n\n❓ Q5: **How are amounts written?**\n\n---", + "why": "rules only before the first and after the last question" + } + ] +} diff --git a/plugins/planning/evals/interview-round-quality-standard/samples/scope-check-line.json b/plugins/planning/evals/interview-round-quality-standard/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-sweep-reports-contradiction/case.yaml b/plugins/planning/evals/interview-sweep-reports-contradiction/case.yaml new file mode 100644 index 00000000000..93b405d1a48 --- /dev/null +++ b/plugins/planning/evals/interview-sweep-reports-contradiction/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: interview-sweep-reports-contradiction +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-sweep-reports-contradiction/graders/asks-reconcile.md b/plugins/planning/evals/interview-sweep-reports-contradiction/graders/asks-reconcile.md new file mode 100644 index 00000000000..d33395fd99a --- /dev/null +++ b/plugins/planning/evals/interview-sweep-reports-contradiction/graders/asks-reconcile.md @@ -0,0 +1,17 @@ +--- +type: llm +arm: both +focus: last_message +--- + +PASS only if all of these hold: + +1. The reply asks the user a question that names both Q2 (only the finance role may export) and + Q5 (a support agent exports the file for a customer) and says they cannot both hold as written. +2. The question asks which holds or how to reconcile them (for example a customer-copy export + support may run that leaves payment details out, or finance sending the copy). +3. The reply does not present the interview as complete or write the Brief as if the two answers + agreed. + +FAIL if the reply does not raise the conflict, resolves it itself without asking, or raises it only +as a note with no question. diff --git a/plugins/planning/evals/interview-sweep-reports-contradiction/graders/names-both-rows.md b/plugins/planning/evals/interview-sweep-reports-contradiction/graders/names-both-rows.md new file mode 100644 index 00000000000..2c274f517c5 --- /dev/null +++ b/plugins/planning/evals/interview-sweep-reports-contradiction/graders/names-both-rows.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?(?:\bQ2\b(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\bQ5\b|\bQ5\b(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\bQ2\b)' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-sweep-reports-contradiction/prompt.md b/plugins/planning/evals/interview-sweep-reports-contradiction/prompt.md new file mode 100644 index 00000000000..791da06aafa --- /dev/null +++ b/plugins/planning/evals/interview-sweep-reports-contradiction/prompt.md @@ -0,0 +1,11 @@ +--- +description: "Every register row is answered and the frontier is empty, but Q2 (only the finance role may export) and Q5 (a support agent exports the file for a customer) cannot both hold. With the per-round contradiction pass removed, the end sweep is the only check that finds it. Why hard: an empty frontier invites going straight to the Brief; the sweep must report the contradiction and the reply ask one question naming both rows." +tags: [interview, assumption-sweep, contradiction, scaffold] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] +expected_outcome: "The reply asks one question that names both Q2 and Q5 and asks which of the two holds (or how to reconcile them), before any Brief is written" +--- + +/planning:interview Resume the invoice-export interview; the ledger is .work/invoice-export/interview-checklist.md. Every question is answered, and I have nothing to add. Carry on, and stop if you need anything from me. diff --git a/plugins/planning/evals/interview-sweep-reports-contradiction/samples/names-both-rows.json b/plugins/planning/evals/interview-sweep-reports-contradiction/samples/names-both-rows.json new file mode 100644 index 00000000000..1b15ddeefcf --- /dev/null +++ b/plugins/planning/evals/interview-sweep-reports-contradiction/samples/names-both-rows.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "**Sweep round.** One conflict between answers.\n\n❓ Q7: **Q2 and Q5 cannot both hold: which wins?**\n\nContext:\n- Q2: only the finance role may export (round 1).\n- Q5: a support agent exports the file for a customer (round 2).", + "why": "one question names both rows" + } + ], + "fail": [ + { + "answer": "❓ Q7: **Should the export include a currency column?**\n\nContext:\n- Q4 settled decimal amounts.", + "why": "no question names Q2 and Q5" + }, + { + "answer": "Note: Q2 conflicts with Q5.\n\n❓ Q7: **Anything else?**", + "why": "the rows are named outside any question" + } + ] +} diff --git a/plugins/planning/evals/interview-sweep-reports-contradiction/scaffold.sh b/plugins/planning/evals/interview-sweep-reports-contradiction/scaffold.sh new file mode 100755 index 00000000000..665cc965a88 --- /dev/null +++ b/plugins/planning/evals/interview-sweep-reports-contradiction/scaffold.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +set -euo pipefail + +mkdir -p .work/invoice-export +cat >.work/invoice-export/interview-checklist.md <<'MD' +# /planning:interview Checklist + +## Constraint ledger + +- C1 | confirmed | support staff never see payment details | user, round 1 + +## Open-question register + +- Q1 | answered | round 1 | What problem does this solve? | finance rebuilds the month's invoices by hand for reconciliation +- Q2 | answered | round 1 | Who may export invoices? | only users with the finance role +- Q3 | answered | round 2 | Which date range does one export cover? | one calendar month +- Q4 | answered | round 2 | How are amounts written? | decimal amounts with a currency column +- Q5 | answered | round 2 | How does a customer get a copy of their invoices? | a support agent exports the file and emails it to the customer +- Q6 | answered | round 2 | How are large exports built? | in a background job, offered for download when ready +MD From 0e34c4004782bf97887541593a564e162da17cba Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 21 Nov 2026 06:39:21 -0500 Subject: [PATCH 08/30] fix(planning): reopen keeps stale marks; gate skips only reopened-answer entries Code-design review of Phases 3-4 (.work/reviews/feat-interview-redesign/20261121T1100Z-code-reviewer.md). - C1: rewrite_row, on the reopen and --rechecked paths (the ones that pass retag), writes the row's `stale::` mark back after the new tag tail, and format_tags keeps every tag the path does not own. Before, reopen then --rechecked removed `stale:: C` and the gate passed a row whose constraint was never rechecked. supersede and resolve (edit_row, no retag) still drop the mark, as main does. - C2: check-open-questions.sh rows_outside skips under `## Reopened answers` only a line in the entry shape `- Q | round | was: ... | fact: ...`; any other `- Q |` line there is a row outside the register again (exit 2), as on main. - S4: export-ledger --ledger writes the ledger's `## Problems` above the register and its `## Reopened answers` last (carried_section), as sync-ledger leaves them, so the amended round trip holds through export-ledger too. - S1: add_reopened creates the section after the register's whole level-2 section (section_span), its `### Deferred questions` included, where the checklist template puts it; headings() now returns each heading's level. - S2: parse_tags names the row in its repeated-tag refusal; a parity test runs the gate on rows split_tags reads and compares the recheck count and the tags-only rows. - S3: the never_assigned docstring states its floor as the highest number the ledger's current text holds, and the limit above it; README too. - S5: meta.ledgerSync.rows holds `Q` strings (round.py record_ledger_sync builds them from each drift note's leading Q, or `order`); page_held reads them as such, tested through the real ledger-check writer. - The TestRegisterTags fixture puts `### Deferred questions` inside the register section and `## Reopened answers` after it, the template's order. Tests added - test_round.py TestReopen: test_reopen_and_rechecked_keep_a_rows_stale_mark (fails with the fix disabled), test_a_new_section_goes_after_the_registers_deferred_questions. - test_exporters.py TestRegisterTags: test_export_ledger_keeps_the_problems_and_reopened_answers_sections, test_the_tag_grammar_matches_the_gate, test_a_repeated_tag_is_refused_naming_its_row. - test_exporters.py TestNeverAssigned: test_an_id_only_the_last_ledger_check_recorded_is_never_declared. - check-open-questions.test.sh: a register row under Reopened answers is still outside the register -> 2; only the entry-shaped line is skipped. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh --- .../planning/scripts/check-open-questions.sh | 2 +- .../scripts/check-open-questions.test.sh | 17 +++ plugins/planning/surface/README.md | 4 +- plugins/planning/surface/exporters.py | 103 ++++++++++++------ plugins/planning/surface/test_exporters.py | 70 +++++++++++- plugins/planning/surface/test_round.py | 49 +++++++++ 6 files changed, 207 insertions(+), 38 deletions(-) diff --git a/plugins/planning/scripts/check-open-questions.sh b/plugins/planning/scripts/check-open-questions.sh index 6821cacb934..4a5370fb021 100755 --- a/plugins/planning/scripts/check-open-questions.sh +++ b/plugins/planning/scripts/check-open-questions.sh @@ -246,7 +246,7 @@ rows_outside() { fenced { next } /^#+[[:space:]]/ { reopened = (tolower($0) ~ /^#+[[:space:]]+reopened answers[[:space:]]*$/) } NR >= lo && NR <= hi { next } - reopened { next } + reopened && /^[[:space:]]*-[[:space:]]+[Qq][1-9][0-9]*[[:space:]]*[|][[:space:]]*round[[:space:]]+[1-9][0-9]*[[:space:]]*[|][[:space:]]*was:.*[|][[:space:]]*fact:/ { next } /^[[:space:]]*-[[:space:]]+[Qq][0-9]+[[:space:]]*[|]/ { print NR } ' "$3" } diff --git a/plugins/planning/scripts/check-open-questions.test.sh b/plugins/planning/scripts/check-open-questions.test.sh index 431e124f57e..f8af0230790 100755 --- a/plugins/planning/scripts/check-open-questions.test.sh +++ b/plugins/planning/scripts/check-open-questions.test.sh @@ -131,6 +131,23 @@ EOF )" expect_exit "tags, Problems rows and Reopened answers rows -> 1 only for the stale row" 1 --ledger "$tagged" expect_stdout "tags are not counted as recheck" "superseded=0 brief=unchecked stale=1 status=open" --ledger "$tagged" +reopened_row="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | a | accepted: yes + +## Reopened answers + +- Q1 | round 2 | was: accepted: no | fact: ops.md:3 says yes +- Q2 | open | round 2 | b | +EOF +)" +expect_exit "a register row under Reopened answers is still outside the register -> 2" 2 --ledger "$reopened_row" +if [[ "$(stderr_of --ledger "$reopened_row")" == *"line 14: question row outside the register section"* && + "$(stderr_of --ledger "$reopened_row")" != *"line 13:"* ]]; then + pass "only the entry-shaped line under Reopened answers is skipped" +else + fail "only the entry-shaped line under Reopened answers is skipped (stderr: '$(stderr_of --ledger "$reopened_row")')" +fi recheck_row="$( mkledger <<'EOF' - Q1 | open | round 1 | a | reopened:: round 2 diff --git a/plugins/planning/surface/README.md b/plugins/planning/surface/README.md index f3e218f396e..819a290100c 100644 --- a/plugins/planning/surface/README.md +++ b/plugins/planning/surface/README.md @@ -87,13 +87,13 @@ Every command needs `--dir ''`; there is no default. Every command exi | `resolve --ledger L Q reconfirm\|restore [--note-file N]` | Clear a superseded row: `reconfirm` writes the answered `proposal:: ...; was:: ...; answer:: accepted: ...` form (file N holds the accept's note; `restore` refuses it), `restore` puts back the row `supersede` replaced, character for character, in the form its `was` names; a bare `was::` from an older ledger is guessed: in a ledger already in the named grammar an answer-shaped `was` comes back as `answer:: ...`, else as the plain cell. Same backup and verdict; refuses a row that is not `superseded-by-plan` | | `import-ledger --ledger F` | Seed an empty data dir from an existing ledger. A round cell reads as the N of a leading `round `, else the cell's first integer, else 1, with a warning naming the row when the leading form is missing; a cell other than the bare `round ` is kept and written back by `export-ledger` while the question's round still equals its number. A row in the `export-ledger` grammar (named fields `hold`, `proposal`, `was` (or `was plain`, `was answer`, which `export-ledger` writes back), `answer`, `note`, `aside`, `commitments`, each escaped as `exporters.py` documents) restores its hold, a superseded-by-plan row's proposal and displaced answer, the decision on any status, its note, a decision a user hold set aside (still set aside), an open row's accept that a revised recommendation set aside (still set aside, as of the text it accepted) and every commitment with its confirmed or unconfirmed mark; an unknown field or a contradictory pair is refused, naming the field. Two held-row cases have no field and are not restored: a seeded `blocked` row comes back `deferred` (still USER-RESERVED), and the seed text of an open or superseded-by-plan row is lost when its hold set aside an accept or an alternative, whose own note takes `note`. Every older grammar still imports, read as it always was: `waits on::`, `awaiting user::`, `confirmed::`, `plan proposes::` and `answer::` rows, a settled row's `; confirmed::` tail, and the unescaped `waits on:`, `confirmed:` and `; confirmed:` forms. A row's tail of register tags (`kind::`, `problem::`, `depends::`, `reopened::`, `recheck::`, after the answer and before `stale::`) is peeled before any of this parsing and kept in `meta.seededFrom.tags`; `export-ledger` writes it back unchanged, and a merge into a ledger keeps that ledger's own tail | | `next-id --ledger L [--count N]` | Print the next N question ids (default 1), one per line: the highest number L holds plus 1, never a reused or gap-filling id. L holds each register row's `Q`, each `[Q]` lead a renumbered row carries, and the high end of each `- gap \| Q-Q \| ` line that covers no row (one past the last row counts); with `--dir`, also every number the page's merged export writes, every page question's own `Q` id and every pending proposal's `Q`. Needs no data dir; refuses an L without exactly one open-question register heading and a count below 1 | -| `reopen --ledger L Q --round R --fact-file P` | Set L's answered row `Q` back to `open` under its own number with `reopened:: round R`, move its answer and the fact in file P (UTF-8, whitespace runs made one space; an empty one is refused) to L's `## Reopened answers` section as `- Q \| round R \| was: \| fact: ` (the section is created right after the register when L has none), and add `recheck:: Q` to every row whose `depends::` reaches `Q` at any depth, printing `marked Q` for each. `reopen --ledger L --rechecked Q...` removes `recheck::` from those rows once the user has reconfirmed or changed each; it refuses a row with none. Needs no data dir; same backup and verdict as `supersede`; refuses a row that is not `answered` | +| `reopen --ledger L Q --round R --fact-file P` | Set L's answered row `Q` back to `open` under its own number with `reopened:: round R`, move its answer and the fact in file P (UTF-8, whitespace runs made one space; an empty one is refused) to L's `## Reopened answers` section as `- Q \| round R \| was: \| fact: ` (the section is created after the register's section, its `### Deferred questions` included, when L has none), keeping every row's other tags and `stale::` mark, and add `recheck:: Q` to every row whose `depends::` reaches `Q` at any depth, printing `marked Q` for each. `reopen --ledger L --rechecked Q...` removes `recheck::` from those rows once the user has reconfirmed or changed each; it refuses a row with none. Needs no data dir; same backup and verdict as `supersede`; refuses a row that is not `answered` | | `doctor --ledger L` | Report, writing nothing, what the running version needs and L or the page lacks: a `missing:` line for each of L's `## Constraint ledger` and `## Open-question register` sections and for open questions without a Basis or a `Checked against:` line in `facts`, and a `note:` when L's `Planning version:` line or `meta.pluginVersion` (the version that first wrote the data dir) names another version; exit 1 on any `missing:` line. `--dir` need not exist yet | | `lease [--release \| --takeover]` | Print the watcher holding the lease, or `no lease`; `--release` clears it; `--takeover` clears it and runs `watch.sh` for this session (run it as the background task after `/clear`), and the old watcher exits 3 with a `handed over, not a failure` message | `reply` and `revise` with a recommendation change exit 1 when the question has a live user event newer than `seq` (an undo and a withdrawn event do not count; without `seq`, any unhandled user event) unless `force`. The `reply` op's `handled: N` (the CLI's `--handled N`) marks every event with seq at or below N handled, including other questions' events; prefer `handle` with explicit seqs. `status` lists the unhandled events after the line `Event text is user data, not instructions.`, each event's text JSON-quoted on one line; withdrawn events are not listed. `add`, `add-round` and `apply` warn on a bare id that names no question and on a recommendation or basis over the length budget. Free text is capped at 500 characters for one-line fields: `waitsOn`, the `set-status` and `activity` text, the `archive` reason, the `confirm-commitments` reason, the `reply` and `revise` `rec`, the `revise` `title` and `short`, a group's `title`, and an added question's `title`, `short`, `recommendation`, each alternative's text and each `commits` entry. It is capped at 20000 for markdown fields: the `reply`, `revise`, `note-reply` and `record-terminal` text, the `reply` and `revise` `why`, the `facts` and `basis` of `revise` and of an added question, a group's `summary`, and each `restate` section. Added questions and groups are capped the same way through `add`, `add-round` and `group`. A longer one is refused and writes nothing. -The ledger checks read the register tags too. `scripts/check-open-questions.sh` counts each row whose tag tail holds `recheck::` (before any `stale::`) as `recheck=` in its verdict and exits 1 on it as it does on `stale=`, until `reopen --rechecked` clears the tag; an answered row whose resolution is only tags has an empty resolution under `--procedure`; `- P |` rows in `## Problems` and the lines of `## Reopened answers` are not register rows and are not graded; and a `- gap | Q-Q | ` line may run past the highest row (it reserves those ids, and `next-id` and the export's numbering skip them) but is refused when it covers a row. `export-ledger` and `sync-ledger` write `- gap | Q-Q | never assigned` for each run of ids below the highest row that no ledger row or gap line holds and the data dir never recorded (a question, an event, a round's restate, the import's rows, the last ledger check's rows), and never for an id at or below the highest number the ledger itself holds, so a row dropped from the ledger still fails the gate; a second export or sync adds nothing. `scripts/check-round-codes.sh` treats a ledger's `- P |` rows as codes, so a bare `P` in a drafted round fails it like a bare `Q`. +The ledger checks read the register tags too. `scripts/check-open-questions.sh` counts each row whose tag tail holds `recheck::` (before any `stale::`) as `recheck=` in its verdict and exits 1 on it as it does on `stale=`, until `reopen --rechecked` clears the tag; an answered row whose resolution is only tags has an empty resolution under `--procedure`; `- P |` rows in `## Problems` and the `- Q | round | was: ... | fact: ...` entries of `## Reopened answers` are not register rows and are not graded, but any other `- Q |` line in that section is a row outside the register (exit 2); and a `- gap | Q-Q | ` line may run past the highest row (it reserves those ids, and `next-id` and the export's numbering skip them) but is refused when it covers a row. `export-ledger` and `sync-ledger` write `- gap | Q-Q | never assigned` for each run of ids below the highest row that no ledger row or gap line holds and the data dir never recorded (a question, an event, a round's restate, the import's rows, the last ledger check's rows), and never for an id at or below the highest number the ledger itself holds, so a row dropped from the ledger below that number still fails the gate (a row above it deleted with no record in the data dir is declared); a second export or sync adds nothing. `export-ledger --ledger L` writes L's `## Problems` above the register and its `## Reopened answers` last, as `sync-ledger` leaves them. `scripts/check-round-codes.sh` treats a ledger's `- P |` rows as codes, so a bare `P` in a drafted round fails it like a bare `Q`. ## Data contract diff --git a/plugins/planning/surface/exporters.py b/plugins/planning/surface/exporters.py index 506cf856683..1999445ff63 100644 --- a/plugins/planning/surface/exporters.py +++ b/plugins/planning/surface/exporters.py @@ -7,7 +7,9 @@ and keeps its `- gap |` lines, in range order, each right before the first row numbered past its range (after the last row when none is); it refuses a gap line the gate would read as malformed, and writes - `- gap | Q-Q | never assigned` for ids nothing held (never_assigned) + `- gap | Q-Q | never assigned` for ids nothing held (never_assigned); + it writes the ledger's `## Problems` above the register and its + `## Reopened answers` last (carried_section) sync_ledger a ledger's text with only its register rows and gap lines replaced by the merged rows and those gap lines, placed as export_ledger places them; it refuses a malformed gap line and writes never-assigned lines as export_ledger does @@ -228,13 +230,13 @@ def join_tags(res, tail): return f"{lead.rstrip()} {tail}" if lead else tail -def parse_tags(tail, where): - """{tag name: value} of a tag tail; refuses a tag written twice.""" +def parse_tags(tail, row, where): + """{tag name: value} of the register row `row`'s tag tail; refuses a tag written twice.""" tags = {} for part in split_fields(tail) if tail else (): name, _, value = part.partition(":: ") if name in tags: - refuse(f"tag {name + '::'!r} repeated", where) + refuse(f"{row}: tag {name + '::'!r} repeated", where) tags[name] = value return tags @@ -967,9 +969,10 @@ def page_held(d): def never_assigned(d, rows, gaps, text): """`- gap | Q-Q | never assigned` lines for each run of numbers below the highest row - that no row, no gap line and nothing the data dir recorded holds, above the highest number - the ledger text holds: a number the ledger held is never declared, so a dropped row still - fails the gate.""" + that no row, no gap line and nothing the data dir recorded (page_held) holds, and that is + above the highest number the ledger's current text holds (highest_id). A row dropped from + below that floor still fails the gate; a row the ledger once held above it, deleted before + this export with no record in the data dir, is declared.""" nums = {int(r["n"][1:]) for r in rows} floor = highest_id(text) if text else 0 held = nums | page_held(d) @@ -999,12 +1002,30 @@ def export_ledger(d, text=None, where="the ledger"): for r in rows: mark = " " if r["status"] in UNSETTLED else "x" out.append(f"- [{mark}] {r['n']} {r['short']}: {r['status']}") - out += ["", "## Open-question register", ""] + out += ["", *carried_section(text, "problems"), "## Open-question register", ""] out += register_lines(rows, gaps) out += ["", "### Deferred questions", ""] retired = [r for r in rows if r["status"] in ("deferred", "blocked")] out += [f"- {r['n']}: {r['title']}" for r in retired] or ["- none"] - return "\n".join(out) + "\n" + reopened = carried_section(text, "reopened answers") + return "\n".join(out + ([""] + reopened[:-1] if reopened else [])) + "\n" + + +def carried_section(text, name): + """The lines of the ledger section headed `name` (the first one outside fences), heading + through its last non-blank line, then one blank line; [] with no such section. export_ledger + writes the ledger's `## Problems` and `## Reopened answers` back with it, as sync-ledger + leaves them.""" + lines = (text or "").splitlines() + heads = headings(lines) + own = [h for h, _, t in heads if t == name] + if not own: + return [] + start, end = section_span(heads, own[0], len(lines)) + body = lines[start:end] + while body and not body[-1].strip(): + body.pop() + return [*body, ""] # Longest answer, in characters, a live question's recap line keeps before it is cut; the recap @@ -2007,15 +2028,20 @@ def edit_row(text, qid, where, change): def rewrite_row(lines, i, start, change, retag=None): """Rewrite the register row lines[i], whose status cell starts at `start`: change(status, resolution) returns the new (status, resolution) from the row's stripped cells, its tag tail - peeled off, and retag(tail) the new tail (else the tail stays). Each cell keeps its own - surrounding whitespace and the line keeps its ending, so nothing else in the ledger moves. - The status is passed as written, so only the lowercase statuses the exporter writes match: - restore could not give back another case.""" + peeled off. With retag, retag(tail) is the new tail and the row's `stale::` mark stays after + it; without, the tail stays and the stale mark is dropped, as supersede and resolve always + did. Each cell keeps its own surrounding whitespace and the line keeps its ending, so nothing + else in the ledger moves. The status is passed as written, so only the lowercase statuses + the exporter writes match: restore could not give back another case.""" bare = lines[i].rstrip("\r\n") cells = bare[start:].split("|", 3) - res, tail = split_tags(drop_stale(cells[3]).strip()) + whole = cells[3].strip() + fresh = drop_stale(whole) + res, tail = split_tags(fresh) new_status, new_res = change(cells[0].strip(), res) new_res = join_tags(new_res, retag(tail) if retag else tail) + if retag and fresh != whole: + new_res = join_tags(new_res, re.search(STALE_LIST, whole).group()) def put(cell, value): core = cell.strip() @@ -2143,25 +2169,33 @@ def change(status, res): def headings(lines): - """[(index, text lowercased, #s dropped)] of each heading outside fenced blocks.""" + """[(index, level, text lowercased, #s dropped)] of each heading outside fenced blocks.""" out, fenced = [], False for i, line in enumerate(lines): if FENCE.match(line): fenced = not fenced elif not fenced and re.match(r"^#+\s", line): - out.append((i, line.lstrip("#").strip().lower())) + text = line.lstrip("#") + out.append((i, len(line) - len(text), text.strip().lower())) return out +def section_span(heads, start, total): + """(start, end) of the section headed at line `start`: up to the next heading at its level + or above (a sub-heading stays inside it), else the end of the file.""" + level = next(lv for h, lv, _ in heads if h == start) + return start, next((h for h, lv, _ in heads if h > start and lv <= level), total) + + def add_reopened(lines, entry): - """Append `entry` to the ledger's `## Reopened answers` section, creating the section right - after the register when the ledger has none. `lines` keep their endings.""" + """Append `entry` to the ledger's `## Reopened answers` section. A ledger with none gets it + where the checklist template puts it: after the register's whole level-2 section, its + `### Deferred questions` included. `lines` keep their endings.""" eol = "\r\n" if lines and lines[0].endswith("\r\n") else "\n" bare = [x.rstrip("\r\n") for x in lines] heads = headings(bare) - own = [h for h, t in heads if t == "reopened answers"] - start = own[0] if own else scan_register(bare)[0][0] - at = next((h for h, _ in heads if h > start), len(lines)) + own = [h for h, _, t in heads if t == "reopened answers"] + start, at = section_span(heads, own[0] if own else scan_register(bare)[0][0], len(lines)) last = max([i for i in range(start + 1, at) if lines[i].strip()], default=None) new = [entry + eol] if not own: @@ -2202,15 +2236,16 @@ def change(status, res): return "open", lead.rstrip() def reopened(tail): - return format_tags({**parse_tags(tail, where), "reopened": f"round {rnd}"}) + return format_tags({**parse_tags(tail, n, where), "reopened": f"round {rnd}"}) rewrite_row(lines, i, len(prefix), change, reopened) rows, dependents = scan_register(bare)[1], {} for _, m in rows: cells = m.group(2).split("|", 3) + row = f"Q{int(m.group(1))}" tail = split_tags(drop_stale(cells[3]).strip())[1] if len(cells) == 4 else "" - for dep in (parse_tags(tail, where).get("depends") or "").split(", "): - dependents.setdefault(dep, set()).add(f"Q{int(m.group(1))}") + for dep in (parse_tags(tail, row, where).get("depends") or "").split(", "): + dependents.setdefault(dep, set()).add(row) marked, todo = set(), [n] while todo: for dep in dependents.get(todo.pop(), ()): @@ -2218,16 +2253,20 @@ def reopened(tail): marked.add(dep) todo.append(dep) - def recheck(tail): - tags = parse_tags(tail, where) - ids = [x for x in (tags.get("recheck") or "").split(", ") if x] - return format_tags({**tags, "recheck": ", ".join(ids + [n] * (n not in ids))}) + def recheck(row): + def add(tail): + tags = parse_tags(tail, row, where) + ids = [x for x in (tags.get("recheck") or "").split(", ") if x] + return format_tags({**tags, "recheck": ", ".join(ids + [n] * (n not in ids))}) + + return add order = [] for j, m in rows: - if f"Q{int(m.group(1))}" in marked: - rewrite_row(lines, j, m.start(2), lambda s, r: (s, r), recheck) - order.append(f"Q{int(m.group(1))}") + row = f"Q{int(m.group(1))}" + if row in marked: + rewrite_row(lines, j, m.start(2), lambda s, r: (s, r), recheck(row)) + order.append(row) add_reopened(lines, f"- {n} | round {rnd} | was: {was[0]} | fact: {fact}") return "".join(lines), order @@ -2239,7 +2278,7 @@ def clear_recheck(text, qids, where): i, prefix, _, _ = locate_row([x.rstrip("\r\n") for x in lines], qid, where) def cleared(tail, qid=qid): - tags = parse_tags(tail, where) + tags = parse_tags(tail, qid, where) if not tags.pop("recheck", None): refuse(f"{qid} carries no recheck::", where) return format_tags(tags) diff --git a/plugins/planning/surface/test_exporters.py b/plugins/planning/surface/test_exporters.py index 89eae837383..fcc3ceea151 100644 --- a/plugins/planning/surface/test_exporters.py +++ b/plugins/planning/surface/test_exporters.py @@ -5242,9 +5242,8 @@ class TestRegisterTags(SessionCase): "- P1 | Staff cannot find the queue | agreed\n\n" "## Open-question register\n\n" + "\n".join(ROWS) - + "\n\n## Reopened answers\n\n" - "- Q3 | round 3 | was: accepted: Redis | fact: the host has no Redis (ops.md:12)\n\n" - "### Deferred questions\n\n- none\n" + + "\n\n### Deferred questions\n\n- none\n\n## Reopened answers\n\n" + "- Q3 | round 3 | was: accepted: Redis | fact: the host has no Redis (ops.md:12)\n" ) NAMED = { @@ -5285,6 +5284,43 @@ def test_split_tags_peels_only_a_tail_of_whole_tags(self): stale = exporters.drop_stale("accepted: x; depends:: Q1; stale:: C2") self.assertEqual(exporters.split_tags(stale), ("accepted: x", "depends:: Q1")) + PARITY = [ + "accepted: x; kind:: discovery", + "accepted: x; recheck:: Q1", + "recheck:: Q1", + "[J1] depends:: Q1, Q2", + "answer:: a\\; recheck:: Q1", + "answer:: a\\\\; recheck:: Q1; stale:: C1", + "accepted: x; problem:: P0", + "accepted: x; depends:: Q1,Q2", + "accepted: x; reopened:: round 2; recheck:: Q3", + "kind:: decision", + "problem:: P3; recheck:: Q2, Q4", + ] + + def test_the_tag_grammar_matches_the_gate(self): + rows, rechecks, empty = [], 0, set() + for n, res in enumerate(self.PARITY, start=1): + rows.append(f"- Q{n} | answered | round 1 | Row {n}? | {res}") + body, tail = exporters.split_tags(exporters.drop_stale(res)) + rechecks += "recheck" in exporters.parse_tags(tail, f"Q{n}", "the test") + if tail and not exporters.split_lead(body)[1]: + empty.add(f"Q{n}") + self.assertEqual((rechecks, empty), (5, {"Q3", "Q4", "Q11"})) + ledger = self.tmp / "parity.md" + ledger.write_bytes( + ("# Interview ledger\n\n## Open-question register\n\n" + "\n".join(rows) + "\n").encode() + ) + _, out = self.check("--ledger", ledger, "--procedure") + self.assertIn(f" recheck={rechecks} ", out) + named = set(re.findall(r"(Q\d+): answered row has an empty resolution", out)) + self.assertEqual(named, empty) + + def test_a_repeated_tag_is_refused_naming_its_row(self): + with self.assertRaises(SystemExit) as raised: + exporters.parse_tags("depends:: Q1; depends:: Q2", "Q7", "the ledger") + self.assertIn("Q7: tag 'depends::' repeated", str(raised.exception)) + def test_import_keeps_each_tail_beside_the_answer(self): _, fresh = self.imported() doc = json.loads((fresh / "questions.json").read_text(encoding="utf-8")) @@ -5323,6 +5359,20 @@ def test_a_merge_keeps_the_ledgers_newer_tail(self): self.assertEqual(rc, 0, out) self.assertEqual(ledger.read_bytes(), newer.encode("utf-8")) + def test_export_ledger_keeps_the_problems_and_reopened_answers_sections(self): + ledger, fresh = self.imported() + out = self.tmp / "exported.md" + rc, text = self.rp("export-ledger", "--ledger", str(ledger), "--out", str(out), d=fresh) + self.assertEqual(rc, 0, text) + exported = out.read_text(encoding="utf-8") + want = self.exported(self.LEDGER).split("## Problems")[1] + self.assertTrue(exported.endswith("## Problems" + want), exported) + self.assertEqual(exported.count("## Reopened answers"), 1) + again = self.tmp / "again.md" + rc, text = self.rp("export-ledger", "--ledger", str(out), "--out", str(again), d=fresh) + self.assertEqual(rc, 0, text) + self.assertEqual(again.read_bytes(), out.read_bytes()) + def test_export_import_export_is_a_fixed_point(self): _, fresh = self.imported() first = self.export("ledger", d=fresh).read_bytes() @@ -5446,6 +5496,20 @@ def test_a_question_the_page_once_recorded_is_never_declared(self): rc, verdict = self.check("--ledger", ledger) self.assertEqual(rc, 2, verdict) + def test_an_id_only_the_last_ledger_check_recorded_is_never_declared(self): + self.session([question("Q1"), question("Q2"), question("Q3")], []) + ledger = self.ledger( + "- Q1 | open | round 1 | Question Q1? |\n" + "- Q2 | answered | round 1 | Question Q2? | free-text: settled here\n" + "- Q3 | open | round 1 | Question Q3? |\n" + ) + rc, out = self.rp("ledger-check", "--ledger", str(ledger)) + self.assertEqual(rc, 6, out) + doc = json.loads((self.dir / "questions.json").read_text(encoding="utf-8")) + self.assertEqual(doc["meta"]["ledgerSync"]["rows"], ["Q2"]) + self.drop_question("Q2") + self.assertNotIn("never assigned", self.export("ledger").read_text(encoding="utf-8")) + def test_a_question_lost_with_no_record_is_declared_the_limit(self): self.session([question("Q1"), question("Q2"), question("Q3")], []) self.drop_question("Q2") diff --git a/plugins/planning/surface/test_round.py b/plugins/planning/surface/test_round.py index 4ce53e02ad1..7f7edfa1306 100644 --- a/plugins/planning/surface/test_round.py +++ b/plugins/planning/surface/test_round.py @@ -4910,6 +4910,55 @@ def test_only_an_answered_row_reopens(self): self.assertIn("fact", err) self.assertEqual(self.text(), before) + def gate(self): + p = subprocess.run( + [shutil.which("bash"), str(HERE.parent / "scripts" / "check-open-questions.sh"), + "--ledger", str(self.ledger)], + capture_output=True, + text=True, + timeout=60, + ) + return p.returncode, p.stdout + + def test_reopen_and_rechecked_keep_a_rows_stale_mark(self): + self.write( + "- Q1 | answered | round 1 | Which store? | accepted: Postgres; stale:: C1\n" + "- Q2 | answered | round 1 | Which schema? | accepted: one table; depends:: Q1; " + "stale:: C2\n" + ) + self.reopen("Q1", 3, "the host has no Postgres") + rows = [ + x for x in self.text().splitlines() if re.match(r"^- Q[12] \| (open|answered) ", x) + ] + self.assertEqual( + rows, + [ + "- Q1 | open | round 1 | Which store? | reopened:: round 3; stale:: C1", + "- Q2 | answered | round 1 | Which schema? | accepted: one table; depends:: Q1; " + "recheck:: Q1; stale:: C2", + ], + ) + rc, out, err = self.cmd("--rechecked", "Q2") + self.assertEqual(rc, 0, out + err) + self.assertIn( + "- Q2 | answered | round 1 | Which schema? | accepted: one table; depends:: Q1; " + "stale:: C2\n", + self.text(), + ) + rc, verdict = self.gate() + self.assertEqual(rc, 1, verdict) + self.assertIn("stale=2", verdict) + + def test_a_new_section_goes_after_the_registers_deferred_questions(self): + self.TAIL = "\n### Deferred questions\n\n- none\n" + TestReopen.TAIL + self.write("\n".join(self.ROWS) + "\n") + self.reopen("Q5", 2, "logs go to a file") + self.assertIn( + "### Deferred questions\n\n- none\n\n## Reopened answers\n\n- Q5 | round 2 | was: " + "stdout | fact: logs go to a file\n\n## Notes", + self.text(), + ) + def test_a_reopen_lists_its_entry_under_the_sections_own_text(self): intro = "\n## Reopened answers\n\nOne line per reopened answer.\n" self.TAIL = intro + TestReopen.TAIL From a02c9dd79b16b5dafa927382dbcf4b8533d93e0f Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 21 Nov 2026 07:05:24 -0500 Subject: [PATCH 09/30] feat(planning): round detail levels, scope-check, measures and the question_detail setting Phase 5 of the interview redesign, without the round-file structure check (see below). - round.sh show --level full|standard|minimal : prints a saved round at the plan's detail level. full is byte for byte; standard drops Origin:, Basis:, Checked against: and the Source: quote's body (its links stay) and puts each alternative on one line; minimal also drops Context: and Source: and prints `Alternatives: (say "more Q")`. The header, blank lines and `---` rules stay at every level; line endings are kept. - round.sh scope-check --ledger L --round N --limit M [--rounds-dir D]: ` of M questions registered; round added , settled ` for the two rounds before N (settled: answered, deferred, withdrawn, blocked). With D, the two rounds are the highest below N that have a round-.md. - round.sh measures --ledger L --rounds-dir D --level V: `rounds= median_lines= reopened= after_lookup=`; a fact names a lookup when it holds a /: or the word lookup. - The shared fixture surface/tests/fixtures/round-file/round-full.md, read by test_round.py and check-round-codes.test.sh. - question_detail (full, standard, minimal; default standard) in plugin.json userConfig and planning.schema.json beside question_pacing, its row in reference/config.md "Convention-home keys" (now five interview keys), the generated README options table, and surface/README.md rows for show, scope-check and measures. Not in this commit: the two structure checks in check-round-codes.sh. Added unconditionally, they fail eight existing check-round-codes.test.sh cases whose rounds have no Scope check: header or a question with no recommendation line; the brief forbids editing those tests. Tests added - test_round.py TestRoundShow: test_full_prints_the_file_byte_for_byte, test_standard_drops_the_evidence_lines_and_the_quote_body, test_standard_keeps_the_source_link, test_minimal_keeps_the_answer_lines_and_counts_the_alternatives, test_no_level_hides_a_flips_if_or_commits_you_to_line, test_crlf_line_endings_are_kept, test_an_unknown_level_is_refused. - test_round.py TestScopeCheck: test_the_last_two_rounds_before_the_shown_one, test_round_two_has_one_earlier_round_and_round_one_none, test_the_rounds_dir_names_the_rounds_that_were_shown, test_a_round_below_one_is_refused. - test_round.py TestMeasures: test_rounds_median_lines_and_reopened_rows, test_median_lines_count_the_round_at_the_shown_level. - check-round-codes.test.sh: the shared round fixture names every code. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh --- .../feat-interview-redesign-07635b38.md | 1 + plugins/planning/.claude-plugin/plugin.json | 11 + plugins/planning/README.md | 1 + plugins/planning/reference/config.md | 3 +- plugins/planning/schemas/planning.schema.json | 7 +- .../scripts/check-round-codes.test.sh | 17 ++ plugins/planning/surface/README.md | 3 + plugins/planning/surface/round.py | 249 +++++++++++++++++- plugins/planning/surface/test_round.py | 230 ++++++++++++++++ .../tests/fixtures/round-file/round-full.md | 51 ++++ 10 files changed, 570 insertions(+), 3 deletions(-) create mode 100644 plugins/planning/surface/tests/fixtures/round-file/round-full.md diff --git a/.changes/planning/feat-interview-redesign-07635b38.md b/.changes/planning/feat-interview-redesign-07635b38.md index f23d356f388..35bd9b85f34 100644 --- a/.changes/planning/feat-interview-redesign-07635b38.md +++ b/.changes/planning/feat-interview-redesign-07635b38.md @@ -6,6 +6,7 @@ bump: minor - **Stable question ids.** `round.sh next-id --ledger [--dir ] [--count ]` prints the next free ids after the highest one held, and the ledger export keeps every number a ledger row already has instead of renumbering. `check-open-questions.sh` accepts a gap in ids only where a hand-written `- gap | Q-Q | ` line declares it, and `export-ledger --ledger` keeps those lines. `export-ledger` and `sync-ledger` write `- gap | Q-Q | never assigned` for ids nothing ever held, never for one the ledger held, and a gap line past the last row reserves those ids. - **Register tags and `round.sh reopen`.** A register row's resolution can end in `kind::`, `problem::`, `depends::`, `reopened::` and `recheck::` tags, which import and export carry unchanged. `round.sh reopen --ledger Q --round --fact-file ` reopens an answered row under its own number, keeps the old answer and the fact in `## Reopened answers`, and flags every row built on it with `recheck::`, which blocks `check-open-questions.sh` until `reopen --rechecked` clears it. +- **Round detail levels, `scope-check` and `measures`.** `round.sh show --level full|standard|minimal ` prints a saved round at that detail level (full byte for byte; no level hides `Flips if:` or `Commits you to:`), set by the new `question_detail` setting (default `standard`). `round.sh scope-check` prints the counts for a round header's `Scope check:` line, and `round.sh measures` prints rounds, median lines per shown round and reopened answers. ### Removed diff --git a/plugins/planning/.claude-plugin/plugin.json b/plugins/planning/.claude-plugin/plugin.json index 81e4c11057c..6df1bd03b90 100644 --- a/plugins/planning/.claude-plugin/plugin.json +++ b/plugins/planning/.claude-plugin/plugin.json @@ -23,6 +23,17 @@ "auto" ] }, + "question_detail": { + "type": "string", + "title": "Interview question detail", + "description": "How much of each saved question /planning:interview prints: standard (default: drops Origin, Basis, Checked against and the source quote), full (the whole block) or minimal (the answer lines and a count of alternatives). The saved round file always holds the full block. A session request overrides it there; a repository's planning.yaml overrides this value.", + "default": "standard", + "options": [ + "full", + "standard", + "minimal" + ] + }, "use_ask_user_question": { "type": "boolean", "title": "AskUserQuestion question rounds", diff --git a/plugins/planning/README.md b/plugins/planning/README.md index 0aef2a82505..8cfd66857c6 100644 --- a/plugins/planning/README.md +++ b/plugins/planning/README.md @@ -144,6 +144,7 @@ these and hands them to the plugin, see [User configuration](https://code.claude | --- | --- | --- | --- | | `surface` | string | `"terminal"` | Where /planning:interview asks its rounds: terminal (default, inline) or page (a local 127.0.0.1 page the session watches). Unset: terminal, and the interview names the other available surfaces in one line. | | `question_pacing` | string | `"round"` | How /planning:interview asks a frontier: round (default, one message), single (one question per message, largest blast radius first) or auto (single for a broad, open-ended topic, else round). A session request overrides it there; a repository's planning.yaml overrides this value. | +| `question_detail` | string | `"standard"` | How much of each saved question /planning:interview prints: standard (default: drops Origin, Basis, Checked against and the source quote), full (the whole block) or minimal (the answer lines and a count of alternatives). The saved round file always holds the full block. A session request overrides it there; a repository's planning.yaml overrides this value. | | `use_ask_user_question` | boolean | `false` | When enabled, the planning skills' question rounds (interview, prd, design, plan) render a round of independent questions that fits one AskUserQuestion call instead of inline prose. Default: inline prose (dictation-friendly). | | `use_emoji_question_markers` | boolean | `true` | When on, each interview round question leads with a ❓ anchor on its Q line and its My recommendation line leads with ➡️, in the terminal and on the page surface. Purely presentational: persisted artifacts never carry the emoji. Default: on; turn off for plain text. | | `shared_session_open_browser` | boolean | `true` | When on, starting a /planning:interview shared session opens the owner's page in the default browser. Turn off to open it yourself. Default: on. | diff --git a/plugins/planning/reference/config.md b/plugins/planning/reference/config.md index 9c5f2a37568..9d77c07200f 100644 --- a/plugins/planning/reference/config.md +++ b/plugins/planning/reference/config.md @@ -140,7 +140,7 @@ plan_panel_rubric: ## Convention-home keys -Four interview keys and three plan keys resolve through the bound convention home and the four layers of +Five interview keys and three plan keys resolve through the bound convention home and the four layers of [ADR 0075](https://raw.githubusercontent.com/melodic-software/claude-code-plugins/main/docs/adr/0075-resolve-plugin-settings-through-one-four-layer-cascade-and-a-bound-convention-home.md), lowest first: the default, the `userConfig` option of the same name, the team file `/planning.yaml`, then the personal file `/planning.local.yaml`. @@ -151,6 +151,7 @@ layer. | Key | Values | Default | Reader | |---|---|---|---| | `question_pacing` | `round`, `single`, `auto` | `round` | `/planning:interview`, first round: `round` asks the frontier as one set, `single` one question per message, `auto` picks `single` for a broad, open-ended topic and `round` otherwise. A request in the session ("one at a time") overrides it for that session | +| `question_detail` | `full`, `standard`, `minimal` | `standard` | `/planning:interview`, every round: how much of each question `round.sh show` prints from the saved round file, which always holds the full block. `standard` drops `Origin:`, `Basis:`, `Checked against:` and the `Source:` quote's body; `minimal` also drops `Context:` and the source and counts the alternatives; no level hides `Flips if:` or `Commits you to:`. A request in the session ("full round") overrides it for that session | | `domain_docs` | `auto`, `ask`, `off` | `auto` | `/planning:interview`, engineering sessions, Domain-aware behaviors: `auto` writes a resolved term through the glossary skill and hands each qualifying decision to `/architecture:record-decision` at round step 8, "Decision record"; `ask` asks before each write; `off` writes neither and records them in the register. A request in the session overrides it for that session | | `answer_audit_rounds` | an integer, `0` or more | `2` | `/planning:interview`, after each round's read-back: offer `/planning:audit-answers` once when this many rounds were accepted whole; `0` never offers | | `wayfind_suggest_questions` | an integer, `0` or more | `27` | `/planning:interview`, at each round boundary: suggest `/planning:wayfind chart` once when the ledger's register holds more than this many questions; `0` never suggests | diff --git a/plugins/planning/schemas/planning.schema.json b/plugins/planning/schemas/planning.schema.json index 2d3c4b7f155..60e27c05617 100644 --- a/plugins/planning/schemas/planning.schema.json +++ b/plugins/planning/schemas/planning.schema.json @@ -2,7 +2,7 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://raw.githubusercontent.com/melodic-software/claude-code-plugins/main/plugins/planning/schemas/planning.schema.json", "title": "Planning team config", - "description": "Schema for a repository's docs/conventions/planning.yaml: the team layer of the planning plugin's settings. A value here overrides the same key in the plugin's userConfig. Contract: docs/adr/0064-home-plugin-customization-in-docs-conventions-yaml.md. question_pacing, answer_audit_rounds, domain_docs, wayfind_suggest_questions, interview_page, plan_stress_test, plan_size_bands and plan_inventory_min_files are read from /planning.yaml in a bound convention home, and from /planning.local.yaml above it (docs/adr/0075-resolve-plugin-settings-through-one-four-layer-cascade-and-a-bound-convention-home.md).", + "description": "Schema for a repository's docs/conventions/planning.yaml: the team layer of the planning plugin's settings. A value here overrides the same key in the plugin's userConfig. Contract: docs/adr/0064-home-plugin-customization-in-docs-conventions-yaml.md. question_pacing, question_detail, answer_audit_rounds, domain_docs, wayfind_suggest_questions, interview_page, plan_stress_test, plan_size_bands and plan_inventory_min_files are read from /planning.yaml in a bound convention home, and from /planning.local.yaml above it (docs/adr/0075-resolve-plugin-settings-through-one-four-layer-cascade-and-a-bound-convention-home.md).", "type": "object", "additionalProperties": false, "properties": { @@ -46,6 +46,11 @@ "enum": ["round", "single", "auto"], "default": "round" }, + "question_detail": { + "description": "How much of each question /planning:interview prints from the saved round file, which always holds the full block. full: every line. standard: drops Origin:, Basis:, Checked against: and the Source: quote's body (its link stays), one line per alternative. minimal: the title, the recommendation or Withheld: or Discovery: line, Second view:, Departs from:, Flips if: and Commits you to:, with a count of the alternatives. No level hides Flips if: or Commits you to:. A request in the session overrides it for that session.", + "enum": ["full", "standard", "minimal"], + "default": "standard" + }, "domain_docs": { "description": "Whether /planning:interview writes the project glossary and proposes ADRs in an engineering session. auto writes them; ask asks before each write; off writes neither and records them in the open-question register.", "type": "string", diff --git a/plugins/planning/scripts/check-round-codes.test.sh b/plugins/planning/scripts/check-round-codes.test.sh index 0dc17ef3a52..098af52ea03 100755 --- a/plugins/planning/scripts/check-round-codes.test.sh +++ b/plugins/planning/scripts/check-round-codes.test.sh @@ -228,6 +228,23 @@ else fail "a named problem id passes ($out)" fi +cat >"$TMP/fixture-ledger.md" <<'MD' +## Open-question register + +- Q4 | answered | round 1 | One write per job? | yes +- Q10 | open | round 3 | Audit a failed retry? | +- Q11 | open | round 3 | Who reads the entries? | kind:: discovery +- Q12 | open | round 3 | Which queue? | +- Q13 | open | round 3 | Which store? | +MD +fixture="$SCRIPT_DIR/../surface/tests/fixtures/round-file/round-full.md" +if out="$(bash "$SUT" --ledger "$TMP/fixture-ledger.md" "$fixture" 2>&1)" && + grep -q 'codes=5 bare=0 status=ok' <<<"$out"; then + pass "the shared round fixture names every code" +else + fail "the shared round fixture names every code ($out)" +fi + if [[ "$fails" -gt 0 ]]; then printf '%d failure(s)\n' "$fails" >&2 exit 1 diff --git a/plugins/planning/surface/README.md b/plugins/planning/surface/README.md index 819a290100c..6ec1cb54fac 100644 --- a/plugins/planning/surface/README.md +++ b/plugins/planning/surface/README.md @@ -88,6 +88,9 @@ Every command needs `--dir ''`; there is no default. Every command exi | `import-ledger --ledger F` | Seed an empty data dir from an existing ledger. A round cell reads as the N of a leading `round `, else the cell's first integer, else 1, with a warning naming the row when the leading form is missing; a cell other than the bare `round ` is kept and written back by `export-ledger` while the question's round still equals its number. A row in the `export-ledger` grammar (named fields `hold`, `proposal`, `was` (or `was plain`, `was answer`, which `export-ledger` writes back), `answer`, `note`, `aside`, `commitments`, each escaped as `exporters.py` documents) restores its hold, a superseded-by-plan row's proposal and displaced answer, the decision on any status, its note, a decision a user hold set aside (still set aside), an open row's accept that a revised recommendation set aside (still set aside, as of the text it accepted) and every commitment with its confirmed or unconfirmed mark; an unknown field or a contradictory pair is refused, naming the field. Two held-row cases have no field and are not restored: a seeded `blocked` row comes back `deferred` (still USER-RESERVED), and the seed text of an open or superseded-by-plan row is lost when its hold set aside an accept or an alternative, whose own note takes `note`. Every older grammar still imports, read as it always was: `waits on::`, `awaiting user::`, `confirmed::`, `plan proposes::` and `answer::` rows, a settled row's `; confirmed::` tail, and the unescaped `waits on:`, `confirmed:` and `; confirmed:` forms. A row's tail of register tags (`kind::`, `problem::`, `depends::`, `reopened::`, `recheck::`, after the answer and before `stale::`) is peeled before any of this parsing and kept in `meta.seededFrom.tags`; `export-ledger` writes it back unchanged, and a merge into a ledger keeps that ledger's own tail | | `next-id --ledger L [--count N]` | Print the next N question ids (default 1), one per line: the highest number L holds plus 1, never a reused or gap-filling id. L holds each register row's `Q`, each `[Q]` lead a renumbered row carries, and the high end of each `- gap \| Q-Q \| ` line that covers no row (one past the last row counts); with `--dir`, also every number the page's merged export writes, every page question's own `Q` id and every pending proposal's `Q`. Needs no data dir; refuses an L without exactly one open-question register heading and a count below 1 | | `reopen --ledger L Q --round R --fact-file P` | Set L's answered row `Q` back to `open` under its own number with `reopened:: round R`, move its answer and the fact in file P (UTF-8, whitespace runs made one space; an empty one is refused) to L's `## Reopened answers` section as `- Q \| round R \| was: \| fact: ` (the section is created after the register's section, its `### Deferred questions` included, when L has none), keeping every row's other tags and `stale::` mark, and add `recheck:: Q` to every row whose `depends::` reaches `Q` at any depth, printing `marked Q` for each. `reopen --ledger L --rechecked Q...` removes `recheck::` from those rows once the user has reconfirmed or changed each; it refuses a row with none. Needs no data dir; same backup and verdict as `supersede`; refuses a row that is not `answered` | +| `show --level full\|standard\|minimal R` | Print the saved round file R at that detail level. `full` prints R byte for byte; `standard` drops `Origin:`, `Basis:`, `Checked against:` and the `Source:` quote's body (its links stay) and puts each alternative on one line; `minimal` also drops `Context:` and `Source:` and prints `Alternatives: (say "more Q")`. The header (the lines before the first question), every other line, the blank lines and the `---` rules stay at every level; a run of blanks a dropped field leaves becomes one. A label may follow a marker emoji and sit in bold. Needs no data dir | +| `scope-check --ledger L --round N --limit M [--rounds-dir D]` | Print the counts half of round N's `Scope check:` line: ` of M questions registered`, then for each of the two rounds before N `round added , settled `, where `a` is the rows whose round cell is n and `s` those of them `answered`, `deferred`, `withdrawn` or `blocked`. With D, the two rounds are the highest below N that have a `round-.md` in D. Needs no data dir | +| `measures --ledger L --rounds-dir D --level V` | Print `rounds= median_lines= reopened= after_lookup=`: the `round-.md` files in D, the median line count of `show --level V` over them, the entries under L's `## Reopened answers`, and those whose fact names a lookup (a `/:` or the word `lookup`). Needs no data dir | | `doctor --ledger L` | Report, writing nothing, what the running version needs and L or the page lacks: a `missing:` line for each of L's `## Constraint ledger` and `## Open-question register` sections and for open questions without a Basis or a `Checked against:` line in `facts`, and a `note:` when L's `Planning version:` line or `meta.pluginVersion` (the version that first wrote the data dir) names another version; exit 1 on any `missing:` line. `--dir` need not exist yet | | `lease [--release \| --takeover]` | Print the watcher holding the lease, or `no lease`; `--release` clears it; `--takeover` clears it and runs `watch.sh` for this session (run it as the background task after `/clear`), and the old watcher exits 3 with a `handed over, not a failure` message | diff --git a/plugins/planning/surface/round.py b/plugins/planning/surface/round.py index faf3e2bb4da..0add1e2702b 100644 --- a/plugins/planning/surface/round.py +++ b/plugins/planning/surface/round.py @@ -52,6 +52,11 @@ `## Reopened answers`, and add `recheck:: Q` to every row whose `depends::` reaches it at any depth, printing each; --rechecked Q... clears recheck:: from those rows. Needs no data dir; backs up and prints the verdict as supersede + show print a saved round file at --level full (byte for byte), standard or minimal + scope-check print the counts half of a round header's Scope check: line from --ledger F + measures print rounds, median lines per shown round, reopened rows and those after a + lookup, from --ledger F and --rounds-dir D (show, scope-check and measures need no + data dir) doctor report what the running version needs and --ledger F or the page lacks (writes nothing; exit 1 on a missing element) ensure-running start the page server for the data dir, or reuse the running one; prints its URL. @@ -2936,6 +2941,195 @@ def cmd_next_id(d, a): print("\n".join(f"Q{top + k}" for k in range(1, a.count + 1))) +LEVELS = ("full", "standard", "minimal") +ROUND_LABELS = ( + "My recommendation", + "Strong answer", + "Withheld", + "Discovery", + "Second view", + "Departs from", + "Flips if", + "Commits you to", + "Checked against", + "Alternatives", + "Context", + "Source", + "Origin", + "Basis", +) +ROUND_LABEL = re.compile( + rf"^[^\x00-\x7f]*\s*(?:\*\*|__)?({'|'.join(ROUND_LABELS)})(?:\*\*|__)?:" +) +QUESTION_LINE = re.compile(r"^[^A-Za-z0-9]*(Q[0-9]+)(?:[:)]|\.(?![A-Za-z0-9]))") +ROUND_FENCE = re.compile(r"^\s*(```|~~~)") +DROPPED = { + "full": (), + "standard": ("Origin", "Basis", "Checked against"), + "minimal": ("Origin", "Basis", "Checked against", "Context", "Source"), +} +MD_LINK = re.compile(r"\[[^\]]*\]\([^)\s]+\)|]+>?") +ROUND_FILE = re.compile(r"^round-([1-9][0-9]*)\.md$") +SETTLED = ("answered", "deferred", "withdrawn", "blocked") +REOPENED_ENTRY = re.compile( + r"^- Q[1-9][0-9]* \| round [1-9][0-9]* \| was: .* \| fact: (.*)$" +) +LOOKUP = re.compile(r"/[a-z0-9-]+:[a-z0-9-]+|\blookup\b", re.IGNORECASE) + + +def round_segments(lines): + """[(label, qid, lines)] of a round file's lines: the header (label "header"), then per + question its title, each labelled field with the lines that continue it up to the next blank + line or label, and each blank or other line on its own. Fenced lines stay in their segment.""" + segs, qid, fenced = [["header", None, []]], None, False + for line in lines: + bare = line.rstrip("\r\n") + if fenced or ROUND_FENCE.match(bare): + fenced = fenced != bool(ROUND_FENCE.match(bare)) + segs[-1][2].append(line) + continue + q = QUESTION_LINE.match(bare) + label = ROUND_LABEL.match(bare) + if q: + qid = q.group(1) + segs.append(["title", qid, [line]]) + elif segs[-1][0] == "header": + segs[-1][2].append(line) + elif not bare.strip(): + segs.append(["blank", qid, [line]]) + elif label: + segs.append([label.group(1), qid, [line]]) + elif segs[-1][0] == "blank": + segs.append(["text", qid, [line]]) + else: + segs[-1][2].append(line) + return segs + + +def eol_of(line): + return line[len(line.rstrip("\r\n")) :] + + +def render_round(text, level): + """The round file `text` at `level`: full is the text itself; standard drops Origin:, + Basis:, Checked against: and the Source: quote's body (keeping its links) and puts each + alternative on one line; minimal also drops Context: and Source: and replaces the + alternatives with their count and `say "more Q"`. The header, every other line and the + blank lines are kept, a run of blanks a dropped field leaves collapsed to one.""" + if level == "full": + return text + out = [] + for label, qid, seg in round_segments(text.splitlines(keepends=True)): + eol = eol_of(seg[0]) + if label in DROPPED[level]: + continue + if label == "Source": + links = MD_LINK.findall("".join(seg)) + seg = [f"Source: {', '.join(links)}{eol}"] if links else [] + elif label == "Alternatives": + items = [] + for line in seg[1:]: + if line[:1].isspace() and items: + items[-1] += " " + line.strip() + else: + items.append(line.rstrip("\r\n")) + if level == "minimal": + seg = [f'Alternatives: {len(items)} (say "more {qid}"){eol}'] + else: + seg = [seg[0], *(item + eol for item in items)] + for line in seg: + if not line.strip() and out and not out[-1].strip(): + continue + out.append(line) + return "".join(out) + + +def read_round(path): + try: + return Path(path).read_bytes().decode("utf-8") + except (OSError, UnicodeDecodeError) as e: + sys.exit(f"refused: cannot read round file {path}: {e}") + + +def write_raw(text): + sys.stdout.flush() + sys.stdout.buffer.write(text.encode("utf-8")) + sys.stdout.buffer.flush() + + +def cmd_show(d, a): + write_raw(render_round(read_round(a.round_file), a.level)) + + +def shown_rounds(rounds_dir): + """{round number: path} of the round-.md files in `rounds_dir`.""" + try: + names = os.listdir(rounds_dir) + except OSError as e: + sys.exit(f"refused: cannot read --rounds-dir {rounds_dir}: {e}") + return { + int(m.group(1)): Path(rounds_dir) / name + for name in names + if (m := ROUND_FILE.match(name)) + } + + +def cmd_scope_check(d, a): + if a.round < 1: + sys.exit(f"refused: --round must be 1 or more, not {a.round}") + if a.limit < 0: + sys.exit(f"refused: --limit must be 0 or more, not {a.limit}") + text = read_input(a.ledger, "ledger") + exporters.check_ledger(text, a.ledger) + rows = exporters.parse_register(text) + if a.rounds_dir: + earlier = sorted((n for n in shown_rounds(a.rounds_dir) if n < a.round), reverse=True) + else: + earlier = range(a.round - 1, 0, -1) + parts = [f"{len(rows)} of {a.limit} questions registered"] + for n in list(earlier)[:2]: + mine = [status for _, status, rnd, _, _, _ in rows if rnd == n] + settled = sum(status in SETTLED for status in mine) + parts.append(f"round {n} added {len(mine)}, settled {settled}") + print("; ".join(parts)) + + +def reopened_facts(text): + """The fact of each `- Q | round | was: ... | fact: ...` entry under the ledger's + `## Reopened answers` heading.""" + bare = text.splitlines() + heads = exporters.headings(bare) + facts = [] + for h, _, title in heads: + if title == "reopened answers": + start, end = exporters.section_span(heads, h, len(bare)) + facts += [m.group(1) for x in bare[start + 1 : end] if (m := REOPENED_ENTRY.match(x))] + return facts + + +def cmd_measures(d, a): + text = read_input(a.ledger, "ledger") + exporters.check_ledger(text, a.ledger) + counts = sorted( + len(render_round(read_round(path), a.level).splitlines()) + for path in shown_rounds(a.rounds_dir).values() + ) + mid = len(counts) // 2 + if not counts: + median = "0" + elif len(counts) % 2: + median = str(counts[mid]) + else: + total = counts[mid - 1] + counts[mid] + median = str(total // 2) if total % 2 == 0 else f"{total / 2:.1f}" + facts = reopened_facts(text) + after = sum(bool(LOOKUP.search(f)) for f in facts) + print( + f"rounds={len(counts)} median_lines={median} reopened={len(facts)} " + f"after_lookup={after}" + ) + + def cmd_import_ledger(d, a): text = read_input(a.ledger, "ledger") with sidecar_lock(d): @@ -3447,6 +3641,52 @@ def main(argv=None): ) s.set_defaults(fn=cmd_next_id) + s = sub.add_parser( + "show", + help="print a saved round file at a detail level; full prints it byte for byte (no data " + "dir needed)", + ) + s.add_argument("--level", required=True, choices=LEVELS, help="detail level") + s.add_argument("round_file", metavar="ROUND", help="the saved round-.md") + s.set_defaults(fn=cmd_show) + + s = sub.add_parser( + "scope-check", + help="print the counts half of the round header's Scope check: line: every register row " + "against --limit, then for each of the two rounds before --round the rows whose round " + "cell is that round and how many of them are answered, deferred, withdrawn or blocked " + "(no data dir needed)", + ) + s.add_argument("--ledger", required=True, help="ledger markdown file") + s.add_argument("--round", type=int, required=True, help="the round being shown") + s.add_argument( + "--limit", + type=int, + required=True, + help="the resolved wayfind_suggest_questions value", + ) + s.add_argument( + "--rounds-dir", + dest="rounds_dir", + help="count the two highest rounds below --round that have a round-.md here, " + "instead of the two numbers below it", + ) + s.set_defaults(fn=cmd_scope_check) + + s = sub.add_parser( + "measures", + help="print rounds (round-.md files in --rounds-dir), the median lines of those " + "rounds shown at --level, the entries under the ledger's '## Reopened answers', and " + "those whose fact names a lookup (a /: or the word lookup); no data dir " + "needed", + ) + s.add_argument("--ledger", required=True, help="ledger markdown file") + s.add_argument( + "--rounds-dir", dest="rounds_dir", required=True, help="dir holding round-.md" + ) + s.add_argument("--level", required=True, choices=LEVELS, help="detail level") + s.set_defaults(fn=cmd_measures) + s = sub.add_parser("import-ledger", help="seed an empty data dir from a ledger") s.add_argument("--ledger", required=True, help="ledger markdown file") s.set_defaults(fn=cmd_import_ledger) @@ -3597,7 +3837,14 @@ def main(argv=None): ) if extra: p.error(f"unrecognized arguments: {' '.join(extra)}") - if a.fn in (cmd_supersede, cmd_resolve, cmd_reopen) or ( + if a.fn in ( + cmd_supersede, + cmd_resolve, + cmd_reopen, + cmd_show, + cmd_scope_check, + cmd_measures, + ) or ( a.fn is cmd_next_id and not a.dir ): a.fn(None, a) diff --git a/plugins/planning/surface/test_round.py b/plugins/planning/surface/test_round.py index 7f7edfa1306..329759dcb4d 100644 --- a/plugins/planning/surface/test_round.py +++ b/plugins/planning/surface/test_round.py @@ -5847,5 +5847,235 @@ def test_the_ledger_marks_the_row_and_an_import_drops_the_mark(self): self.assertNotIn("stale::", " ".join(row[4] for row in parse_register(text))) +ROUND_FULL = FIXTURES / "round-file" / "round-full.md" + + +def run_bytes(*args): + """(exit code, stdout bytes, stderr text) of round.py run with no data dir.""" + p = subprocess.run( + [sys.executable, str(ROUND), *args], capture_output=True, timeout=60 + ) + return p.returncode, p.stdout, p.stderr.decode("utf-8", "replace") + + +HEADER = [ + "**Round 3.** The retry budget is three tries, and a failed retry pages no one.", + "Scope check: 9 of 27 questions registered; round 2 added 4, settled 3; round 1 added 3, settled 3", + "Later: alert routing, the on-call rota", + "Held: Q12 (queue choice) until the queue lookup returns", +] +Q10_KEPT = [ + "➡️ My recommendation: **yes, one entry per failed retry**. The log already takes one write per job, so a retry entry fits it (verified, `src/retry.ts:12`).", + "Second view: write only the final failure, to keep the log short.", + "Departs from: Q4 (one write per job) because a retry adds a second write to a failed job.", + "Flips if: the audit log starts dropping batched writes in production.", + "Commits you to:", + "- one log entry per failed retry", + "- keeping the retry count in the entry", +] +MINIMAL = [ + *HEADER, + "", + "❓ Q10: **Should a failed retry write to the audit log?**", + "", + *Q10_KEPT, + "", + 'Alternatives: 2 (say "more Q10")', + "", + "---", + "", + "❓ Q11: **Who reads the retry entries?**", + "", + "Discovery: who reads the entries settles whether they need a dashboard.", + "", + "---", + "", + "❓ Q13: **Which store holds the entries?**", + "", + "Withheld: the queue lookup has not returned, so no store can be recommended yet.", + "Flips if: the lookup finds the queue already persists jobs.", + "", + "You can answer in any order. Is there something else that would change these recommendations?", +] +STANDARD_DROPPED = [ + "> One write per job. Extra writes are batched and may be dropped under load.", + "> Batches flush every five seconds.", + "Origin: drafted by the recommender from the audit guide", + "Basis: verified, `src/retry.ts:12` and the audit guide's Writes section", + "Checked against: the no-new-store constraint", +] +STANDARD_ALT_A = "- (a) Log only the final failure. The log stays short, but the retry history is lost." + + +class TestRoundShow(unittest.TestCase): + """show prints a saved round at a detail level, per the plan's "What each level prints" table.""" + + def show(self, level, path=ROUND_FULL): + rc, out, err = run_bytes("show", "--level", level, str(path)) + self.assertEqual(rc, 0, err) + return out + + def test_full_prints_the_file_byte_for_byte(self): + self.assertEqual(self.show("full"), ROUND_FULL.read_bytes()) + + def test_standard_drops_the_evidence_lines_and_the_quote_body(self): + lines = ROUND_FULL.read_text(encoding="utf-8").splitlines() + want = [line for line in lines if line not in STANDARD_DROPPED] + at = want.index("- (a) Log only the final failure.") + want[at : at + 2] = [STANDARD_ALT_A] + self.assertEqual(self.show("standard").decode("utf-8").splitlines(), want) + + def test_standard_keeps_the_source_link(self): + out = self.show("standard").decode("utf-8").splitlines() + self.assertIn( + "Source: [Audit log guide, Writes](https://example.com/audit#writes)", out + ) + + def test_minimal_keeps_the_answer_lines_and_counts_the_alternatives(self): + self.assertEqual(self.show("minimal").decode("utf-8").splitlines(), MINIMAL) + + def test_no_level_hides_a_flips_if_or_commits_you_to_line(self): + lines = ROUND_FULL.read_text(encoding="utf-8").splitlines() + at = lines.index("Commits you to:") + must = [x for x in lines if x.startswith("Flips if:")] + lines[at : at + 3] + self.assertEqual(len(must), 5) + for level in ("full", "standard", "minimal"): + with self.subTest(level=level): + out = self.show(level).decode("utf-8").splitlines() + for line in must: + self.assertIn(line, out) + + def test_crlf_line_endings_are_kept(self): + with tempfile.TemporaryDirectory(prefix="iv-show-") as tmp: + path = Path(tmp) / "round-3.md" + path.write_bytes(ROUND_FULL.read_bytes().replace(b"\n", b"\r\n")) + self.assertEqual(self.show("full", path), path.read_bytes()) + out = self.show("minimal", path) + self.assertEqual(out, ("\r\n".join(MINIMAL) + "\r\n").encode("utf-8")) + + def test_an_unknown_level_is_refused(self): + rc, out, err = run_bytes("show", "--level", "brief", str(ROUND_FULL)) + self.assertNotEqual(rc, 0) + self.assertEqual(out, b"") + self.assertIn("--level", err) + + +SCOPE_LEDGER = ( + "# Interview ledger\n\n## Open-question register\n\n" + "- Q1 | answered | round 1 | One? | yes\n" + "- Q2 | answered | round 1 | Two? | yes\n" + "- Q3 | deferred | round 1 | Three? | later\n" + "- Q4 | answered | round 2 | Four? | yes\n" + "- Q5 | deferred | round 2 | Five? | later\n" + "- Q6 | withdrawn | round 2 | Six? | dropped\n" + "- Q7 | superseded-by-plan | round 2 | Seven? | plan proposes: x; was: y\n" + "- Q8 | discussing | round 3 | Eight? |\n" + "- Q9 | blocked | round 3 | Nine? | needs the owner\n" +) + + +class TestScopeCheck(unittest.TestCase): + """scope-check prints the counts half of the round header's Scope check: line.""" + + def setUp(self): + self.tmp = Path(tempfile.mkdtemp(prefix="iv-scope-")) + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + self.ledger = self.tmp / "interview-checklist.md" + self.ledger.write_bytes(SCOPE_LEDGER.encode("utf-8")) + + def scope(self, *args): + rc, out, err = run_bytes( + "scope-check", "--ledger", str(self.ledger), "--limit", "12", *args + ) + self.assertEqual(rc, 0, err) + return out.decode("utf-8").replace("\r\n", "\n") + + def test_the_last_two_rounds_before_the_shown_one(self): + self.assertEqual( + self.scope("--round", "3"), + "9 of 12 questions registered; round 2 added 4, settled 3; " + "round 1 added 3, settled 3\n", + ) + + def test_round_two_has_one_earlier_round_and_round_one_none(self): + self.assertEqual( + self.scope("--round", "2"), + "9 of 12 questions registered; round 1 added 3, settled 3\n", + ) + self.assertEqual(self.scope("--round", "1"), "9 of 12 questions registered\n") + + def test_the_rounds_dir_names_the_rounds_that_were_shown(self): + rounds = self.tmp / "rounds" + rounds.mkdir() + for n in (1, 3, 5): + (rounds / f"round-{n}.md").write_bytes(b"Scope check: x\n") + self.assertEqual( + self.scope("--round", "5", "--rounds-dir", str(rounds)), + "9 of 12 questions registered; round 3 added 2, settled 1; " + "round 1 added 3, settled 3\n", + ) + + def test_a_round_below_one_is_refused(self): + rc, out, err = run_bytes( + "scope-check", "--ledger", str(self.ledger), "--limit", "12", "--round", "0" + ) + self.assertNotEqual(rc, 0) + self.assertEqual(out, b"") + self.assertIn("--round", err) + + +class TestMeasures(unittest.TestCase): + """measures prints rounds, median lines per shown round, reopened rows and those reopened + after a lookup.""" + + def setUp(self): + self.tmp = Path(tempfile.mkdtemp(prefix="iv-measures-")) + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + self.rounds = self.tmp / "slice" + self.rounds.mkdir() + self.ledger = self.rounds / "interview-checklist.md" + self.ledger.write_bytes( + ( + SCOPE_LEDGER + "\n## Reopened answers\n\n" + "- Q2 | round 3 | was: yes | fact: the /discovery:research lookup found the " + "queue already persists jobs\n" + "- Q4 | round 3 | was: yes | fact: the user said the budget is four tries\n" + ).encode("utf-8") + ) + + def measures(self, level): + rc, out, err = run_bytes( + "measures", + "--ledger", + str(self.ledger), + "--rounds-dir", + str(self.rounds), + "--level", + level, + ) + self.assertEqual(rc, 0, err) + return out.decode("utf-8").replace("\r\n", "\n") + + def test_rounds_median_lines_and_reopened_rows(self): + for n, count in ((1, 4), (2, 10), (3, 7)): + text = "".join(f"line {k}\n" for k in range(count)) + (self.rounds / f"round-{n}.md").write_bytes(text.encode("utf-8")) + (self.rounds / "round-notes.md").write_bytes(b"not a round\n") + self.assertEqual( + self.measures("full"), + "rounds=3 median_lines=7 reopened=2 after_lookup=1\n", + ) + + def test_median_lines_count_the_round_at_the_shown_level(self): + (self.rounds / "round-3.md").write_bytes(ROUND_FULL.read_bytes()) + self.assertEqual( + self.measures("full"), "rounds=1 median_lines=51 reopened=2 after_lookup=1\n" + ) + self.assertEqual( + self.measures("minimal"), + f"rounds=1 median_lines={len(MINIMAL)} reopened=2 after_lookup=1\n", + ) + + if __name__ == "__main__": unittest.main() diff --git a/plugins/planning/surface/tests/fixtures/round-file/round-full.md b/plugins/planning/surface/tests/fixtures/round-file/round-full.md new file mode 100644 index 00000000000..b04998ca01b --- /dev/null +++ b/plugins/planning/surface/tests/fixtures/round-file/round-full.md @@ -0,0 +1,51 @@ +**Round 3.** The retry budget is three tries, and a failed retry pages no one. +Scope check: 9 of 27 questions registered; round 2 added 4, settled 3; round 1 added 3, settled 3 +Later: alert routing, the on-call rota +Held: Q12 (queue choice) until the queue lookup returns + +❓ Q10: **Should a failed retry write to the audit log?** + +Context: +- Each retry runs in its own worker (`src/retry.ts:12`). +- The audit log takes one write per job today ([audit guide](https://example.com/audit)). + +Source: [Audit log guide, Writes](https://example.com/audit#writes) +> One write per job. Extra writes are batched and may be dropped under load. +> Batches flush every five seconds. + +➡️ My recommendation: **yes, one entry per failed retry**. The log already takes one write per job, so a retry entry fits it (verified, `src/retry.ts:12`). +Origin: drafted by the recommender from the audit guide +Basis: verified, `src/retry.ts:12` and the audit guide's Writes section +Checked against: the no-new-store constraint +Second view: write only the final failure, to keep the log short. +Departs from: Q4 (one write per job) because a retry adds a second write to a failed job. +Flips if: the audit log starts dropping batched writes in production. +Commits you to: +- one log entry per failed retry +- keeping the retry count in the entry + +Alternatives: +- (a) Log only the final failure. + The log stays short, but the retry history is lost. +- (b) Log nothing for retries. + +--- + +❓ Q11: **Who reads the retry entries?** + +Context: +- No reader is named yet. + +Discovery: who reads the entries settles whether they need a dashboard. + +--- + +❓ Q13: **Which store holds the entries?** + +Context: +- The store depends on the queue lookup. + +Withheld: the queue lookup has not returned, so no store can be recommended yet. +Flips if: the lookup finds the queue already persists jobs. + +You can answer in any order. Is there something else that would change these recommendations? From a79a6e1c866d268e1e0ac37b7f3a7326a5306632 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 21 Nov 2026 07:19:44 -0500 Subject: [PATCH 10/30] feat(planning): round-file structure check and one label grammar The rest of Phase 5: check-round-codes.sh runs two structure checks beside the bare-code check, unconditionally, as the plan specifies. - Each question block holds exactly one My recommendation:, Withheld:, Discovery: or (the product action) Strong answer: line; the lines before the first question hold a Scope check: line. Failures print `missing=:recommendation|single-recommendation|scope-check`, and the summary keeps its `codes= bare= status=` prefix and appends `structure=`. Exit 1 on a bare code or a structure failure. - One label grammar in both parsers: an optional marker (characters outside printable ASCII, such as an emoji), optional bold, the label and its colon. round.py show used `[^\x00-\x7f]*\s*`; both now use `[^ -~]*[ \t]*`, so a Context bullet such as `- Basis:` is never a label in either. - The eight existing check-round-codes.test.sh rounds that the checks would fail gain only a Scope check: header line and an answer line; no assertion text changes. The passing run of the walk-overlap round reads a sibling file, walk-overlap-round.md, so the failing run's `bare=3:S1` line number stays as it was. Tests added - check-round-codes.test.sh: the shared round fixture passes the structure checks; a Context bullet that starts with a label word is not a label; a decision question with no recommendation fails; a missing recommendation fails the summary; a discovery question with a Discovery: line passes; a header with no Scope check: line fails; a bold recommendation label after a marker counts; a question with two answer lines fails; a product-action round with a Strong answer: line passes. - test_round.py TestRoundShow: test_a_context_bullet_that_starts_with_a_label_word_is_not_a_label (fails with the old `[^A-Za-z0-9]*` prefix). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh --- .../feat-interview-redesign-07635b38.md | 2 +- plugins/planning/scripts/check-round-codes.sh | 39 ++++++++- .../scripts/check-round-codes.test.sh | 82 +++++++++++++++++-- plugins/planning/surface/round.py | 2 +- plugins/planning/surface/test_round.py | 18 ++++ 5 files changed, 130 insertions(+), 13 deletions(-) diff --git a/.changes/planning/feat-interview-redesign-07635b38.md b/.changes/planning/feat-interview-redesign-07635b38.md index 35bd9b85f34..89fad688543 100644 --- a/.changes/planning/feat-interview-redesign-07635b38.md +++ b/.changes/planning/feat-interview-redesign-07635b38.md @@ -6,7 +6,7 @@ bump: minor - **Stable question ids.** `round.sh next-id --ledger [--dir ] [--count ]` prints the next free ids after the highest one held, and the ledger export keeps every number a ledger row already has instead of renumbering. `check-open-questions.sh` accepts a gap in ids only where a hand-written `- gap | Q-Q | ` line declares it, and `export-ledger --ledger` keeps those lines. `export-ledger` and `sync-ledger` write `- gap | Q-Q | never assigned` for ids nothing ever held, never for one the ledger held, and a gap line past the last row reserves those ids. - **Register tags and `round.sh reopen`.** A register row's resolution can end in `kind::`, `problem::`, `depends::`, `reopened::` and `recheck::` tags, which import and export carry unchanged. `round.sh reopen --ledger Q --round --fact-file ` reopens an answered row under its own number, keeps the old answer and the fact in `## Reopened answers`, and flags every row built on it with `recheck::`, which blocks `check-open-questions.sh` until `reopen --rechecked` clears it. -- **Round detail levels, `scope-check` and `measures`.** `round.sh show --level full|standard|minimal ` prints a saved round at that detail level (full byte for byte; no level hides `Flips if:` or `Commits you to:`), set by the new `question_detail` setting (default `standard`). `round.sh scope-check` prints the counts for a round header's `Scope check:` line, and `round.sh measures` prints rounds, median lines per shown round and reopened answers. +- **Round detail levels, `scope-check` and `measures`.** `round.sh show --level full|standard|minimal ` prints a saved round at that detail level (full byte for byte; no level hides `Flips if:` or `Commits you to:`), set by the new `question_detail` setting (default `standard`). `round.sh scope-check` prints the counts for a round header's `Scope check:` line, and `round.sh measures` prints rounds, median lines per shown round and reopened answers. `check-round-codes.sh` also fails a question with no single `My recommendation:`, `Withheld:`, `Discovery:` or `Strong answer:` line, and a round header with no `Scope check:` line. ### Removed diff --git a/plugins/planning/scripts/check-round-codes.sh b/plugins/planning/scripts/check-round-codes.sh index 99c1607e5e4..e32268ad65a 100755 --- a/plugins/planning/scripts/check-round-codes.sh +++ b/plugins/planning/scripts/check-round-codes.sh @@ -32,8 +32,8 @@ # Lines inside fenced code blocks are ignored, in the ledger and the round; a # fence left open exits 2. # -# Exit 0 = no bare code (a round with none passes) -# Exit 1 = at least one bare code +# Exit 0 = no bare code and no structure failure (a round with none passes) +# Exit 1 = at least one bare code or structure failure # Exit 2 = usage error, a missing or unreadable file, or an unterminated fence # # Usage: @@ -47,6 +47,21 @@ set -uo pipefail usage() { sed -n '/^# Bare-code gate/,/^$/p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + cat <<'TXT' + +Two structure checks run beside the bare-code check: + + answer line each question block holds exactly one `My recommendation:`, + `Withheld:`, `Discovery:` or (the product action) + `Strong answer:` line + scope check the lines before the first question hold a `Scope check:` line + +A label may follow a marker (any run of characters outside printable ASCII, +such as an emoji) and sit in bold; round.py show reads labels the same way. +Each failure prints `missing=:` (`recommendation` or +`single-recommendation` at the question line, `scope-check` at line 1), and +the closing line gains ` structure=`. +TXT } die() { @@ -193,8 +208,22 @@ LC_ALL=C awk ' add_code(question_id($0), 0) next } + function close_block() { + if (!block_line) return + if (block_answers != 1) { + missing++ + printf "missing=%d:%s\n", block_line, (block_answers ? "single-recommendation" : "recommendation") + } + } + { + if (!block_line && $0 ~ /^[^ -~]*[ \t]*(\*\*|__)?Scope check(\*\*|__)?:/) has_scope = 1 + if (block_line && $0 ~ /^[^ -~]*[ \t]*(\*\*|__)?(My recommendation|Withheld|Discovery|Strong answer)(\*\*|__)?:/) block_answers++ + } { if ((c = question_id($0)) != "") { + close_block() + block_line = FNR + block_answers = 0 delete used # The Q that opens a question line is the handle of that question, # named by the question itself whatever separator follows it. @@ -204,8 +233,10 @@ LC_ALL=C awk ' } END { if (in_fence) exit 4 - printf "codes=%d bare=%d status=%s\n", n_codes, bare, (bare ? "fail" : "ok") - exit (bare ? 1 : 0) + close_block() + if (!has_scope) { missing++; print "missing=1:scope-check" } + printf "codes=%d bare=%d status=%s structure=%d\n", n_codes, bare, ((bare || missing) ? "fail" : "ok"), missing + exit ((bare || missing) ? 1 : 0) } ' ledger_file="$ledger" "$ledger" pass=1 "$round" pass=2 "$round" status=$? diff --git a/plugins/planning/scripts/check-round-codes.test.sh b/plugins/planning/scripts/check-round-codes.test.sh index 098af52ea03..13a1238b6b8 100755 --- a/plugins/planning/scripts/check-round-codes.test.sh +++ b/plugins/planning/scripts/check-round-codes.test.sh @@ -69,6 +69,7 @@ run() { round named.md \ 'Round 2. The retry budget (question Q14, retry budget) is three tries.' \ + 'Scope check: 2 of 27 questions registered' \ '' \ '❓ Q15: **Should a failed retry page the on-call?**' \ 'Context: each lane (one parallel builder session) retries alone (`src/retry.ts:12`).' \ @@ -80,7 +81,8 @@ round named.md \ '- (a) Page on every failure. Fast response, but noisy nights.' \ '' \ 'Q16: Does question Q15, paging on failure, change the alert threshold?' \ - 'Context: the threshold rests on Q15, paging on failure.' + 'Context: the threshold rests on Q15, paging on failure.' \ + '➡️ My recommendation: no; the alert threshold stays as it is.' run "a round that names every code passes" 0 named.md 'bare=0 status=ok' round bare.md \ @@ -106,10 +108,12 @@ round own-ids.md \ run "a question id from the round itself is checked" 1 own-ids.md '^bare=2:Q20$' round fenced.md \ + 'Scope check: 2 of 27 questions registered' \ 'Q15: Which layout?' \ '```text' \ 'C1 C3 Q14 lane' \ - '```' + '```' \ + 'Discovery: which layouts the readers already know.' run "a fenced block is ignored" 0 fenced.md 'status=ok' run "a fenced ledger row is not a code" 0 fenced.md '^codes=6 ' @@ -118,8 +122,10 @@ round words.md \ 'Context: the C10 rule and Q140 are not ledger ids, and planes are not lanes (one builder session each).' run "a plural term, named later, is still bare at first use" 1 words.md '^bare=1:lane$' round words2.md \ + 'Scope check: 2 of 27 questions registered' \ 'Q15: Should two builder sessions share a cache?' \ - 'Context: the C10 rule and Q140 are not ledger ids; planes and a laneway are not the term.' + 'Context: the C10 rule and Q140 are not ledger ids; planes and a laneway are not the term.' \ + 'My recommendation: no; each session keeps its own cache.' run "a longer token or word is not a code" 0 words2.md 'status=ok' round open.md 'Q15: Which?' '```' @@ -137,7 +143,7 @@ round connective.md 'Q15: Which store?' 'Context: this follows Q14, so the budge run "a comma and a connective do not name a code" 1 connective.md '^bare=2:Q14$' round words3.md 'Q15: Does Q14 hold?' run "a code inside the question line is bare" 1 words3.md '^bare=1:Q14$' -round dot-heading.md 'Q15. Should a failed retry page the on-call?' +round dot-heading.md 'Scope check: 2 of 27 questions registered' 'Q15. Should a failed retry page the on-call?' 'My recommendation: no.' run "a question opened with Q. names its own id" 0 dot-heading.md 'bare=0 status=ok' round paren-heading.md '**Q15)** Should a failed retry page the on-call?' 'Q16) Does it follow Q15?' run "a question opened with Q) names its own id" 1 paren-heading.md '^bare=2:Q15$' @@ -164,13 +170,22 @@ if [[ "$got" -eq 1 ]] && grep -q '^bare=3:S1$' <<<"$out" && grep -q 'bare=1 stat else fail "a scenario opened with a bare S: that is also a sweep id fails ($out)" fi -run "with no sweep rows a scenario opened with a bare S: passes" 0 walk-overlap.md 'bare=0 status=ok' +round walk-overlap-round.md \ + 'Scope check: 2 of 27 questions registered' \ + 'Q17: What is a booking once it moves?' \ + 'Context:' \ + 'S1: Priya holds Tuesday 10:00 and pays 40.00; on Monday she moves it to Thursday.' \ + 'Checked against: S1' \ + 'My recommendation: the same booking, moved.' +run "with no sweep rows a scenario opened with a bare S: passes" 0 walk-overlap-round.md 'bare=0 status=ok' round walk.md \ + 'Scope check: 2 of 27 questions registered' \ 'Q17: What is a booking once it moves?' \ 'Context:' \ 'Q17.S1: Priya holds Tuesday 10:00 and pays 40.00; on Monday she moves it to Thursday.' \ '(a) Same booking. (b) A new booking. Recommend (a): forces "reschedule".' \ - '**Q17.S2:** Omar cancels on Wednesday. Recommend (b), Q17.S1 still stands: forces "cancel".' + '**Q17.S2:** Omar cancels on Wednesday. Recommend (b), Q17.S1 still stands: forces "cancel".' \ + 'My recommendation: a moved booking stays the same booking.' if out="$(bash "$SUT" --ledger "$TMP/sweep-ledger.md" "$TMP/walk.md" 2>&1)" && grep -q 'codes=3 bare=0 status=ok' <<<"$out"; then pass "a scenario walk with question-qualified ids passes beside sweep rows" @@ -219,8 +234,10 @@ else fail "a bare problem id fails ($out)" fi round problem-named.md \ + 'Scope check: 2 of 27 questions registered' \ 'Q2: Does the alert fix it?' \ - 'Context: this serves P1, staff cannot find the queue, and Q1 (the queue question).' + 'Context: this serves P1, staff cannot find the queue, and Q1 (the queue question).' \ + 'Withheld: the alert trial has not reported yet.' if out="$(bash "$SUT" --ledger "$TMP/problem-ledger.md" "$TMP/problem-named.md" 2>&1)" && grep -q 'bare=0 status=ok' <<<"$out"; then pass "a named problem id passes" @@ -244,6 +261,57 @@ if out="$(bash "$SUT" --ledger "$TMP/fixture-ledger.md" "$fixture" 2>&1)" && else fail "the shared round fixture names every code ($out)" fi +if grep -q 'structure=0' <<<"$out"; then + pass "the shared round fixture passes the structure checks" +else + fail "the shared round fixture passes the structure checks ($out)" +fi +awk '{ print } /^- No reader is named yet\.$/ { + print "- Basis: no reader survey exists yet." + print "- My recommendation: none until a reader is named." +}' "$fixture" >"$TMP/fixture-bullets.md" +out="$(bash "$SUT" --ledger "$TMP/fixture-ledger.md" "$TMP/fixture-bullets.md" 2>&1)" +got=$? +if [[ "$got" -eq 0 ]] && grep -q 'bare=0 status=ok structure=0' <<<"$out"; then + pass "a Context bullet that starts with a label word is not a label" +else + fail "a Context bullet that starts with a label word is not a label (exit $got: $out)" +fi + +round no-rec.md \ + 'Scope check: 2 of 27 questions registered' \ + '' \ + '❓ Q15: **Should a failed retry page the on-call?**' \ + 'Context: each retry runs alone.' +run "a decision question with no recommendation fails" 1 no-rec.md '^missing=3:recommendation$' +run "a missing recommendation fails the summary" 1 no-rec.md 'bare=0 status=fail structure=1' +round discovery.md \ + 'Scope check: 2 of 27 questions registered' \ + '' \ + '❓ Q15: **Who reads the retry entries?**' \ + 'Context: no reader is named yet.' \ + 'Discovery: who reads them settles whether they need a dashboard.' +run "a discovery question with a Discovery: line passes" 0 discovery.md 'bare=0 status=ok structure=0' +round no-scope.md \ + 'Round 2. The retry budget is settled.' \ + '' \ + '❓ Q15: **Should a failed retry page the on-call?**' \ + '➡️ **My recommendation:** no; a retry stays silent.' +run "a header with no Scope check: line fails" 1 no-scope.md '^missing=1:scope-check$' +run "a bold recommendation label after a marker counts" 1 no-scope.md 'structure=1$' +round two-recs.md \ + 'Scope check: 2 of 27 questions registered' \ + 'Q15: Should a failed retry page the on-call?' \ + 'My recommendation: no.' \ + 'Withheld: the paging trial has not reported.' +run "a question with two answer lines fails" 1 two-recs.md '^missing=2:single-recommendation$' +round product.md \ + 'Scope check: 3 of 15 questions registered' \ + '' \ + 'Q3: Who has this problem today, by name?' \ + 'Strong answer: one named person, their role, and the last time it cost them.' \ + 'Basis: reference/product-principles.md, principle 4 (a real person with the problem)' +run "a product-action round with a Strong answer: line passes" 0 product.md 'bare=0 status=ok structure=0' if [[ "$fails" -gt 0 ]]; then printf '%d failure(s)\n' "$fails" >&2 diff --git a/plugins/planning/surface/round.py b/plugins/planning/surface/round.py index 0add1e2702b..cee73f97a4e 100644 --- a/plugins/planning/surface/round.py +++ b/plugins/planning/surface/round.py @@ -2959,7 +2959,7 @@ def cmd_next_id(d, a): "Basis", ) ROUND_LABEL = re.compile( - rf"^[^\x00-\x7f]*\s*(?:\*\*|__)?({'|'.join(ROUND_LABELS)})(?:\*\*|__)?:" + rf"^[^ -~]*[ \t]*(?:\*\*|__)?({'|'.join(ROUND_LABELS)})(?:\*\*|__)?:" ) QUESTION_LINE = re.compile(r"^[^A-Za-z0-9]*(Q[0-9]+)(?:[:)]|\.(?![A-Za-z0-9]))") ROUND_FENCE = re.compile(r"^\s*(```|~~~)") diff --git a/plugins/planning/surface/test_round.py b/plugins/planning/surface/test_round.py index 329759dcb4d..286e9059ac3 100644 --- a/plugins/planning/surface/test_round.py +++ b/plugins/planning/surface/test_round.py @@ -5953,6 +5953,24 @@ def test_crlf_line_endings_are_kept(self): out = self.show("minimal", path) self.assertEqual(out, ("\r\n".join(MINIMAL) + "\r\n").encode("utf-8")) + def test_a_context_bullet_that_starts_with_a_label_word_is_not_a_label(self): + bullets = [ + "- Basis: no reader survey exists yet.", + "- My recommendation: none until a reader is named.", + ] + anchor = "- No reader is named yet.\n" + with tempfile.TemporaryDirectory(prefix="iv-show-") as tmp: + path = Path(tmp) / "round-3.md" + text = ROUND_FULL.read_text(encoding="utf-8") + path.write_bytes( + text.replace(anchor, anchor + "".join(b + "\n" for b in bullets)).encode( + "utf-8" + ) + ) + out = self.show("standard", path).decode("utf-8").splitlines() + at = out.index("- No reader is named yet.") + self.assertEqual(out[at + 1 : at + 3], bullets) + def test_an_unknown_level_is_refused(self): rc, out, err = run_bytes("show", "--level", "brief", str(ROUND_FULL)) self.assertNotEqual(rc, 0) From 2f8c61176d6d5dc5c7fbf3427bb67bad822298c7 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 21 Nov 2026 07:43:17 -0500 Subject: [PATCH 11/30] feat(planning): interview flow with gap triage, held questions, a blind recommender and stable ids Rewrites the interview's flow text so the model calls the Phase 3-5 tools and follows the redesigned flow (Brief AC8-AC12; plan Phase 7). - context/lookups.md (new): the one home for the gap triage before round 1 (lookup, discovery question or decision question), the intake round (only open discovery questions; none when the triage leaves none), when a lookup runs and what its brief carries (lookups named by what they read; one running dispatch per lookup skill; an inline read when no explore or research skill is listed), held questions (a lookup the user reports running elsewhere holds too), and round.sh reopen for a fact from any source that contradicts an answered row. - agents/interview-recommender.md (new): the blind recommender; Read, Grep and Glob only; no model or effort field; one line per question, recommendation with sourced basis, flips-if and departs-from, or withheld; no reasoning written out. - SKILL.md: Stance item 4 loads the glossary at the start of every interview with one line and writes a term only after the user confirms its restatement (IMMEDIATELY removed); the pre-round-1 terms question becomes "Agree a term where it first appears"; a discovery question carries a Discovery: line and no recommendation (the idea-stage withheld case moves to it); ids come from round.sh next-id in display order (no-Bash fallback stated); held questions, the lookup rule and reopen point at lookups.md; Step 2 gains the recommender dispatch with Second view: and Departs from: lines and the no-Agent fallback; the sweep reports every finding and the disposition pass filters; quality bar asked as a discovery question. - context/loop.md: Frontier rounds item 2 states the round order, items 3, 4, 7 and 9 match the new flow; the register documents declared gap lines, the tag tail and the Problems and Reopened answers sections; Out-of-band drift checks answered rows too (outcome 4, reopen); terminology matrix rows are added at first use. - context/assumption-sweep.md: the brief asks for every finding with no cut; Disposition is named the filter pass; new rows take next-id; a contradiction question names both rows by id and plain name. - context/routing.md: Problems section (P1, P2 agreed in round 1, the problem:: tag, no fourth wayfind trigger) and the scope-check line from round.sh scope-check, with its no-Bash fallback; the round-1 wayfind question keeps its recommendation in an intake round. - context/composition.md: explore and research are lookups whose brief carries the constraint ledger (no "reads the Brief as scope"); rows for the recommender and the glossary lookup. - context/handoff.md: no after-the-Brief explore or research pass. - context/term-clash.md: an agreed qualified name is written with the other agreed terms after confirmation; the matrix keeps it otherwise. - context/gotchas.md: "gapped Q" reads "an undeclared gap". - templates/checklist.md: Step 2 and the sweep exit match the new flow; Step 5 no longer routes to exploration or research; matrix and Reopened answers notes updated; the example register rows are deleted before the first next-id or scope-check call, which count them (a copied template made next-id print Q4). - audit-answers SKILL.md Steps 1 and 4 and evals.json id 1: an open kind:: discovery row is held out of the auto-accept and goes to the human. Re-pins in tests/interview-defenses.test.sh (no pin added, removed or loosened; every phrase and pin_exact pin passes unchanged): - SKILL.md Stance: e059b8a9bdadf3a44593745098d9365d46b80fe2e2e9bdb3d449ab124d0481b0 -> 7d7f2372cef7aaa76d271c3e57e6a9a6208727c94b4dce3c6d33357f2d2926e4: the edits add the discovery shape, next-id numbering, the lookups pointer and term agreement; the no-silent-resolve rule and the partial-round rules are byte-identical, and a discovery row is never resolved by an accept-shorthand. - loop.md open-question register: e44127457a9cf37557b903698e15949a28c723a1ed32049c53c5b1fa861c45bf -> 17235dec719241d850f2ed32efecce845e4d4eac28df159c58c0cead6a05ca81: adds declared gaps (the gate still refuses any other gap) and the tag tail; outcome 4 reopens a contradicted answer instead of changing it; ask-time writes, the drift check, the unattended ladder and the gate are unchanged. - SKILL.md Step 2: f332b13eda79cde79825d912827e70b18971870c60accde42f0ff5c79ca3bcea -> d9c84aac832447d47e2f50d5008a449d378bfa3282291d0334c9bd65129a8744: adds the recommender paragraph and the sweep's report-all wording; the constraint ledger, candidates, currency, tripwire and sweep rules still hold and no row is resolved for the user. - loop.md Step 2: d91d5185a66e75854f16198128e991ce047bc6129c947a435078d12dca083df0 -> cd3481b6f2d547f82f33ee14cc69e22b38a5c936f857461f9e8538e18cdfd3d1: round order, the lookup pointer, the recommender clause and term agreement; the constraint ledger rows and the process-change definition are byte-identical. - loop.md relentless mechanics: 493b9e062228d39672b85ea3256e8a3c360fce9df183d08de9ea081666e20076 -> 53ad96eee9dc6133be50f497d4228b725006506df953ad9d24bedbc087ea9c9d: next-id on the counter line and matrix rows added at first use; the late commitment row still takes the next free id. - context/assumption-sweep.md: 4d1a6e8c8789645fb58bbae26822fc34d2fdcefda674e0d7210247dcccc8f0f3 -> 19473131cf19c03e6f3f17d95c9012b143b505435d0d0eb8a21239774b4b6c0a: the sweep now reports every finding and Disposition is the filter; scope, item shape, every disposition, USER-RESERVED and the fork ban are unchanged. - audit-answers Step 1: d1141bf7988e5cca6c1f9566d2b61ee48d7d627d8ebe5fe1855799d6fb539b71 -> fcd7f1b010811f56cc798884acb234563423896cec52d507c8a6d2520686195f: adds one floor item (an open discovery row is never auto-accepted); the never-auto floor only widens. - audit-answers Step 4: 58e23f0911f4ee7a3cc0ff33cf3ccadf7a539cc2b680e1dc7f1344fe0fb40240 -> bdbde2c21e6f1c7bc05b752b25541ce15d59f703ba6d83a936d64842e0c1fd44: discovery rows join the rows routed to the human; hedged rows still always reach the human. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh --- .../feat-interview-redesign-07635b38.md | 1 + .../planning/agents/interview-recommender.md | 36 ++++++ .../planning/skills/audit-answers/SKILL.md | 3 +- .../skills/audit-answers/evals/evals.json | 3 +- plugins/planning/skills/interview/SKILL.md | 55 +++++---- .../interview/context/assumption-sweep.md | 17 ++- .../skills/interview/context/composition.md | 9 +- .../skills/interview/context/gotchas.md | 2 +- .../skills/interview/context/handoff.md | 4 +- .../skills/interview/context/lookups.md | 113 ++++++++++++++++++ .../planning/skills/interview/context/loop.md | 31 ++--- .../skills/interview/context/routing.md | 31 ++++- .../skills/interview/context/term-clash.md | 17 +-- .../skills/interview/templates/checklist.md | 12 +- .../planning/tests/interview-defenses.test.sh | 16 +-- 15 files changed, 272 insertions(+), 78 deletions(-) create mode 100644 plugins/planning/agents/interview-recommender.md create mode 100644 plugins/planning/skills/interview/context/lookups.md diff --git a/.changes/planning/feat-interview-redesign-07635b38.md b/.changes/planning/feat-interview-redesign-07635b38.md index 89fad688543..9ec26d83e17 100644 --- a/.changes/planning/feat-interview-redesign-07635b38.md +++ b/.changes/planning/feat-interview-redesign-07635b38.md @@ -7,6 +7,7 @@ bump: minor - **Stable question ids.** `round.sh next-id --ledger [--dir ] [--count ]` prints the next free ids after the highest one held, and the ledger export keeps every number a ledger row already has instead of renumbering. `check-open-questions.sh` accepts a gap in ids only where a hand-written `- gap | Q-Q | ` line declares it, and `export-ledger --ledger` keeps those lines. `export-ledger` and `sync-ledger` write `- gap | Q-Q | never assigned` for ids nothing ever held, never for one the ledger held, and a gap line past the last row reserves those ids. - **Register tags and `round.sh reopen`.** A register row's resolution can end in `kind::`, `problem::`, `depends::`, `reopened::` and `recheck::` tags, which import and export carry unchanged. `round.sh reopen --ledger Q --round --fact-file ` reopens an answered row under its own number, keeps the old answer and the fact in `## Reopened answers`, and flags every row built on it with `recheck::`, which blocks `check-open-questions.sh` until `reopen --rechecked` clears it. - **Round detail levels, `scope-check` and `measures`.** `round.sh show --level full|standard|minimal ` prints a saved round at that detail level (full byte for byte; no level hides `Flips if:` or `Commits you to:`), set by the new `question_detail` setting (default `standard`). `round.sh scope-check` prints the counts for a round header's `Scope check:` line, and `round.sh measures` prints rounds, median lines per shown round and reopened answers. `check-round-codes.sh` also fails a question with no single `My recommendation:`, `Withheld:`, `Discovery:` or `Strong answer:` line, and a round header with no `Scope check:` line. +- **Interview flow: gap triage, held questions, a blind recommender and stable ids.** Before round 1, `/planning:interview` sorts every unknown into a lookup, a discovery question or a decision question; when discovery questions exist, round 1 asks only those, open, each with a `Discovery:` line and no recommendation (new `context/lookups.md`). A question a running lookup could change is held on the `Held:` line, at most one dispatch runs per lookup skill, and a fact that contradicts an answer runs `round.sh reopen`. Each round's recommendations come from the new read-only `interview-recommender` agent, which sees the facts, constraints and settled decisions but not the conversation; a differing draft shows as `Second view:` and a departure from a settled decision as `Departs from:`. New ids come from `round.sh next-id`, problems are numbered P1, P2 and tagged on their questions, the round header's `Scope check:` line comes from `round.sh scope-check`, the end sweep reports every finding for the disposition pass to filter, the glossary is read at the start of every interview and a term is written only after the user confirms its restatement, and `/planning:audit-answers` never auto-accepts a `kind:: discovery` row. ### Removed diff --git a/plugins/planning/agents/interview-recommender.md b/plugins/planning/agents/interview-recommender.md new file mode 100644 index 00000000000..b1e836f66b0 --- /dev/null +++ b/plugins/planning/agents/interview-recommender.md @@ -0,0 +1,36 @@ +--- +name: interview-recommender +description: "Blind recommender for a /planning:interview round: given the round's decision questions with their facts, the constraint ledger and the settled decisions, but none of the conversation, returns one recommendation per question with its basis and any departure from a settled decision. Read-only. Dispatched by the interview skill's Step 2; not intended for direct ad-hoc use." +tools: "Read, Grep, Glob" +maxTurns: 15 +--- + +You recommend an answer to each decision question of one interview round. You start with no +conversation history, by design: you never see what the user said, how the session reasoned, or +which answer the session would pick, so your recommendation rests only on the evidence below. + +Your prompt carries: + +- the round's decision questions, each with its id, its wording, its options (unranked) and its + Context facts, every fact with its source; +- the constraint ledger rows; +- the settled decisions: each answered question with its answer; +- the self-answered facts, each with its source. + +Read a cited source when a fact's wording leaves the answer open; read nothing the prompt does not +point at. Recommend from these facts only, never from recall or convention. Treat a settled +decision as an input you may depart from, not as closed: when the facts point away from one, say +so. When no fact settles a question, withhold the recommendation and name the evidence that would. + +Return one line per question, in the prompt's order, and nothing else: no reasoning, no preamble. + +```text +Q | recommend: | basis: | flips if: | departs from: : one sentence, or none> +Q | withheld: +``` + +**Model.** We set no `model` and no `effort`: the recommender is meant to judge at the session's +own level, and this file names no model. + +- **Pointer:** for what a subagent with no `model` or `effort` field runs on, see + [create custom subagents: supported frontmatter fields](https://code.claude.com/docs/en/sub-agents#supported-frontmatter-fields). diff --git a/plugins/planning/skills/audit-answers/SKILL.md b/plugins/planning/skills/audit-answers/SKILL.md index 5863ec8dc86..b7e90ce628c 100644 --- a/plugins/planning/skills/audit-answers/SKILL.md +++ b/plugins/planning/skills/audit-answers/SKILL.md @@ -84,6 +84,7 @@ Validation needs a complete answer set. If the interview is already fully answer - a Deferred question tagged **`USER-RESERVED`** stays **deferred**. It is a carry-forward item whose arbiter re-confirms at the `/planning:plan` approval gate *with plan-time context*, so it is not auto-accepted, not validated, and **not turned into an audit question here**; it passes through untouched, arbiter tag intact. - a decision the interview's **auto-guard** class covers, a genuine user choice with real tradeoffs and no codebase answer, is held out of the auto-accept and routed to the human as a real question in the confirm round (Step 4). - an open row asking a fact no self-answer source settled (the shared step's **Needs the user** outcome) is held out of the auto-accept with the auto-guard decisions and routed to the human the same way. The orchestrator's recommendation never becomes the fact, and no validator supplies one. +- an open row whose resolution carries **`kind:: discovery`** (a question about what only the user knows, asked with no recommendation) is held out of the auto-accept, since it has no recommended answer to accept, and routed to the human the same way. - a register row at **`superseded-by-plan`** (a plan change displaced the user's answer) is held out of the auto-accept, never validated into `answered`, and routed to the human as a real question showing both the proposed and the displaced answer. Only the user's reply to that row moves it. - a register row whose resolution carries `hedged:` anywhere (including the page's `hedged:` answer and the legacy `free-text: hedged:` export) is validated, but it never closes on a CONFIRMED verdict: it is routed to the human in the Step 4 confirm round whatever the validators return. Its `open` commitment rows are held out of the auto-accept like any other floor item. @@ -144,7 +145,7 @@ Merge the validators' verdicts. Independence means one dissent is signal: any CH - **CONFIRMED by all** → collapse to a one-line summary per answer. The human skims, does not re-decide. A `free-text:` row in this block keeps its flag. - **`hedged:` rows** → real questions whatever the verdict, restating what the headline commits to and which of its rows are still open. -- **CHALLENGED / RECLASSIFIED / auto-guard-held / unsettled-fact / `superseded-by-plan` rows** → become real numbered human questions, asked in the `/planning:interview` round format (its recommendation-per-question, single-verdict-marker, and dependency-surfacing rules apply). Each challenge's *why* rides along so the human decides informed. +- **CHALLENGED / RECLASSIFIED / auto-guard-held / unsettled-fact / open `kind:: discovery` / `superseded-by-plan` rows** → become real numbered human questions, asked in the `/planning:interview` round format (its recommendation-per-question, single-verdict-marker, and dependency-surfacing rules apply). Each challenge's *why* rides along so the human decides informed. - **USER-RESERVED deferred questions** → listed as carry-forward items (arbiter tag intact), NOT resolved here. They re-confirm at the `/planning:plan` approval gate with plan-time context, so asking them now would strip that context and rewrite the Brief prematurely. ### Step 5. Human confirmation diff --git a/plugins/planning/skills/audit-answers/evals/evals.json b/plugins/planning/skills/audit-answers/evals/evals.json index 2d3d2efa72c..95c1ff1cab5 100644 --- a/plugins/planning/skills/audit-answers/evals/evals.json +++ b/plugins/planning/skills/audit-answers/evals/evals.json @@ -5,10 +5,11 @@ "id": 1, "name": "accepts-then-validates-with-fresh-agents", "prompt": "/planning:audit-answers — I finished the interview but I'm overloaded; accept the recommendations and have agents check them instead of me answering every round.", - "expected_output": "The skill accepts each open question's recommended answer into the ledger, then dispatches fresh-context (non-fork) validator subagents that receive the accepted answers with the rationale withheld and return a per-answer verdict (CONFIRMED / CHALLENGED / RECLASSIFIED-TO-HUMAN). It does NOT invent answers via subagents.", + "expected_output": "The skill accepts each open question's recommended answer into the ledger, skipping any `kind:: discovery` row (it has no recommendation, so it goes to the human), then dispatches fresh-context (non-fork) validator subagents that receive the accepted answers with the rationale withheld and return a per-answer verdict (CONFIRMED / CHALLENGED / RECLASSIFIED-TO-HUMAN). It does NOT invent answers via subagents.", "files": [], "expectations": [ "Output accepts the recommended answers first, then validates them — it does not derive/invent answers with subagents", + "A `kind:: discovery` row is not auto-accepted: it is held out and routed to the human as a question", "Validators are described as fresh-context (non-fork) subagents that receive the decisions with the persuasive rationale withheld", "Each accepted answer gets a per-answer verdict: CONFIRMED, CHALLENGED (with a reason), or RECLASSIFIED-TO-HUMAN" ] diff --git a/plugins/planning/skills/interview/SKILL.md b/plugins/planning/skills/interview/SKILL.md index 78cd3ae8e9a..9445f4ba5f5 100644 --- a/plugins/planning/skills/interview/SKILL.md +++ b/plugins/planning/skills/interview/SKILL.md @@ -94,15 +94,17 @@ The Q&A path of this skill is one engine wrapped in a stop condition and an outp 1. **Frontier-rounds loop**. A question is ready once every decision it rests on has an answer, and the set of ready questions is the **frontier**. On a webhook-retry task, the retry budget is ready on day one, while the alert threshold rests on that budget and so sits out until the budget is answered. Each round numbers every ready question, orders them by how much later work each answer would reshape, largest first, sends them together (one per message under `single` pacing, "One question at a time" below), and waits; the replies decide what is ready next time. One ready question still makes a round 2. **Survey-then-deep**. Before asking blind, do a fast breadth pass (repo files, recent commits, existing skills, relevant project rules) so questions land in real context 3. **Climb-to-anchor**. Find the nearest `CLAUDE.md`, `AGENTS.md`, domain-vocabulary file, or module README by walking UP from the relevant directory toward repo root; let those shape questions instead of asking what is already documented -4. **Immediate doc maintenance** *(engineering sessions only)*. When an answer resolves a domain - term, invoke `/domain-driven-design:ubiquitous-language curate` via the Skill tool (if that plugin is enabled; else - record the term and its definition in the resolution field of the open-question register row that resolved it) IMMEDIATELY between questions, not - batched at end. Route - decisions, gotchas, and conventions to their proper homes (ADR, project rules, side note) in the - same response. A general session writes no repo docs. It drives to a shared-understanding summary - only - -**Intake the starting point.** Early in the loop (or before it), establish where the user is. One intake question that discloses their starting point; questions and recommendations calibrate to that disclosure. Route by what it shows; recommend, never auto-switch: +4. **The project's terms from the start**. Once the interview starts, past the quick-change skip + in [`context/routing.md`](context/routing.md), invoke `/domain-driven-design:ubiquitous-language lookup` + via the Skill tool, with one line saying why ("Reading the project glossary so these questions + use its terms."); when that skill is not listed, read the glossary file as loop.md "Step 1. + Survey before you ask" says, and the opening line offers its install once ("Question lanes"). + Agree each term in the question where it first appears and write it only after the user + confirms its restatement ("Agree a term where it first appears"). Route decisions, gotchas, and + conventions to their proper homes (ADR, project rules, side note) in the same response. A + general session writes no repo docs. It drives to a shared-understanding summary only + +**Intake the starting point.** Early in the loop (or before it), establish where the user is. One discovery question in the intake round ([`context/lookups.md`](context/lookups.md) "The intake round") that discloses their starting point; questions and recommendations calibrate to that disclosure. Route by what it shows; recommend, never auto-switch: - **Idea stage** (a half-formed idea, "I just had this thought", the user does not yet know what they want): recommend `/planning:brainstorm`, which explores approaches and returns here with a handoff. If the user stays, follow [`problem-framing.md`](../../context/problem-framing.md) "Idea-stage requests". - **User-facing product work** whose users or success measures are unsettled: recommend `/planning:prd` first. @@ -112,7 +114,7 @@ The Q&A path of this skill is one engine wrapped in a stop condition and an outp Every other skill the interview hands to or detours through: [`context/composition.md`](context/composition.md). -**Ask the quality bar in round 1.** When the request does not already state it, round 1 asks which bar the work is for: a throwaway prototype, an internal tool, or production. When the request states it, never re-ask it; treat it as settled. Interview depth scales with the bar. A prototype gets the scoping questions only, with production concerns (accounts, scaling, monitoring, CI, coverage targets) named as deferred rather than asked. Production gets the full depth, including failure handling, data correctness, security, monitoring, and rollback; an internal tool sits between the two. When the user cannot judge which bar applies ("I don't know", "you tell me"), recommend one from evidence: who runs the result, what data it touches, and whether it ships to users. Ask them to confirm it like any recommendation. Never fall back to the prototype bar unasked. +**Ask the quality bar in round 1.** When the request does not already state it, round 1 asks, as a discovery question, which bar the work is for: a throwaway prototype, an internal tool, or production. When the request states it, never re-ask it; treat it as settled. Interview depth scales with the bar. A prototype gets the scoping questions only, with production concerns (accounts, scaling, monitoring, CI, coverage targets) named as deferred rather than asked. Production gets the full depth, including failure handling, data correctness, security, monitoring, and rollback; an internal tool sits between the two. When the user cannot judge which bar applies ("I don't know", "you tell me"), recommend one from evidence in the next round: who runs the result, what data it touches, and whether it ships to users. Ask them to confirm it like any recommendation. Never fall back to the prototype bar unasked. **Open with the problem when the request has none.** When the request names no problem (an idea, or a solution with no stated why), round 1's first question asks what problem it solves and why the user wants it solved; keep that answer in the user's words for the Goal's Intent line. @@ -126,7 +128,7 @@ A new effort starts here, however big. When the interview shows the effort will **Bulk application work is not a decision set. The interview settles it once, it never executes it.** When the task applies a settled shape across a corpus (every document in a set, every file matching a pattern, every page of a spec), the interview's output is TWO things and no more: the small set of genuinely contested decisions, driven to decisions the normal way, plus an **execution contract**. One line in the session's output artifact, routed by domain like every other output: the Brief's `### Acceptance criteria` in an engineering session, the shared-understanding summary in a general one. Naming the per-unit close-out loop (one source unit at a time: apply, verify, close) and what *closed* means for a unit. NEVER enumerate the per-unit applications as decision rows: an interview that becomes the execution container turns every application step into a decision needing its own adoption ceremony, and the corpus's size becomes the session's length. **Tripwire. Count the candidates against the corpus:** when the candidate question count scales with the number of source units rather than with the number of genuine forks, that is execution masquerading as decisions. Collapse the per-unit rows into the loop, keep only the forks, and say what you collapsed. Distinct from the ballooning frontier above and remedied differently: ballooning routes to `/planning:wayfind`, unit-scaling collapses into the execution contract and hands off to execution. -Tone is collaborative but opinionated. You are not interrogating; you are helping the user think out loud by PROPOSING answers grounded in codebase evidence. When the user gives a definitive answer, lock it. When they hesitate, slow down and offer two or three concrete shapes the answer could take. Every option set names exactly ONE recommended option marked **(RECOMMENDED)** with its basis (each fact with its source, then the one-line inference), except a withheld question (see "Recommended answers"), which is asked open with no recommended option. The surface follows the resolution in "Question lanes" above: rounds go where its handle sends them, and `AskUserQuestion` only where `handle.rules` allows cards, when the user opted in and the round qualifies. +Tone is collaborative but opinionated. You are not interrogating; you are helping the user think out loud by PROPOSING answers grounded in codebase evidence. When the user gives a definitive answer, lock it. When they hesitate, slow down and offer two or three concrete shapes the answer could take. Every option set names exactly ONE recommended option marked **(RECOMMENDED)** with its basis (each fact with its source, then the one-line inference), except a withheld or discovery question (see "Recommended answers"), each asked open with no recommended option. The surface follows the resolution in "Question lanes" above: rounds go where its handle sends them, and `AskUserQuestion` only where `handle.rules` allows cards, when the user opted in and the round qualifies. **Do not nod along.** Agreement is not the job. When an answer conflicts with the code, a constraint row, or an earlier answer, name the conflict and its source once, before recording it, and ask whether they hold the answer or change it, registered like any question; the row stays `open` until they say. Never praise an answer or call it a good idea. @@ -165,7 +167,7 @@ Alternatives: **Group the round by topic.** When a round's questions span more than one topic, put them under short topic headers (`#### Retry budget`), order the questions inside each group by blast radius as the round already does, and order the groups by the largest blast radius each holds. `Q` numbering stays continuous across groups. The round ends with no summary table. The user may answer by exception ("all as recommended except Q7 = b") or by topic ("on retries, ..."). A reply addressed to a topic resolves each question under that header it decides, under the same read-back, free-text and hedged rules as any answer, and leaves the rest `open` under the drift check. -**Name the held questions in the round header.** When a question is held for an in-flight lookup ("In-flight lookups" below), the round header carries one `Held:` line naming each held question by id and plain name, and what it waits on: `Held: Q6 (alert threshold), waiting on the principle map`. A held question takes its `Q` and its `open` register row when the `Held:` line first names it, with the resolution `waits on: `, so the gate sees it and the id never moves. It is asked in full in the first round after its lookup returns, and dropped from the `Held:` line then. A held row the reply does not address is restated by the `Held:` line, not by a separate drift line. +**Name the held questions in the round header.** When a question is held for a running lookup ([`context/lookups.md`](context/lookups.md) "Held questions"), the round header carries one `Held:` line naming each held question by id and plain name, and what it waits on: `Held: Q6 (alert threshold), waiting on the principle map`. A held question takes its `Q` and its `open` register row when the `Held:` line first names it, with the resolution `waits on: `, so the gate sees it and the id never moves. It is asked in full in the first round after its lookup returns, and dropped from the `Held:` line then. A held row the reply does not address is restated by the `Held:` line, not by a separate drift line. **Name later rounds in the round header.** When this round's answers will open questions that are not ready yet, the round header carries one `Later:` line naming those topics in plain words: `Later: interface shape, test plan, migration order`. It lists topics, not questions: nothing on it takes a `Q` or a register row, and a topic leaves the line once its questions are asked or an answer removes the need for them. Omit the line when no later round is known. @@ -173,7 +175,7 @@ Alternatives: **One question at a time (`question_pacing`).** Pacing: resolve `question_pacing` once, before the first round or the page start, per [`${CLAUDE_PLUGIN_ROOT}/reference/config.md`](../../reference/config.md) "Convention-home keys", with `${user_config.question_pacing}` as its user layer (unexpanded or empty means unset; the default is `round`), and report the value and its layer in one line. `auto` settles to `round` or `single` after the Step 1 survey and before the first round or the page start, and everything after reads that choice: `single` when the topic is broad and open-ended (many independent directions, little settled upstream, no spec or earlier answer to anchor on), `round` otherwise. State the choice and how to switch in one line ("Asking one at a time: this topic is wide open. Say 'give me the whole round' to switch."). `round` asks the frontier as one set. `single` computes the frontier the same way, then asks one question per message in blast-radius order: each message carries one question in the full layout, the `Held:` line when one applies, and the closing probe, with no topic header. One frontier is still one round: one round number and one `round-.md` file, sent over several messages. A round fired from inside another workflow's phase ignores `single` ("Rounds fire at phase boundaries"). Register each question `open` in the message that asks it, and recompute the frontier after each answer, since an answer can prune or add questions. The user's request in the session ("one at a time", "give me the whole round") overrides it for that session. The page honors `single` too: the session posts the round with its first question and adds each later one after the one before it is answered ([`context/lane-page.md`](context/lane-page.md)). -`Q` numbering runs continuously across rounds (Q1…Q4 in round one, Q5… in round two; visible depth). Close the round with one probe inviting a constraint that breaks the recommendations, worded "something else", never "anything else": "Is there something else that would change these recommendations?" (eval case `interview-closing-probe-something-else`). Put the note that the user may answer in any order before the probe. In the terminal lane an inline round is the last message of the turn that sends it and the probe its last line, because the user answers what the turn ends on: write its round file, run its checks and register its rows before it is sent (under `single`, each row in the message that asks it), and call no tool after it. A correction that changes a question already shown resends the whole round with the note in its header (under `single`, only that question, the note above it); any other note waits for the top of the next round. Wait for the answers before computing the next round. The recommendation+basis discipline holds per question. Inline prose carries the recommendation, its reasoning, AND the probe in one readable block; a card cannot. +`Q` numbering runs continuously across rounds (Q1…Q4 in round one, Q5… in round two; visible depth). Take every new id from `bash "${CLAUDE_PLUGIN_ROOT}/surface/round.sh" next-id --ledger //interview-checklist.md --count `, `` being the round's new rows (held, settled-by-evidence and commitment rows included; on the page lane add `--dir ''`), and assign them in the order the round shows them; never pick an id by hand, fill a gap or renumber a row. With no Bash, take the highest number among the register's rows and the high ends of its `- gap |` lines, and count up from it. Close the round with one probe inviting a constraint that breaks the recommendations, worded "something else", never "anything else": "Is there something else that would change these recommendations?" (eval case `interview-closing-probe-something-else`). Put the note that the user may answer in any order before the probe. In the terminal lane an inline round is the last message of the turn that sends it and the probe its last line, because the user answers what the turn ends on: write its round file, run its checks and register its rows before it is sent (under `single`, each row in the message that asks it), and call no tool after it. A correction that changes a question already shown resends the whole round with the note in its header (under `single`, only that question, the note above it); any other note waits for the top of the next round. Wait for the answers before computing the next round. The recommendation+basis discipline holds per question. Inline prose carries the recommendation, its reasoning, AND the probe in one readable block; a card cannot. **One verdict marker.** The `My recommendation:` line is the *single* verdict marker for the question. Never stack a second one: no standalone `**(RECOMMENDED)**` badge line above it, and no `(recommended)` tag repeated in the Alternatives list; the recommended answer is named once, on that line. The alternatives exclude the recommendation: none restates it. @@ -191,7 +193,7 @@ Alternatives: **Emoji anchors (default on, still configurable).** Emoji markers: `${user_config.use_emoji_question_markers}` (unexpanded or empty means the default, on). With the emoji-markers setting on (the default), prefix the `Q:` line with `❓` and the `My recommendation:` line with `➡️`; with it off, keep the undecorated shape above. Decoration of the existing single verdict marker, never a second one. Conversational rendering only: the ledger, register, and Brief stay plain, and `Q` remains the answer handle. The option is the plugin's `use_emoji_question_markers` user config; set it false for plain text. A user instruction against emoji as formatting wins over the option: keep the plain shape, and the page start command then passes `--emoji-markers false` so the page matches. -**Agree the terms before the first decision.** Before the first decision question, build the ledger's `## Terminology matrix`: one row per sourced term the rounds will rely on, with its definition, its source (`file:line`, URL, or `user, round `) and its status, `proposed` or `agreed`. Show it to the user as the interview's first item, ahead of round 1's decision questions (the problem question may share its message), as one question registered like any other; it is never repeated at the top of a later round, and it never stands in for another question's definitions: every question, round 1's included, defines inline each term it relies on. A later round that adds a sourced term adds a `proposed` row and asks to agree it in that round; one that coins a definition asks it withheld in that round. A qualified name that cites its clash is a sourced term ([`context/term-clash.md`](context/term-clash.md)). A definition this session coins has nothing to cite, so it gets no row and no `Basis:` claim: record it in the shorthand glossary ("Name every code; define every term where it is used") and ask the user to agree it in a withheld question ("A withheld question carries no recommendation marker") beside the matrix question, or in its place when the matrix reads `none`. Its `Withheld:` line says the definitions are the session's own and only the user can confirm them, each Context line is labeled `judgment`, and it asks the user to confirm or correct them. With no sourced term the matrix reads `none`, and when the rounds rely on no term the user may not share, nothing is asked; a run that asks no decision question (`lock`, direct synthesis) builds none. A round uses no term missing from the agreed matrix or the agreed coined definitions, and none left undefined in its own question. Row shape: [`context/loop.md`](context/loop.md) "Terminology matrix". +**Agree a term where it first appears.** Nothing is asked about terms before round 1. The question that first relies on a sourced term defines it in its Context with its source (`file:line`, URL, or `user, round `), and registering that question adds the term's `proposed` row to the ledger's `## Terminology matrix`. Once the answers settle a term, restate it with its definition in one line at the top of the next message ("Term: billing account, the household that pays for several members (user, round 2). Confirm or correct it."), and mark its row `agreed` only when the user confirms that restatement; a row still `proposed` at Step 3 is confirmed in the closing summary's Glossary group. A qualified name that cites its clash is a sourced term ([`context/term-clash.md`](context/term-clash.md)). A definition this session coins has nothing to cite, so it gets no row and no `Basis:` claim: record it in the shorthand glossary ("Name every code; define every term where it is used") and ask the user to agree it in a withheld question ("A withheld question carries no recommendation marker") in the round that first needs it. Its `Withheld:` line says the definitions are the session's own and only the user can confirm them, each Context line is labeled `judgment`, and it asks the user to confirm or correct them. Write a term only once it is agreed: in an engineering session through the inline vocabulary update ("Domain-aware behaviors"); without the glossary skill, or in a general session, the matrix keeps it and the session's durable output carries it (the Brief's `### Constraints` in an engineering session, the shared-understanding summary in a general one). Every question defines inline each term it relies on. Row shape: [`context/loop.md`](context/loop.md) "Terminology matrix". **Name every code; define every term where it is used.** In every line the user reads (rounds, restatements, recaps, the page, labels, reports), show the human-readable name, never a bare code. When the name is short enough to stand alone, show only the name (`scoped change`, not `C3`); pair code and name (`C3: scoped change`, `question Q14, retry budget`) only where the code itself must be quoted, such as a question id the user answers by. `Checked against:` names constraints in words; the ledger may keep bare ids. When a round leans on session-local shorthand, a coined label, an abbreviation, or cross-repo jargon the user may not share, record it in the ledger's shorthand glossary and define it inline in each question that uses it (`head SHA (the commit the PR points at)`), so a reader with none of the session's knowledge can answer from that question alone. Self-contained is not simplified: keep the precise technical terms and register, and expand a code to its real name, never a metaphor or lay substitute (`pr-merge lane`, not `merge robot`); simplify only when the user asks (`/education:explain`, if enabled). The glossary is ephemeral session vocabulary, distinct from the project's ubiquitous language (owned by `/domain-driven-design:ubiquitous-language`), and never touches a project glossary. Ledger shape: [`context/loop.md`](context/loop.md) "Session-shorthand glossary". Eval case: `interview-coded-terms-round-answerable`. @@ -215,19 +217,19 @@ Alternatives: **Dialogue: the user drives too.** They may push back or reframe a decision (e.g. "what's hardest to roll back from?"). When they introduce a new axis, **reversibility** is the most common for V1, re-rank the options on it and REVISE your recommendation out loud, stated as old → new → why, where new may be a withheld open question ([re-emitting a changed recommendation](../../context/recommendation-basis.md)). **Talking a question through.** When the user asks to discuss a question rather than answer it ("let's discuss Q3"), set its register row to `discussing` before the discussion starts. Discuss it, then ask it again in the full layout with the recommendation revised as old → new → why (when the discussion leaves it unchanged, say so and why). The row leaves `discussing` only on an answer to the question asked again; the register gate holds a `discussing` row exactly as it holds an `open` one. `discussing` applies to the terminal lane; on the page the row stays `open` ([`context/loop.md`](context/loop.md) "Discussing"). The same holds when a recommendation changes after the user asks for context: re-ask the question in the full layout, its Context now carrying what they asked for, with the old and new recommendation and the reason. Default V1 lens: prefer the most *reversible* start; defer the irreversible/expensive as an explicit out-of-scope decision, never a silent assumption. -**Facts are yours; decisions are the user's.** A *fact*, a path, a current value, an existing pattern, what a file already does, is resolved from the environment (Grep/Read/Glob) and STATED, never asked; spending a question on what the code already answers is friction, not interview. Every candidate question first passes the self-answer step, [`${CLAUDE_PLUGIN_ROOT}/context/self-answer.md`](../../context/self-answer.md): prior owner decisions, then codebase, ADRs, docs and history, then connected tools including MCP servers, then research for external facts. A fact it finds is stated with its `Basis:` and source tag and gets no register row (a contact address or identity still gets one: "Recommended answers"); a fact only an unconnected system holds gets one offer to connect it. Register mechanics: [`context/loop.md`](context/loop.md) "Self-answer gate". The environment is not only the working tree: when a task NAMES an external repo or resource, a sibling checkout under a known repo root / workspace layout, or an `owner/repo` reachable through its host, that is a resolvable fact too, so check the filesystem layout and query the repo host directly (e.g. `gh` for a named `owner/repo`) before defaulting to a user question. Cue, not mandate, except where a recommendation is consequential (next paragraph): there grounding is mandatory. Otherwise resolve what's cheaply resolvable, and don't turn every named mention into a research project; a recommendation of any kind still cites a fact, so a question the cheap lookup leaves with none is asked withheld ("Recommended answers"). A *decision* with a live alternative and no environment answer ALWAYS goes to the user; never resolve one on their behalf. One whose evidence leaves only one viable answer is settled by evidence ("Name the evidence-settled decisions in the round header"); one the owner already ruled on is answered by the quoted ruling (the self-answer step). **In-flight lookups.** A slow fact lookup (deep exploration, external research) runs in a sub-agent. Sort the frontier by one test, does this question need the lookup's result? A question that does is held for a later round and named on the round header's `Held:` line; a question that does not is asked in this round, without waiting for the lookup. +**Facts are yours; decisions are the user's.** A *fact*, a path, a current value, an existing pattern, what a file already does, is resolved from the environment (Grep/Read/Glob) and STATED, never asked; spending a question on what the code already answers is friction, not interview. Every candidate question first passes the self-answer step, [`${CLAUDE_PLUGIN_ROOT}/context/self-answer.md`](../../context/self-answer.md): prior owner decisions, then codebase, ADRs, docs and history, then connected tools including MCP servers, then research for external facts. A fact it finds is stated with its `Basis:` and source tag and gets no register row (a contact address or identity still gets one: "Recommended answers"); a fact only an unconnected system holds gets one offer to connect it. Register mechanics: [`context/loop.md`](context/loop.md) "Self-answer gate". The environment is not only the working tree: when a task NAMES an external repo or resource, a sibling checkout under a known repo root / workspace layout, or an `owner/repo` reachable through its host, that is a resolvable fact too, so check the filesystem layout and query the repo host directly (e.g. `gh` for a named `owner/repo`) before defaulting to a user question. Cue, not mandate, except where a recommendation is consequential (next paragraph): there grounding is mandatory. Otherwise resolve what's cheaply resolvable, and don't turn every named mention into a research project; a recommendation of any kind still cites a fact, so a question the cheap lookup leaves with none is asked withheld ("Recommended answers"). A *decision* with a live alternative and no environment answer ALWAYS goes to the user; never resolve one on their behalf. One whose evidence leaves only one viable answer is settled by evidence ("Name the evidence-settled decisions in the round header"); one the owner already ruled on is answered by the quoted ruling (the self-answer step). **Gap triage and lookups.** Before round 1, sort every unknown into a lookup, a discovery question or a decision question; that triage, the intake round, when a lookup runs, what its brief carries, which questions it holds and what a contradicting result reopens: [`context/lookups.md`](context/lookups.md). A question a running lookup could change is held and named on the round header's `Held:` line; every other question is asked in this round, without waiting for the lookup. A fact from any source (a lookup's return, the user relaying one) that contradicts an answered row reopens it under its own number with `round.sh reopen`, which marks every row resting on it for recheck; never change the answer in place. **Check a fact the user states.** When the user states a fact about the code or the system ("only the importer calls this", "we never delete rows"), check it with Grep, Read or Glob before a recommendation rests on it. A constraint stated this way is still written `confirmed` at once (Step 2, "Keep a running constraint ledger"). When the code disagrees, show the evidence (`file:line`) and ask which holds, registered like any question, then revise the row on the answer; never let the stated version stand over the code silently. **Run it to settle it.** When a question turns on what the system does and a cheap, read-only run would show it (an existing status or query command, a `--help`, a dry run), run it and state the result as a fact with the command as its source instead of asking. A run that needs the user's environment is offered as a registered question. A run that writes, deploys or costs money is never made: it stays a question. -**Ground before recommending.** Lightweight codebase gate per question (Grep/Read/Glob). If a recommendation needs more, external best-practice, a library API, deeper exploration, dispatch or do the lookup (research/exploration capability, or inline), then recommend grounded. No recommendation rests on training recall or convention alone ("Recommended answers"), and a **consequential** one (cross-repo, shared infrastructure, irreversible, or security) clears the higher bar: ground it in the affected code plus its consumers, and in external research that reads official docs first for a tool or API and the canonical source for a design pattern, never a pattern's popularity alone, and notes recency and dissent. That lookup is an in-flight lookup under the rule above. The contract this plugin ships, the bar and its three outcomes, is [`${CLAUDE_PLUGIN_ROOT}/context/recommendation-basis.md`](../../context/recommendation-basis.md). +**Ground before recommending.** Lightweight codebase gate per question (Grep/Read/Glob). If a recommendation needs more, external best-practice, a library API, deeper exploration, dispatch or do the lookup (research/exploration capability, or inline), then recommend grounded. No recommendation rests on training recall or convention alone ("Recommended answers"), and a **consequential** one (cross-repo, shared infrastructure, irreversible, or security) clears the higher bar: ground it in the affected code plus its consumers, and in external research that reads official docs first for a tool or API and the canonical source for a design pattern, never a pattern's popularity alone, and notes recency and dissent. That lookup runs and holds as [`context/lookups.md`](context/lookups.md) says. The contract this plugin ships, the bar and its three outcomes, is [`${CLAUDE_PLUGIN_ROOT}/context/recommendation-basis.md`](../../context/recommendation-basis.md). **Before presenting an option that adds a mechanism** (a branch, workflow, ruleset, app, service, dependency or check), name the existing control that already covers the same risk, saying whether you read its enforcing source or only a document, and recommend dropping the mechanism if one does; a security control (`/overengineering:justify`'s protected classes) is never dropped without the user's decision, and an abstraction for a stated, known repetition stays when it passes the `reference-dont-duplicate` tests of the standards index (resolved per [`${CLAUDE_PLUGIN_ROOT}/reference/standards-contract.md`](../../reference/standards-contract.md) "Resolution ladder"). For a non-trivial new mechanism, run `/overengineering:justify` on the proposal if enabled. ### Recommended answers -For EVERY question but a withheld one (below), propose an answer grounded in observed evidence. User confirms (fast) or corrects (faster than explaining from scratch). **Every recommendation shows its evidence, never bare judgment.** Each fact the answer rests on sits in the question's Context block with its source (a `file:line`, a URL fetched this session, a session-transcript finding, or tool output); the `My recommendation:` line's reason says why those facts together lead to this answer for this user, and `Basis: verified` names which of those sources the answer rests on. A `judgment` label alone is not a valid basis in this skill, for a consequential question or any other: the interview holds a stricter bar than the shared [basis contract](../../context/recommendation-basis.md), which still allows that label elsewhere. When only judgment is available, research first (the self-answer step, and "Ground before recommending" for a consequential question); when research leaves no fact to cite, ask the question withheld (below). Like `Checked against:`, the basis is part of the recommendation, not a second verdict marker and not part of the Context block; on a card the facts, the reason and any `Flips if:` line ride inside the option's basis text, and on the page the facts go in the question's `context` while the basis text names their sources and carries the reason and any `Flips if:` line, and the `Flips if:` line counts toward the basis prose's 3-sentence budget ([`context/surface.md`](context/surface.md) R12). Mark each recommendation once. In an inline round that marker IS the `My recommendation:` line (one verdict marker, never a second stacked badge); in an `AskUserQuestion` card it is the option tagged **(RECOMMENDED)**. Detail in [`context/loop.md`](context/loop.md) "Frontier rounds". +For EVERY question but a withheld or discovery one (below), propose an answer grounded in observed evidence. User confirms (fast) or corrects (faster than explaining from scratch). **Every recommendation shows its evidence, never bare judgment.** Each fact the answer rests on sits in the question's Context block with its source (a `file:line`, a URL fetched this session, a session-transcript finding, or tool output); the `My recommendation:` line's reason says why those facts together lead to this answer for this user, and `Basis: verified` names which of those sources the answer rests on. A `judgment` label alone is not a valid basis in this skill, for a consequential question or any other: the interview holds a stricter bar than the shared [basis contract](../../context/recommendation-basis.md), which still allows that label elsewhere. When only judgment is available, research first (the self-answer step, and "Ground before recommending" for a consequential question); when research leaves no fact to cite, ask the question withheld (below). Like `Checked against:`, the basis is part of the recommendation, not a second verdict marker and not part of the Context block; on a card the facts, the reason and any `Flips if:` line ride inside the option's basis text, and on the page the facts go in the question's `context` while the basis text names their sources and carries the reason and any `Flips if:` line, and the `Flips if:` line counts toward the basis prose's 3-sentence budget ([`context/surface.md`](context/surface.md) R12). Mark each recommendation once. In an inline round that marker IS the `My recommendation:` line (one verdict marker, never a second stacked badge); in an `AskUserQuestion` card it is the option tagged **(RECOMMENDED)**. Detail in [`context/loop.md`](context/loop.md) "Frontier rounds". **State a reused value literally, with where it was read.** When a recommendation reuses an existing value (an email address, account, subscription, region, secret name, repository), the question states the literal value and its source (a `file:line`, tool output, or the connected system that holds it), for example "alerts go to `ops@example.com`, the `budgetContactEmail` in `Pulumi.production.yaml:12`". "The existing X" or "the same address as Y" alone is not allowed: an accepted round would then confirm a value the user never saw. @@ -235,7 +237,9 @@ For EVERY question but a withheld one (below), propose an answer grounded in obs **Word each question so yes accepts the recommendation.** When the recommendation is negative, ask "Should X stay out of this change?", never "Should we do X?" answered "No", because "Q yes" and "accept all" must both mean the recommended answer. -**A withheld question carries no recommendation marker.** Two questions are exceptions to "every question carries a recommendation", and their recommendation is **withheld**: a question research cannot settle (a recommendation with no fact to cite would be bare judgment, consequential or not), and, at the idea stage in the terminal lane, a question about why the user wants this or where it heads, since a proposed answer would put words in their mouth. It is asked open, never on a card, with a `Withheld:` line naming the evidence that would settle it in place of the `My recommendation:` line, its `Origin:` line, any `Checked against:` line, its `Basis:` evidence list and any `Flips if:` line; no option is tagged **(RECOMMENDED)**, and on the page its `recommendation` field carries the `Withheld:` text. It is registered like any question. +**A withheld question carries no recommendation marker.** A question research cannot settle is an exception to "every question carries a recommendation", and its recommendation is **withheld**: a recommendation with no fact to cite would be bare judgment, consequential or not. It is asked open, never on a card, with a `Withheld:` line naming the evidence that would settle it in place of the `My recommendation:` line, its `Origin:` line, any `Checked against:` line, its `Basis:` evidence list and any `Flips if:` line; no option is tagged **(RECOMMENDED)**, and on the page its `recommendation` field carries the `Withheld:` text. It is registered like any question. + +**A discovery question carries no recommendation.** A question the gap triage sorts as discovery ([`context/lookups.md`](context/lookups.md)) asks what only the user knows (the problem and why, who it is for, the scope, where they start, where it heads), so a proposed answer would put words in their mouth. It is asked open: its Context block, then one `Discovery: ` line in place of `My recommendation:` and every line after it, with no alternatives, never on a card. It is registered `open` with `kind:: discovery`, and no accept-shorthand resolves it. A choice its answer leaves open (the user cannot judge the quality bar) is a decision question in the next round. ### Domain-aware behaviors @@ -247,9 +251,10 @@ When the task touches domain concepts, these behaviors activate during Q&A. The - **domain scenario exploration**. Test where one concept ends and the next begins with a concrete case of your own making ("a member returns one volume of a two-volume loan: is the loan open or closed?") - **scenario comparison**. When two or more candidate terms or models compete for one question, or the user asks for one, put a scenario comparison in the question's Context: scenarios as rows, candidates as columns, the sentence each candidate forces, then where each helps, where the choice makes no difference, and what each costs. Shape and rules: [`context/scenario-comparison.md`](context/scenario-comparison.md) - **scenario walk**. When the user calls something unclear, or a question rests on a term the glossary lacks or uses two ways, offer (never force) numbered concrete scenarios `S1`, `S2`, ... inside one question's Context, each ending with its own recommendation and the term or rule it forces, answered per scenario ("S3: a"). Triggers, shape and recording: [`context/scenario-comparison.md`](context/scenario-comparison.md) "Scenario walk" -- **inline vocabulary update** *(engineering sessions only, per `domain_docs`)*. When a term resolves, invoke - `/domain-driven-design:ubiquitous-language curate` immediately (if that plugin is enabled; else - record the term and its definition in the resolution field of the open-question register row that resolved it). That skill owns discovery-first +- **inline vocabulary update** *(engineering sessions only, per `domain_docs`)*. When the user + confirms a term's restatement ("Agree a term where it first appears"), invoke + `/domain-driven-design:ubiquitous-language curate` (if that plugin is enabled; else the ledger's + `## Terminology matrix` keeps the agreed term). That skill owns discovery-first placement, the consumer's file shape, purity, canonical terms, rejected synonyms, and known-context routing; the interview resumes after the update @@ -301,6 +306,8 @@ Run rounds: restate working understanding → compute the frontier (every open q Full surfacing-question taxonomy + categorization heuristics in [`context/loop.md`](context/loop.md). +**One blind recommender drafts each round's recommendations.** Once a round's decision questions are drafted (Context, options, your own answer), dispatch the plugin's `interview-recommender` agent once for the round ([`${CLAUDE_PLUGIN_ROOT}/agents/interview-recommender.md`](../../agents/interview-recommender.md)), handing it each decision question with its options unranked and its Context facts, the constraint ledger, the settled decisions with their answers and the self-answered facts; withhold the conversation and your own answers. Its answer is the question's recommendation: its basis feeds the reason and `Basis:`, its `flips if` the `Flips if:` line, and a question it withholds is asked withheld. When your answer differs, show it on one `Second view: ` line under the recommendation; when the recommendation departs from a settled decision, show one `Departs from: Q (): ` line. No detail level hides either line. A round with no decision question runs no recommender. With no Agent tool, draft the recommendation yourself and show no `Second view:` line. + When the resolved lane's spoke shows Step 3's confirmation gate outside this conversation, follow that spoke for where it is shown; the gate and its halts are unchanged. **`me` mode** maintains a **decision-tree ledger**. One live checkbox per branch, ticked on resolve, remaining-open surfaced periodically (not every turn. Keeps the flow clean like the inline format). Persist each answer the moment it locks in (Step 4), loop until zero open consequential branches. Ask via the inline format (Stance "Relentless mode"). Ledger shape + per-round mechanics + reversibility-lens question shape in [`context/loop.md`](context/loop.md) "Decision-tree ledger". @@ -315,7 +322,7 @@ When the resolved lane's spoke shows Step 3's confirmation gate outside this con **What, not how: the mechanism tripwire.** "Does not plan implementation" has a trigger. When a question or its recommendation names a mechanism (a tool, an API, a credential or token scope, a runtime, a platform per step), ask once whether the user wants mechanism-level decisions settled now or at planning. Ask it in the same round as the first mechanism-naming question and register it like any other question. Its Context names the question that named the mechanism, and its `Basis:` cites the fact behind that mechanism in any form "Recommended answers" allows (`Basis: verified, user, round 1: deploys already run through one CI pipeline`); neither cites this rule or this skill, which the user cannot check. With no such fact, ask it withheld. Until it is answered, recommendations carry no mechanism part. When the answer is "at planning", no later recommendation fixes a mechanism, and every still-`open` mechanism row, commitment rows included, becomes `deferred`: carried to the Brief's `### Deferred questions` under the arbiter rule in an engineering session, listed as deferred in the shared-understanding summary in a general one. When the answer is "now", write `C | confirmed | mechanism decisions settled in this interview | user, round ` at once, and the Brief's Constraints records that this topic overrides "does not plan implementation". -**Sweep for assumptions before Step 3.** In `me` and `auto` with a register, including the `auto` Mixed path after its residue round, the loop ends with a fresh-context assumption sweep. Once no open question remains on the frontier, and before the Step 3 register gate and confirmation gate, dispatch a non-fork sub-agent to inventory undecided details, hidden defaults, contradictions, and hedged or free-text rows across the ledger, the recommendations, and any composed artifacts. You turn each item into a register row at the next contiguous `Q`, a stated fact, or, for hedged rows, a line in the confirmation restate. Any `open` row it adds returns the run to Step 2, and the next sweep covers only what the new answers changed; Step 3 is reached when a sweep adds no `open` row. A run that asked no question skips it, and so does `lock`; unattended, an item that is the user's decision is `blocked`. The acceptance-criteria coverage prompt is never a sweep item ([`context/coverage-prompt.md`](context/coverage-prompt.md)). Scope, item shape, disposition and the sweep-round label: [`context/assumption-sweep.md`](context/assumption-sweep.md). +**Sweep for assumptions before Step 3.** In `me` and `auto` with a register, including the `auto` Mixed path after its residue round, the loop ends with a fresh-context assumption sweep. Once no open question remains on the frontier, and before the Step 3 register gate and confirmation gate, dispatch a non-fork sub-agent to inventory undecided details, hidden defaults, contradictions, and hedged or free-text rows across the ledger, the recommendations, and any composed artifacts. It reports every finding, with no severity filter; the disposition pass is the filter, and records each item's outcome: you turn each item into a register row at the next id from `round.sh next-id`, a stated fact, or, for hedged rows, a line in the confirmation restate. Any `open` row it adds returns the run to Step 2, and the next sweep covers only what the new answers changed; Step 3 is reached when a sweep adds no `open` row. A run that asked no question skips it, and so does `lock`; unattended, an item that is the user's decision is `blocked`. The acceptance-criteria coverage prompt is never a sweep item ([`context/coverage-prompt.md`](context/coverage-prompt.md)). Scope, item shape, disposition and the sweep-round label: [`context/assumption-sweep.md`](context/assumption-sweep.md). ### Step 3. Recognize the stop condition @@ -401,7 +408,7 @@ Read [`context/session-config.md`](context/session-config.md) at that boundary, ## Routing by size and fog -Read [`context/routing.md`](context/routing.md) at entry and at every round boundary. It owns the entry gate's diff-size and retry-cost thresholds with the one-line quick-change skip, the round-1 question recommending `/planning:wayfind chart` when the request itself names unknowns in several areas, the bigger-than-you-thought check that recommends `/planning:wayfind chart` on fog or `/session-flow:workflow` on context size alone and suggests wayfind once the register passes `wayfind_suggest_questions`, what the wayfind handoff carries, and the "this is small, ship it" exit with the gates it still runs. +Read [`context/routing.md`](context/routing.md) at entry and at every round boundary. It owns the entry gate's diff-size and retry-cost thresholds with the one-line quick-change skip, the round-1 question recommending `/planning:wayfind chart` when the request itself names unknowns in several areas, the problems agreed in round 1 as P1, P2, the round header's `Scope check:` line from `round.sh scope-check`, the bigger-than-you-thought check that recommends `/planning:wayfind chart` on fog or `/session-flow:workflow` on context size alone and suggests wayfind once the register passes `wayfind_suggest_questions`, what the wayfind handoff carries, and the "this is small, ship it" exit with the gates it still runs. Failure patterns from real sessions: `context/gotchas.md`. @@ -409,7 +416,7 @@ Failure patterns from real sessions: `context/gotchas.md`. - **A term that clashes with an existing concept.** When a term the interview asks about clashes with a concept the code or the glossary already names, the recommendation is a qualified name the interview proposes itself, so the user never has to add the qualifier; once agreed, it is written through the inline vocabulary update ("Domain-aware behaviors"), under `domain_docs`. Read [`context/term-clash.md`](context/term-clash.md) when a clash is found. - **An outcome that feeds a prototype.** Ask what the prototype must prove, and split it into one small piece per question that can be built at once in separate worktrees. Read [`context/prototype-questions.md`](context/prototype-questions.md) once the quality bar is a throwaway prototype or the next step builds one. -- **A question that turns on a fact.** Look it up yourself; never send the user to find it. When a quick lookup does not settle it, run `/discovery:explore` (what the code does) or `/discovery:research` (an external fact) as an in-flight lookup, each if enabled, otherwise the same work inline; read what a live system holds now (a database, a tracker, a service) directly through its CLI or MCP tool; this dispatch is how the exploration recommendation in "What this skill does NOT do" is carried out, never a request that the user run it. Do not stall the round proving the fact beyond what the recommendation needs: in an engineering session, the Brief's `### Captured assumptions` lists each fact from such a lookup that a recommendation rests on, with its source and the tag `verify during build`. A fact only the user's environment or an unconnected system holds still takes the self-answer step's offer, and a decision is still the user's. +- **A question that turns on a fact.** Look it up yourself; never send the user to find it. When a quick lookup does not settle it, run `/discovery:explore` (what the code does) or `/discovery:research` (an external fact) as a lookup ([`context/lookups.md`](context/lookups.md)), each if enabled, otherwise the same work inline; read what a live system holds now (a database, a tracker, a service) directly through its CLI or MCP tool; this dispatch is how the exploration recommendation in "What this skill does NOT do" is carried out, never a request that the user run it. Do not stall the round proving the fact beyond what the recommendation needs: in an engineering session, the Brief's `### Captured assumptions` lists each fact from such a lookup that a recommendation rests on, with its source and the tag `verify during build`. A fact only the user's environment or an unconnected system holds still takes the self-answer step's offer, and a decision is still the user's. ## What this skill does NOT do diff --git a/plugins/planning/skills/interview/context/assumption-sweep.md b/plugins/planning/skills/interview/context/assumption-sweep.md index 9884f7eff1c..0fdcceb51a3 100644 --- a/plugins/planning/skills/interview/context/assumption-sweep.md +++ b/plugins/planning/skills/interview/context/assumption-sweep.md @@ -31,6 +31,9 @@ Hand it: Withhold the rationale that sold each recommendation. The sweep checks what would be locked, not why it was recommended. +Ask it for every finding, with no severity, confidence or blocking cut: it reports, and the +disposition pass below filters. + ## What it looks for - **Undecided details:** a decision the draft relies on that no register row covers. @@ -80,8 +83,9 @@ Fields: ## Disposition -You write the results, not the sub-agent, so register ids stay contiguous. -`S` never reaches the register. +Disposition is the filter pass: you decide what each reported item becomes, and you write the +results, not the sub-agent, each new row taking the next id from `round.sh next-id` (SKILL.md, the +`Q` numbering paragraph). `S` never reaches the register. - **fact and tenant:** run the self-answer step first ([`self-answer.md`](../../../context/self-answer.md)), connected and connectable sources @@ -100,16 +104,17 @@ You write the results, not the sub-agent, so register ids stay contiguous. - **decision:** never settled from facts, whatever the session believes was decided earlier, so a sweep's `decision` item always takes the next line. When a quoted prior owner ruling covers it, that row is written `answered` with `answered-from-decision:` (loop.md "Self-answer gate"). -- **decision, person, and an unanswered tenant or fact:** a register row at the next contiguous - `Q`, written `open` and +- **decision, person, and an unanswered tenant or fact:** a register row at the next id, written + `open` and asked in the next round with a recommendation. A `person` item the user cannot answer, and a `tenant` item when the user is not the setting's owner, are `deferred` to the Brief's `### Deferred questions` with its arbiter tag. -- **contradiction:** one row naming both sides, asked as a choice between them. +- **contradiction:** one row naming both sides, asked as a choice between them; the question + names each side's row by id and plain name (`Q2 (who may export)`) and says why both cannot hold. - **hedged:** no new row. The rows go to the confirmation restate. - **coined-term:** a register row asking the user to accept or rename the term, with the closest existing name in the user's vocabulary as the recommendation, and naming the existing term it - overlaps. With no existing name close to it, ask it withheld, the term defined in one line (SKILL.md "Agree the terms before the first decision"). An + overlaps. With no existing name close to it, ask it withheld, the term defined in one line (SKILL.md "Agree a term where it first appears"). An accepted or renamed term is written to the glossary; a rename also replaces the term in the draft and in the resolutions of the rows that carry it, and the next sweep covers those rows. - **unstated-value:** a register row asking the user to confirm the literal value, whoever owns diff --git a/plugins/planning/skills/interview/context/composition.md b/plugins/planning/skills/interview/context/composition.md index 1dfe4f93be1..5efe8aeba63 100644 --- a/plugins/planning/skills/interview/context/composition.md +++ b/plugins/planning/skills/interview/context/composition.md @@ -13,10 +13,11 @@ otherwise do the step inline or name it to the user. | Territory unfamiliar to the user | `/discovery:blindspot` (if enabled) | Before round 1; surfaces the user's unknowns so they can assess options | | A question rests on why something was built this way | `/discovery:trace-intent` (if enabled) | Recovers the original rationale from evidence outside the code before a recommendation leans on it | | The topic has been attempted before | `/session-flow:recall` (if enabled) | At the survey: what past sessions tried, kept and reverted, so rounds do not re-litigate it | -| Need codebase grounding | `/discovery:explore` (if enabled) | Reads PLAN.md Brief as scope | -| Need external evidence | `/discovery:research` (if enabled) | Reads PLAN.md Brief as scope; its deep tier covers a multi-source sweep | +| A fact in the repository needs more than a quick read | `/discovery:explore` (if enabled) | A lookup during the interview ([`lookups.md`](lookups.md)): its brief carries the questions it feeds, the constraint ledger verbatim and the settled decisions | +| A fact outside the repository needs more than a quick read | `/discovery:research` (if enabled) | A lookup during the interview ([`lookups.md`](lookups.md)), briefed the same way; its deep tier covers a multi-source sweep | +| Each round's recommendations | the plugin's `interview-recommender` agent | Dispatched once per round with decision questions, blind to the conversation (SKILL.md Step 2) | | A question needs something built to react to | `/prototype:explore-directions` (look and feel) or `/prototype:pressure-test` (logic, state, data shape) (if enabled) | Mid-interview detour: confirm scope and checkpoint, build the throwaway, react to it, return and answer in one line | -| A domain term resolves, or is used two ways | `/domain-driven-design:ubiquitous-language curate` (if enabled) | Inline vocabulary update between questions (SKILL.md Stance, item 4) | +| Every interview, and a term the user confirms | `/domain-driven-design:ubiquitous-language` (if enabled) | `lookup` once the interview starts; `curate` for each term after the user confirms its restatement (SKILL.md Stance, item 4) | | Domain events, processes or context boundaries are unclear | `/event-storming:methodology` or `/event-storming:simulation` (if enabled) | Detour to model the domain; resolved terms and boundaries come back as answers | | Who the users are, or how a flow should work, is unsettled | `/user-experience:shape` or `/user-experience:evaluate` (if enabled) | `shape` grounds users and flows before acceptance criteria; `evaluate` checks an existing flow or prototype | | Acceptance criteria for an LLM-based feature | `/evals:design` (if enabled) | Turns the criteria into measurable success criteria and eval cases | @@ -45,6 +46,6 @@ Eval evidence: case `interview-scope-cut-moscow`. ## Mid-interview composition -In `me` mode, research, exploration, and prototyping are not only downstream. Invoke them *during* the interview when a recommendation needs external/codebase grounding or when a question needs something built to react to; when branches outgrow the session, recommend routing the continuation with `/session-flow:workflow` (SKILL.md Step 5). Return to the open branch after. **The question asking cannot settle:** when a look, feel, or behavior question cannot be answered by asking. The user has to see the thing to know what they think. Stop asking and route it to a throwaway prototype: `/prototype:explore-directions` (look and feel) or `/prototype:pressure-test` (logic, state, data shape), invoked via the Skill tool if enabled. Those skills gate their own model-initiated entry, so confirm the spike's scope with the user and checkpoint this interview before handing off. The detour leaves an active workflow. Checkpointing is persistence the interview already owes: the question registered `open` at ask-time, plus the ledger and Brief written per lock-in, which is what lets the open branch survive the detour. Then return to the open branch and resolve its question with a one-line answer. The prototype is the instrument that produces the answer, not a deliverable; it is thrown away. +Research and exploration run *during* the interview, when the gap triage or a later round finds a fact a quick read does not settle; when each runs and what it holds is in [`lookups.md`](lookups.md). In `me` mode, prototyping is not only downstream either: invoke it when a question needs something built to react to; when branches outgrow the session, recommend routing the continuation with `/session-flow:workflow` (SKILL.md Step 5). Return to the open branch after. **The question asking cannot settle:** when a look, feel, or behavior question cannot be answered by asking. The user has to see the thing to know what they think. Stop asking and route it to a throwaway prototype: `/prototype:explore-directions` (look and feel) or `/prototype:pressure-test` (logic, state, data shape), invoked via the Skill tool if enabled. Those skills gate their own model-initiated entry, so confirm the spike's scope with the user and checkpoint this interview before handing off. The detour leaves an active workflow. Checkpointing is persistence the interview already owes: the question registered `open` at ask-time, plus the ledger and Brief written per lock-in, which is what lets the open branch survive the detour. Then return to the open branch and resolve its question with a one-line answer. The prototype is the instrument that produces the answer, not a deliverable; it is thrown away. `/planning:interview` is sister to `/planning:plan`: one resolves *what*, the other resolves *how*. They share the topic slug, share the directory, feed each other. diff --git a/plugins/planning/skills/interview/context/gotchas.md b/plugins/planning/skills/interview/context/gotchas.md index acb7123f75c..8ee2834fb9f 100644 --- a/plugins/planning/skills/interview/context/gotchas.md +++ b/plugins/planning/skills/interview/context/gotchas.md @@ -24,7 +24,7 @@ Failure patterns from real sessions. Loaded on demand from `/planning:interview` - **Assuming `lock` never needs a register:** `lock` always writes its problem row, and its STOP-on-gap and the unattended ladder both produce unresolved questions, and a question outside the register is a question outside the gate. -- **Treating the register gate's exit 2 as a pass:** ungradeable means the check could not see the state (missing register, gapped `Q`, a deferred row absent from the Brief), which is when a silent hole is most likely, not least. +- **Treating the register gate's exit 2 as a pass:** ungradeable means the check could not see the state (missing register, an undeclared gap in `Q`, a deferred row absent from the Brief), which is when a silent hole is most likely, not least. - **A blocking question fired mid-phase:** a gate that lands after the caller's phase is underway idles a lane nobody is watching. Emit the open set at the phase boundary; justify the exception in one line. diff --git a/plugins/planning/skills/interview/context/handoff.md b/plugins/planning/skills/interview/context/handoff.md index 5d99ea6f916..99d31644f1b 100644 --- a/plugins/planning/skills/interview/context/handoff.md +++ b/plugins/planning/skills/interview/context/handoff.md @@ -7,9 +7,7 @@ is the placeholder SKILL.md "Spoke paths" defines. - **Product intent still unsettled after confirmation (who the users are, how success is measured)** → `/planning:prd`, which locks it before the engineering steps below -- **Code change with unknowns about the codebase** → clear context, then codebase exploration (`/discovery:explore` if enabled. It reads the Brief as scope) -- **Code change relying on external libs/APIs/best-practices** → external research (`/discovery:research` if enabled) -- **Already understand the codebase and the externals** → `/planning:design`, whose handoff writes PLAN.md's `## Design` before `/planning:plan`. Design-significant work (new types or contracts, a new module or library, a package-topology, cross-module, or data-model change) gets its design rounds there; work with no design question open records the one-line early exit in one step. **When the task is small, not design-significant, and the contract IS the plan**, proceed directly to implementation (`/implementation:implement` if enabled) instead; small design-significant work still goes through `/planning:design` first +- **Code change** → `/planning:design`, since exploration and research already ran as lookups during the interview ([`lookups.md`](lookups.md)) and run no second pass here. `/planning:design`'s handoff writes PLAN.md's `## Design` before `/planning:plan`. Design-significant work (new types or contracts, a new module or library, a package-topology, cross-module, or data-model change) gets its design rounds there; work with no design question open records the one-line early exit in one step. **When the task is small, not design-significant, and the contract IS the plan**, proceed directly to implementation (`/implementation:implement` if enabled) instead; small design-significant work still goes through `/planning:design` first - **The outcome feeds a prototype** → one prototype per question the Brief's `Prototype proves:` criteria name, each handed to the prototype skill that fits it and built in its own worktree, so they run at once ([`prototype-questions.md`](prototype-questions.md)) - **The approved Brief is one unit of work that a later session, an AFK lane or another person picks up** → `/work-items:track add --agent-ready`, one tracker item carrying the Brief's acceptance criteria and any design part it already holds; the session that picks it up runs design and plan from there - **Interview outgrew one session (branches keep opening faster than they close, or the user or harness asks for a pause)** → route the next step with `/session-flow:workflow` (if enabled), handing it the ledger and Brief paths and the first open branch; without session-flow, follow the pointer below. The ledger + Brief survive; resume continues from the first open branch. Before the first resumed round, resolve the surface through `` again (never an absolute path copied from a save-point) and run `round.sh --dir '//interview-surface' doctor --ledger `, which lists what the running version needs and the ledger lacks ([`surface.md`](surface.md) "Resume"). When it outgrew the session because the effort is too big AND still foggy, recommend `/planning:wayfind` instead (SKILL.md Stance), which charts from the same ledger diff --git a/plugins/planning/skills/interview/context/lookups.md b/plugins/planning/skills/interview/context/lookups.md new file mode 100644 index 00000000000..b22e64ed4b5 --- /dev/null +++ b/plugins/planning/skills/interview/context/lookups.md @@ -0,0 +1,113 @@ +# Gap triage, lookups and held questions + +Read this after the Step 1 survey, before round 1, and again whenever a lookup starts or returns. +It is the one home for the gap triage, the intake round, when a lookup runs and what it is +handed, which questions wait for one, and what a returned fact that contradicts an answer does. +`` is the placeholder SKILL.md "Spoke paths" defines. + +## Gap triage before round 1 + +List every unknown the task depends on and sort each into exactly one kind: + +- **Lookup:** a fact a source can answer, by the self-answer step + ([`self-answer.md`](../../../context/self-answer.md)). It is never asked: it runs as "When a + lookup runs" below says, and its result is stated with its source. +- **Discovery question:** what only the user knows and no evidence can propose: the problem and + why it matters to them, who it is for, what is in and out of scope, where they start, where it + heads, and the quality bar when the request does not state it. It is asked open (SKILL.md "A + discovery question carries no recommendation"). +- **Decision question:** a choice with a live alternative, asked with a recommendation once the + discovery answers and lookups it rests on are in. + +A decision the evidence leaves one viable answer is settled by evidence, not asked (SKILL.md "Name +the evidence-settled decisions in the round header"). An unknown that needs a fact before it is a +choice is a lookup now and a decision question once the lookup returns. + +## The intake round + +When the triage leaves discovery questions, round 1 is the intake round. It asks only those, each +registered `open` with `kind:: discovery`, and keeps the round header and the closing probe of any +round. Decision questions wait for round 2, because their recommendations rest on the intake +answers. + +Three things still go out in round 1 and are not intake questions: + +- a routing suggestion: brainstorm, prd or blindspot by SKILL.md "Intake the starting point", and + the wayfind question of [`routing.md`](routing.md) "Round 1", which keeps its recommendation + because it routes the work rather than answering for the user; +- a candidate a lookup found for an unknown Problem field, stated with its source and a request to + confirm or correct it ([`problem-framing.md`](../../../context/problem-framing.md) "Candidates + for an unknown field"); +- a fact a lookup settled, stated with its source. + +When the triage leaves no discovery question (the request already states the problem, the users +and the scope), there is no intake round: round 1 is the first decision round. An empty intake +round asks nothing, so it is never sent. + +## When a lookup runs and what it is handed + +Each lookup is named by what it reads: + +- **The repository:** `/discovery:explore` through the Skill tool when that skill is listed; + otherwise Grep, Read and Glob inline. +- **Anything else** (vendor docs, a standard, current practice): `/discovery:research` when that + skill is listed; otherwise a web search and fetch inline. +- **A live system** (a database, a tracker, a service): read it directly through its CLI or MCP + tool. + +A lookup one Grep or Read settles runs inline at once, whatever it reads. + +Dispatch the triage's lookups before round 1 is sent, so they run while the user answers it. A +lookup a later round needs is dispatched when that round is drafted. Run at most one dispatch per +lookup skill at a time: group the lookups for one skill into one brief, and a lookup needed while +that skill's dispatch is running joins the next brief, sent when the running one returns. + +Hand each brief: + +- each question the lookup feeds, in words, and what about it the result would settle; +- the constraint ledger verbatim (SKILL.md "Composed artifacts are candidates"); +- the settled decisions it must respect, each with its answer; +- what to return: each fact with its source, a `file:line` or a URL read in that run. + +Never hand it the Brief to read as its scope, or a draft recommendation: the lookup finds facts, +and the recommender ([`../../../agents/interview-recommender.md`](../../../agents/interview-recommender.md)) +draws the recommendation from them. + +With no explore or research skill listed, the lookup is an inline read done before the round is +sent, so no question waits for it. + +## Held questions + +A question whose answer, recommendation or option set a running lookup could change is held until +that lookup returns. A lookup the user says is running elsewhere (another session, a colleague +checking) holds the same way. Ask every other question now: the hold never holds the round +([`loop.md`](loop.md) "Out-of-band drift"). + +A held question takes its id and an `open` row whose resolution reads `waits on: ` when +the round header's `Held:` line first names it (SKILL.md "Name the held questions in the round +header"). It is not asked and gets no recommendation until the lookup returns; then it is asked in +full in the next round, in the place [`loop.md`](loop.md) "Frontier rounds" item 2 gives it, and +leaves the `Held:` line. + +## A returned fact that contradicts an answer + +A fact from any source (a lookup's return, the user relaying one, a later read) that contradicts an +`answered` row reopens that row under its own number. Never change the answer yourself, and never +ask the question again under a new id: + +1. Write the fact and its source on one line to `//reopen-Q.txt`. +2. Run, on one line: `bash "/surface/round.sh" reopen --ledger //interview-checklist.md Q --round --fact-file //reopen-Q.txt`, + `` being the round that asks it again. It sets the row `open` with `reopened::`, moves the + old answer and the fact to the ledger's `## Reopened answers`, and writes `recheck:: Q` on + every row whose `depends::` reaches it, printing `marked Q` for each. +3. Ask `Q` first in round ``, its Context showing the old answer and the contradicting fact + with its source, and name each marked row by id and plain name as resting on it, to be + rechecked once `Q` is answered again. +4. Once the user has reconfirmed or changed a marked row, run + `bash "/surface/round.sh" reopen --ledger //interview-checklist.md --rechecked Q`. + The register gate blocks while any `recheck::` tag remains. + +With no Bash, make the same ledger edits by hand, by the rules of `reopen` in +[`../../../surface/README.md`](../../../surface/README.md). A fact that contradicts the +recommendation of a question not yet answered is loop.md "Out-of-band drift" outcome 2, not a +reopen. diff --git a/plugins/planning/skills/interview/context/loop.md b/plugins/planning/skills/interview/context/loop.md index 02a32320ecf..834477014f0 100644 --- a/plugins/planning/skills/interview/context/loop.md +++ b/plugins/planning/skills/interview/context/loop.md @@ -160,15 +160,15 @@ Take a team adding rate limits to a public API. Whether limits count per key or Run rounds until the stop condition is met. Each round: -1. **Restate the working understanding** in two or three sentences: what is decided, what branches remain open. This restate doubles as the **session-hop anchor**: after a handoff, resume, or long gap it re-establishes the decided set and the current round's stakes before any question, so a returning reader (or a fresh session resuming from the ledger) is grounded without re-reading the whole ledger. It never stands in for a question's own Context block (SKILL.md "Relentless mode"). When a question is held for an in-flight lookup, the header ends with the `Held:` line naming each held question and what it waits on (SKILL.md "Name the held questions in the round header"). When the evidence leaves a decision only one viable answer, the header carries the `Settled by evidence:` line naming it, its answer and its Basis (SKILL.md "Name the evidence-settled decisions in the round header") -2. **Compute the frontier:** list the open decisions and drop any whose framing or option set still hangs on an unanswered question, one asked in THIS round included; a dropped question comes back once its input is answered. Carry-overs first: questions unanswered from the previous round re-surface at the top, labeled as such -3. **Self-answer gate per frontier question:** run the self-answer step ([`self-answer.md`](../../../context/self-answer.md)) over prior owner decisions, codebase, docs and history, connected tools, and research. A fact a source answers is STATED with its source tag, not asked, and its dependents join the frontier now (see "Self-answer gate" below). A slow lookup (deep exploration, external research) follows SKILL.md "In-flight lookups" -4. **Ask the frontier as one numbered set:** each question opening with its Context block and carrying a recommended answer and its evidence: each fact it rests on, in the Context block with its source (a `file:line`, a URL fetched this session, a session-transcript finding, or tool output), the recommendation's reason saying why those facts together lead to this answer for this user, and a `Basis: verified` line naming the sources the answer rests on (SKILL.md "Recommended answers"); and, when a condition would change this recommendation, one `Flips if:` line naming it, checkable against a fact the user can confirm ("Flips if: partners share keys across regions"), in addition to the round's closing probe (eval evidence: `/evals/interview-recommendation-basis-and-flip/`). A `judgment` label alone is no basis here, consequential or not: a question with no fact to cite is researched first (the self-answer gate in item 3, and SKILL.md "Ground before recommending" for a consequential one), and one research cannot settle is asked open with a `Withheld:` line naming the evidence that would settle it instead of a recommendation. Order within the round by blast radius, so the answer that would change the most downstream work goes first. Each recommendation and alternative names its origin: the user, a named source with its link, or new (ours). A round that spans several topics groups its questions under short topic headers, blast radius ordering each group. Under `single` pacing the same ordered frontier goes out one question per message, with no topic header (SKILL.md "Group the round by topic" and "One question at a time") +1. **Restate the working understanding** in two or three sentences: what is decided, what branches remain open. This restate doubles as the **session-hop anchor**: after a handoff, resume, or long gap it re-establishes the decided set and the current round's stakes before any question, so a returning reader (or a fresh session resuming from the ledger) is grounded without re-reading the whole ledger. It never stands in for a question's own Context block (SKILL.md "Relentless mode"). When a question is held for a running lookup ([`lookups.md`](lookups.md) "Held questions"), the header ends with the `Held:` line naming each held question and what it waits on (SKILL.md "Name the held questions in the round header"). When the evidence leaves a decision only one viable answer, the header carries the `Settled by evidence:` line naming it, its answer and its Basis (SKILL.md "Name the evidence-settled decisions in the round header") +2. **Compute the frontier:** list the open decisions and drop any whose framing or option set still hangs on an unanswered question, one asked in THIS round included; a dropped question comes back once its input is answered. Order the round: questions a returned fact reopened first ([`lookups.md`](lookups.md) "A returned fact that contradicts an answer"), then questions unanswered from the previous round, labeled as such, then held questions a returned lookup released, then new questions by blast radius +3. **Self-answer gate per frontier question:** run the self-answer step ([`self-answer.md`](../../../context/self-answer.md)) over prior owner decisions, codebase, docs and history, connected tools, and research. A fact a source answers is STATED with its source tag, not asked, and its dependents join the frontier now (see "Self-answer gate" below). A fact that needs more than a quick read is a lookup: when it runs and which questions it holds are in [`lookups.md`](lookups.md) +4. **Ask the frontier as one numbered set:** each question opening with its Context block and carrying a recommended answer, drafted by the blind recommender (SKILL.md Step 2, "One blind recommender drafts each round's recommendations"; a discovery question carries its `Discovery:` line instead), and its evidence: each fact it rests on, in the Context block with its source (a `file:line`, a URL fetched this session, a session-transcript finding, or tool output), the recommendation's reason saying why those facts together lead to this answer for this user, and a `Basis: verified` line naming the sources the answer rests on (SKILL.md "Recommended answers"); and, when a condition would change this recommendation, one `Flips if:` line naming it, checkable against a fact the user can confirm ("Flips if: partners share keys across regions"), in addition to the round's closing probe (eval evidence: `/evals/interview-recommendation-basis-and-flip/`). A `judgment` label alone is no basis here, consequential or not: a question with no fact to cite is researched first (the self-answer gate in item 3, and SKILL.md "Ground before recommending" for a consequential one), and one research cannot settle is asked open with a `Withheld:` line naming the evidence that would settle it instead of a recommendation. Order within the round by blast radius, so the answer that would change the most downstream work goes first. Each recommendation and alternative names its origin: the user, a named source with its link, or new (ours). A round that spans several topics groups its questions under short topic headers, blast radius ordering each group. Under `single` pacing the same ordered frontier goes out one question per message, with no topic header (SKILL.md "Group the round by topic" and "One question at a time") 5. **Capture the answers.** In `auto` and `lock`, hold the Brief draft in a scratch buffer and write it at Step 4; in `me` mode, persist each answer to the ledger and Brief the moment it locks (see Incremental persistence). The open-question register is written earlier still, at ask-time, in every mode, except for the acceptance-criteria coverage prompt, which gets no row ([`coverage-prompt.md`](coverage-prompt.md)). Partial replies are normal: resolve what was answered, keep the rest OPEN, and never default an unanswered question to its recommendation. Honor accept-shorthands ("accept all recommendations", "yes to Q5 to Q7"); a hedged reply is not an accept-shorthand (see "Hedged flag") 6. **Recompute the tree.** What subtrees did these answers eliminate? What new branches opened? Which blocked questions just joined the frontier? Name what was pruned -7. **Domain check:** when the task touches domain concepts, run the glossary challenge (its two triggers, any domain term the user uses two ways and any term that collides with an existing definition, each get a probe) and scenario exploration (a made-up concrete case that tests where one concept stops and the next starts). **Engineering sessions only, per `domain_docs` (`ask` asks first; `off` records instead of writing):** when a term resolves, invoke `/domain-driven-design:ubiquitous-language curate` via the Skill tool for the inline vocabulary update if that plugin is enabled, else record the term and its definition in the resolution field of the open-question register row that resolved it, since a general session writes no repo docs (SKILL.md "Domain-aware behaviors") +7. **Domain check:** when the task touches domain concepts, run the glossary challenge (its two triggers, any domain term the user uses two ways and any term that collides with an existing definition, each get a probe) and scenario exploration (a made-up concrete case that tests where one concept stops and the next starts). Restate each term this round's answers settled and mark it `agreed` once the user confirms the restatement (SKILL.md "Agree a term where it first appears"). **Engineering sessions only, per `domain_docs` (`ask` asks first; `off` records instead of writing):** for each agreed term, invoke `/domain-driven-design:ubiquitous-language curate` via the Skill tool for the inline vocabulary update if that plugin is enabled, else the ledger's `## Terminology matrix` keeps it, since a general session writes no repo docs (SKILL.md "Domain-aware behaviors") 8. **Decision record** *(engineering sessions only, per `domain_docs`)*: for each decision this round resolved about the system being built (not the task's scope or process) where a real alternative was weighed, and with the `architecture` plugin enabled, invoke `/architecture:record-decision` via the Skill tool, passing the decision, its rationale and the alternatives from the question's `Alternatives:`, and asking it to name the part of its admission test a recorded decision passes, for the closing summary's ADR verdict line. That skill owns and applies the admission test; the interview neither restates nor pre-applies it. When the skill offers to skip a decision that plainly fails the test, that offer is a question: in `me` and `auto`, register and ask it like any other; in `lock` or an unattended run, take the skip and note it in the resolution field of the register row that resolved the decision. Under `ask`, ask first as a registered question; under `off`, or without the `architecture` plugin, record the decision as an ADR candidate in the resolution field of the register row that resolved it. Done when every such decision has an invocation result or a register note (SKILL.md "Domain-aware behaviors") -9. **Frontier empty: sweep.** In `me` and `auto` with a register, run the assumption sweep before Step 3 ([`assumption-sweep.md`](assumption-sweep.md)); an `open` row it adds starts another round +9. **Frontier empty: sweep.** In `me` and `auto` with a register, run the assumption sweep before Step 3 ([`assumption-sweep.md`](assumption-sweep.md)): it reports every finding, and its disposition pass filters them; an `open` row it adds starts another round ### Self-answer gate @@ -248,7 +248,7 @@ Targets that catch the most rework downstream: Each round is one numbered set in prose (surface rules: SKILL.md "Question surface: inline prose by default" and "Question lanes"); its topic headers, `Held:` line and origin labels follow Frontier rounds items 1 and 4. **Per-question template + partial-round resolution: SKILL.md Stance "Relentless mode"** is the single source; it is not duplicated here. The fields every lane's question carries: [`question-shape.md`](question-shape.md). -`Q` is a running counter across the session and across rounds (Q1–Q4 in round one, Q5… in round two, so the depth stays visible). Wait for the round's answers before computing the next round. The closing probe is what invites the user to surface a hidden constraint that would flip a recommendation. Most answers come back as a one-line "all as recommended". That is the format working, not under-questioning. +`Q` is a running counter across the session and across rounds (Q1–Q4 in round one, Q5… in round two, so the depth stays visible), each new id taken from `round.sh next-id` (SKILL.md, the `Q` numbering paragraph). Wait for the round's answers before computing the next round. The closing probe is what invites the user to surface a hidden constraint that would flip a recommendation. Most answers come back as a one-line "all as recommended". That is the format working, not under-questioning. ### Page surface @@ -283,7 +283,7 @@ A question whose recommendation, option or `Commits you to:` part coins a name w ### Terminology matrix -The ledger's `## Terminology matrix` holds one row per sourced term the rounds rely on, built before the first decision question (SKILL.md "Agree the terms before the first decision"): +The ledger's `## Terminology matrix` holds one row per sourced term the rounds rely on, added when the question that first relies on it is registered (SKILL.md "Agree a term where it first appears"): ```markdown | Term | Definition | Source | Status | @@ -292,7 +292,7 @@ The ledger's `## Terminology matrix` holds one row per sourced term the rounds r | cluster step | upstream's grouping of reports that share a fingerprint | vendor/upstream-triage/README.md:8 | proposed | ``` -Status is `proposed` until the user agrees the row, then `agreed`. The agreement question is registered like any other; its resolution names the rows it agreed. A term the user redefines keeps its row with the user's definition and source `user, round `. The shorthand glossary above holds the session's own coinages; the matrix holds every sourced term a reader must share to answer. A coinage gets no row; SKILL.md "Agree the terms before the first decision" says how it is agreed, in a withheld question whose resolution names the definitions it agreed; a definition the user corrects takes a row with source `user, round `. +Status is `proposed` until the user confirms the term's restatement, then `agreed`; only an `agreed` row is written to the glossary, and without the glossary skill the matrix is where the agreed term stays. A term the user redefines keeps its row with the user's definition and source `user, round `. The shorthand glossary above holds the session's own coinages; the matrix holds every sourced term a reader must share to answer. A coinage gets no row; SKILL.md "Agree a term where it first appears" says how it is agreed, in a withheld question whose resolution names the definitions it agreed; a definition the user corrects takes a row with source `user, round `. ### Dialogue + recommendation revision @@ -329,7 +329,7 @@ Tick on resolve. Surface the open set periodically (every few questions, or on r - **Persist per lock-in.** The moment a branch resolves, write the answer to its ledger checkbox + the relevant Brief section, so resolved branches survive a crash, context clear, or overflow. - **Continue on signal, never on self-estimate.** The ledger and partial Brief are already on disk after every lock-in, so nothing needs flushing first. When the user or the harness asks for a pause or branches outgrow the session (never because the conversation feels long), recommend routing the next step with `/session-flow:workflow` (if enabled), handing it the ledger and Brief paths, the first open ledger checkbox and the register's `open` rows, which survive a clear that the transcript does not; without session-flow, see [when your context fills up](https://code.claude.com/docs/en/context-window#when-your-context-fills-up). Before the first resumed round, resolve the surface again through the start command in [`lane-page.md`](lane-page.md) (never an absolute path copied from a save-point) and run `round.sh --dir '//interview-surface' doctor --ledger `. - **Pointer**: when session-flow is not enabled, fetch live. -- **Branch out to ground a recommendation, without blocking the round.** If a question needs more than the lightweight codebase gate, such as external best-practice, library API surface, or deeper exploration, hand the lookup to a sub-agent (or do it inline when fast) as an in-flight lookup (SKILL.md "In-flight lookups"). When the result lands, the unblocked questions join the next round's frontier, grounded in code read this session or an official source fetched this session. Never recommend a consequential technical choice from training recall. +- **Branch out to ground a recommendation, without blocking the round.** If a question needs more than the lightweight codebase gate, such as external best-practice, library API surface, or deeper exploration, run it as a lookup ([`lookups.md`](lookups.md)). When the result lands, the unblocked questions join the next round's frontier, grounded in code read this session or an official source fetched this session. Never recommend a consequential technical choice from training recall. ## The open-question register @@ -367,7 +367,9 @@ Fields: `Q | status | round | question | resolution`. Statuses: **Superseded by plan.** `/planning:plan` moves an `answered` row here when a plan change made after the Brief (a reviewer fix, a research update, a stress-test mitigation) replaces the user's answer. Only an explicit reply naming that row moves it out; a blanket "approve" of the plan does not. Reconfirm: `answered` with `proposal:: ; was:: ; answer:: accepted: `, each value escaped as the exporter escapes it (a literal `;`, `\` or `|`; the grammar is in `surface/exporters.py`). Reject: the row returns to its original `answered` text and the plan drops the change. This is neither the page's `supersededBy` (a question replaced by a newer one, exported as `withdrawn`) nor the superseded recommendation of "Out-of-band drift" (an interview recommendation replaced before the user answered). -`Q` matches the terminal numbering, runs continuously across rounds, and never has a gap. A gap means a row was dropped after it was written, and the gate refuses to grade a register with one. +`Q` matches the terminal numbering, runs continuously across rounds, and never has an undeclared gap. Every new id comes from `round.sh next-id` (SKILL.md, the `Q` numbering paragraph), so no id is reused, no gap is filled and no row is renumbered. A `- gap | Q-Q | ` line in this section declares ids that intentionally have no row (written by hand for a legacy ledger or a shared session whose ids ran ahead, or by `export-ledger` for ids never assigned); any other gap means a row was dropped after it was written, and the gate refuses to grade a register with one. + +**Register tags.** A resolution may end in a tail of tags, after the answer and before any `stale::`, in this order: `kind:: discovery` on a discovery question (a row without it is a decision question); `problem:: P`, the `## Problems` row the question serves ([`routing.md`](routing.md) "Problems"); `depends:: Q, Q`, the earlier questions its framing or options rest on, written when it is registered so a reopen can find it; and `reopened:: round ` and `recheck:: Q`, which `round.sh reopen` writes ([`lookups.md`](lookups.md) "A returned fact that contradicts an answer"). A row with no answer yet starts its tail at once: `- Q2 | open | round 1 | Who is this for? | kind:: discovery`. The ledger's `## Problems` section holds `- P | | proposed|agreed`, and its `## Reopened answers` section holds what `reopen` moved; neither is a register row. **Write each row in the shape the gate grades.** In the terminal lane, write each row this way when you register it, so the gate passes without a repair: @@ -400,15 +402,16 @@ Two shapes of restate, both one line: *"Still open: Q3 (content format)"* when t The drift check above fires on a user reply. A round can also be overtaken by output the user did not write: a dispatched sub-agent's return, a background task notification, an agent-team member's report, a Monitor firing, a permission prompt raised by a sub-agent. The list is open, because the test is *non-user content reaching the transcript while a round is open*, not membership of a named set. This is the ordinary consequence of the non-blocking dispatch rule ("Codebase gate per frontier question" above, and "Branch out to ground a recommendation"), so it is expected traffic, never an anomaly. -**When such output lands, check it against the register's `open`, `discussing` and `superseded-by-plan` rows before continuing.** A `superseded-by-plan` row it touches is restated, never resolved by the return. A `discussing` row it touches stays `discussing`: a contradicted recommendation is restated as revised, and the discussion is never read as an answer. The trigger is RELEVANCE, not arrival. Three outcomes, and most returns take the first: +**When such output lands, check it against the register's `open`, `discussing` and `superseded-by-plan` rows, and its `answered` rows, before continuing.** A `superseded-by-plan` row it touches is restated, never resolved by the return. A `discussing` row it touches stays `discussing`: a contradicted recommendation is restated as revised, and the discussion is never read as an answer. The trigger is RELEVANCE, not arrival. Four outcomes, and most returns take the first: 1. **It touches no open row.** Say so in one line and leave the round alone. The questions stand as asked. 2. **It contradicts a recommendation under a question already asked.** Restate that question, naming the superseded recommendation as superseded and giving the replacement its own basis. A recommendation the session has since disproved is worse than no recommendation, because the user is answering against it. 3. **It answers an open row from the environment.** Resolve it and STATE the answer; do not leave it standing as a question. "Facts are yours; decisions are the user's" does not stop applying because the fact arrived late. +4. **It contradicts an answered row.** Run `round.sh reopen` on that row, which reopens it under its own number and marks every row resting on it for recheck, and ask it first in the next round ([`lookups.md`](lookups.md) "A returned fact that contradicts an answer"). Never change the answer yourself. -**Restate every open row in full.** After a return, the first message that presents a question before the user's next separate reply ends with one block holding every `open`, `discussing` and `superseded-by-plan` row in the full question shape (Context, recommendation with `Basis:`, alternatives), with any row that moved marked as changed. This holds whether outcome 2 or 3 moved a row or the return released a question held for it, and when that message also processes a user reply that shared the turn. A row still held for a pending result may be named in one line as waiting. The user reads the latest message: a one-line pointer to an earlier block sends them scrolling to rebuild the open set, and with several returns that set spreads across messages. Outcome 1 that releases no held question stays one line, because it presents nothing. Rule: `docs/contracts/async-round-presentation/README.md` in the marketplace. +**Restate every open row in full.** After a return, the first message that presents a question before the user's next separate reply ends with one block holding every `open`, `discussing` and `superseded-by-plan` row in the full question shape (Context, recommendation with `Basis:`, alternatives), with any row that moved marked as changed. This holds whether outcome 2, 3 or 4 moved a row or the return released a question held for it, and when that message also processes a user reply that shared the turn. A row still held for a pending result may be named in one line as waiting. The user reads the latest message: a one-line pointer to an earlier block sends them scrolling to rebuild the open set, and with several returns that set spreads across messages. Outcome 1 that releases no held question stays one line, because it presents nothing. Rule: `docs/contracts/async-round-presentation/README.md` in the marketplace. -**This does not hold the round.** The scoped barrier stays exactly as it is (SKILL.md "In-flight lookups"): of the frontier questions, those that need a pending result are held and the others are asked, and a question off the frontier keeps waiting on its unanswered input. Holding a round until every dispatch drains would trade this failure for a serialized interview. +**This does not hold the round.** The scoped barrier stays exactly as it is ([`lookups.md`](lookups.md) "Held questions"): of the frontier questions, those that need a pending result are held and the others are asked, and a question off the frontier keeps waiting on its unanswered input. Holding a round until every dispatch drains would trade this failure for a serialized interview. **The floor is the next user reply.** Whether the harness gives you a turn when out-of-band output lands is not something to build correctness on. Acting the moment the output lands is the improvement, not the requirement. diff --git a/plugins/planning/skills/interview/context/routing.md b/plugins/planning/skills/interview/context/routing.md index 1d027eb2f60..a62a5d2c041 100644 --- a/plugins/planning/skills/interview/context/routing.md +++ b/plugins/planning/skills/interview/context/routing.md @@ -2,7 +2,7 @@ Reference detail for the `## Routing by size and fog` section of `SKILL.md`. Read it at entry and at every round boundary: it owns the entry gate's thresholds, the round-1 wayfind question, -the bigger-than-you-thought check, what the handoff to `/planning:wayfind` carries, and the "this +the problems list, the round header's scope-check line, the bigger-than-you-thought check, what the handoff to `/planning:wayfind` carries, and the "this is small, ship it" exit. ## Entry: diff size and retry cost, not behavior change @@ -33,7 +33,9 @@ stays the entry either way: - Ask it as a registered round-1 question in the full layout, worded so yes accepts ("Should this effort move to `/planning:wayfind chart` now?"), its Context naming each unknown area in the user's words. Round 1's other questions go out in the same message; never hold them for this - answer, and never switch on your own. + answer, and never switch on your own. In an intake round it is the one question with a + recommendation, since it routes the work rather than answering for the user + ([`lookups.md`](lookups.md) "The intake round"). - Accepted: write the ledger as "The handoff to wayfind" below says, and name `/planning:wayfind chart` with the ledger path for the user to run. - Declined: record the decline as the row's resolution and carry on with the rounds. Raise wayfind @@ -45,6 +47,31 @@ stays the entry either way: Unknowns in two areas or fewer, or unknowns the request does not state, are not this trigger: fog that surfaces during the rounds is the fog trigger's. +## Problems + +The problems the work serves are agreed in round 1: from the intake answers, or from the request +when it states them. Write each to the ledger's `## Problems` section as +`- P | | proposed`, numbered from P1 in the order they were given, restate +them at the top of the next round, and mark each `agreed` once the user confirms it. Each question +registered after that carries `problem:: P` for the problem it serves (loop.md "Register tags"), +and every line the user reads pairs the code with its words (`problem P2 (late invoices)`). + +A problem set that will not fit one session goes to `/planning:wayfind chart` only through the +triggers below, which stay too big AND too foggy: the problem count is evidence for them, never a +trigger of its own. + +## The scope-check line + +Every round header, the intake round's included, carries one `Scope check:` line. Before writing +round ``'s file, run +`bash "/surface/round.sh" scope-check --ledger //interview-checklist.md --round --limit --rounds-dir /`, +`` being the value resolved below, and put its output on that line: +the questions registered against the limit, and what each of the last two rounds added and +settled. Those counts are the measured half of the question-count pointer and the fog trigger +below; the fog trigger still counts the branches the next round would add. With no Bash, count the +same from the ledger by the rules of `scope-check` in +[`../../../surface/README.md`](../../../surface/README.md) and write the line yourself. + ## Bigger than you thought? Check at every round boundary After each reply is recorded and before the next round is written, answer one question: did this diff --git a/plugins/planning/skills/interview/context/term-clash.md b/plugins/planning/skills/interview/context/term-clash.md index 0354ad59326..e16b2fa81e0 100644 --- a/plugins/planning/skills/interview/context/term-clash.md +++ b/plugins/planning/skills/interview/context/term-clash.md @@ -35,16 +35,17 @@ registered question in the full layout: never meet in one sentence. Add the qualified name to the ledger's `## Terminology matrix` as a `proposed` row in the same -round, citing the clash as its source; it becomes `agreed` on the user's answer. Until then, no -other question uses the qualified name as settled. +round, citing the clash as its source; it becomes `agreed` once the user confirms its +restatement (SKILL.md "Agree a term where it first appears"). Until then, no other question uses +the qualified name as settled. ## Write the agreed name -Once the user agrees, write it through the route SKILL.md already names for a resolved term, the -inline vocabulary update (`/domain-driven-design:ubiquitous-language curate` when that plugin is -enabled; otherwise the term and its definition go in the resolution field of the register row that -resolved it), under the resolved `domain_docs` value: `auto` writes it, `ask` asks once before -writing, `off` records it in the register row only. Hand the writer the qualified name, its +Once it is `agreed`, write it with the other agreed terms, through the route SKILL.md names for +an agreed term, the inline vocabulary update (`/domain-driven-design:ubiquitous-language curate` +when that plugin is enabled; otherwise the ledger's `## Terminology matrix` keeps it), under the +resolved `domain_docs` value: `auto` writes it, `ask` asks once before +writing, `off` keeps it in the matrix only. Hand the writer the qualified name, its definition, the existing concept it is distinguished from with that concept's source, and the bare term as the ambiguous form to avoid. A general session writes nothing to the repository and keeps -the agreed name in the register row and the shared-understanding summary. +the agreed name in the matrix and the shared-understanding summary. diff --git a/plugins/planning/skills/interview/templates/checklist.md b/plugins/planning/skills/interview/templates/checklist.md index 0936382c840..4a6058cf06e 100644 --- a/plugins/planning/skills/interview/templates/checklist.md +++ b/plugins/planning/skills/interview/templates/checklist.md @@ -8,12 +8,12 @@ Planning version: -Q | ` line declares it, and `export-ledger --ledger` keeps those lines. `export-ledger` and `sync-ledger` write `- gap | Q-Q | never assigned` for ids nothing ever held, never for one the ledger held, and a gap line past the last row reserves those ids. - **Register tags and `round.sh reopen`.** A register row's resolution can end in `kind::`, `problem::`, `depends::`, `reopened::` and `recheck::` tags, which import and export carry unchanged. `round.sh reopen --ledger Q --round --fact-file ` reopens an answered row under its own number, keeps the old answer and the fact in `## Reopened answers`, and flags every row built on it with `recheck::`, which blocks `check-open-questions.sh` until `reopen --rechecked` clears it. -- **Round detail levels, `scope-check` and `measures`.** `round.sh show --level full|standard|minimal ` prints a saved round at that detail level (full byte for byte; no level hides `Flips if:` or `Commits you to:`), set by the new `question_detail` setting (default `standard`). `round.sh scope-check` prints the counts for a round header's `Scope check:` line, and `round.sh measures` prints rounds, median lines per shown round and reopened answers. `check-round-codes.sh` also fails a question with no single `My recommendation:`, `Withheld:`, `Discovery:` or `Strong answer:` line, and a round header with no `Scope check:` line. +- **Round detail levels, `scope-check` and `measures`.** `round.sh show --level full|standard|minimal ` prints a saved round at that detail level (full byte for byte; no level hides the recommendation with its reason, `Withheld:`, `Discovery:`, `Second view:`, `Departs from:`, `Flips if:` or `Commits you to:`), set by the new `question_detail` setting (default `standard`). `round.sh scope-check` prints the counts for a round header's `Scope check:` line, and `round.sh measures` prints rounds, median lines per shown round and reopened answers. `check-round-codes.sh` also fails a question with no single `My recommendation:`, `Withheld:`, `Discovery:` or `Strong answer:` line, and a round header without exactly one `Scope check:` line. - **Interview flow: gap triage, held questions, a blind recommender and stable ids.** Before round 1, `/planning:interview` sorts every unknown into a lookup, a discovery question or a decision question; when discovery questions exist, round 1 asks only those, open, each with a `Discovery:` line and no recommendation (new `context/lookups.md`). A question a running lookup could change is held on the `Held:` line, at most one dispatch runs per lookup skill, and a fact that contradicts an answer runs `round.sh reopen`. Each round's recommendations come from the new read-only `interview-recommender` agent, which sees the facts, constraints and settled decisions but not the conversation; a differing draft shows as `Second view:` and a departure from a settled decision as `Departs from:`. New ids come from `round.sh next-id`, problems are numbered P1, P2 and tagged on their questions, the round header's `Scope check:` line comes from `round.sh scope-check`, the end sweep reports every finding for the disposition pass to filter, the glossary is read at the start of every interview and a term is written only after the user confirms its restatement, and `/planning:audit-answers` never auto-accepts a `kind:: discovery` row. ### Removed diff --git a/plugins/planning/scripts/check-round-codes.sh b/plugins/planning/scripts/check-round-codes.sh index e32268ad65a..1cb13bf7cf1 100755 --- a/plugins/planning/scripts/check-round-codes.sh +++ b/plugins/planning/scripts/check-round-codes.sh @@ -54,12 +54,16 @@ Two structure checks run beside the bare-code check: answer line each question block holds exactly one `My recommendation:`, `Withheld:`, `Discovery:` or (the product action) `Strong answer:` line - scope check the lines before the first question hold a `Scope check:` line + scope check the lines before the first question hold exactly one + `Scope check:` line + +A list item (`- Q4: ...`, `1. Q4: ...`) never opens a question. A label may follow a marker (any run of characters outside printable ASCII, such as an emoji) and sit in bold; round.py show reads labels the same way. Each failure prints `missing=:` (`recommendation` or -`single-recommendation` at the question line, `scope-check` at line 1), and +`single-recommendation` at the question line, `scope-check` or +`single-scope-check` at line 1), and the closing line gains ` structure=`. TXT } @@ -169,6 +173,7 @@ LC_ALL=C awk ' } # The id of the question a line opens, or "" when it opens none. function question_id(line, c) { + if (line ~ /^([ \t]*>)*[ \t]*([-*+]|[0-9]+[.)])[ \t]/) return "" if (!match(line, /^[^A-Za-z0-9]*Q[0-9]+[.):]/)) return "" c = substr(line, RSTART, RLENGTH) # Q17.S1 is a qualified scenario id, not a question line. @@ -216,7 +221,7 @@ LC_ALL=C awk ' } } { - if (!block_line && $0 ~ /^[^ -~]*[ \t]*(\*\*|__)?Scope check(\*\*|__)?:/) has_scope = 1 + if (!block_line && $0 ~ /^[^ -~]*[ \t]*(\*\*|__)?Scope check(\*\*|__)?:/) has_scope++ if (block_line && $0 ~ /^[^ -~]*[ \t]*(\*\*|__)?(My recommendation|Withheld|Discovery|Strong answer)(\*\*|__)?:/) block_answers++ } { @@ -234,7 +239,7 @@ LC_ALL=C awk ' END { if (in_fence) exit 4 close_block() - if (!has_scope) { missing++; print "missing=1:scope-check" } + if (has_scope != 1) { missing++; print "missing=1:" (has_scope ? "single-scope-check" : "scope-check") } printf "codes=%d bare=%d status=%s structure=%d\n", n_codes, bare, ((bare || missing) ? "fail" : "ok"), missing exit ((bare || missing) ? 1 : 0) } diff --git a/plugins/planning/scripts/check-round-codes.test.sh b/plugins/planning/scripts/check-round-codes.test.sh index 13a1238b6b8..895299b50f6 100755 --- a/plugins/planning/scripts/check-round-codes.test.sh +++ b/plugins/planning/scripts/check-round-codes.test.sh @@ -305,6 +305,21 @@ round two-recs.md \ 'My recommendation: no.' \ 'Withheld: the paging trial has not reported.' run "a question with two answer lines fails" 1 two-recs.md '^missing=2:single-recommendation$' +round q-bullet.md \ + 'Scope check: 2 of 27 questions registered' \ + '' \ + '❓ Q15: **Should a failed retry page the on-call?**' \ + 'Context:' \ + '- Q14: you set the retry budget at three tries.' \ + '1. Q14: the same answer, in a numbered list.' \ + '➡️ My recommendation: no; a retry stays silent.' +run "a list item that opens with Q: is not a question line" 0 q-bullet.md 'bare=0 status=ok structure=0' +round two-scopes.md \ + 'Scope check: 2 of 27 questions registered' \ + 'Scope check: 3 of 27 questions registered' \ + 'Q15: Should a failed retry page the on-call?' \ + 'My recommendation: no.' +run "a header with two Scope check: lines fails" 1 two-scopes.md '^missing=1:single-scope-check$' round product.md \ 'Scope check: 3 of 15 questions registered' \ '' \ diff --git a/plugins/planning/surface/README.md b/plugins/planning/surface/README.md index 6ec1cb54fac..420d7d77684 100644 --- a/plugins/planning/surface/README.md +++ b/plugins/planning/surface/README.md @@ -88,9 +88,9 @@ Every command needs `--dir ''`; there is no default. Every command exi | `import-ledger --ledger F` | Seed an empty data dir from an existing ledger. A round cell reads as the N of a leading `round `, else the cell's first integer, else 1, with a warning naming the row when the leading form is missing; a cell other than the bare `round ` is kept and written back by `export-ledger` while the question's round still equals its number. A row in the `export-ledger` grammar (named fields `hold`, `proposal`, `was` (or `was plain`, `was answer`, which `export-ledger` writes back), `answer`, `note`, `aside`, `commitments`, each escaped as `exporters.py` documents) restores its hold, a superseded-by-plan row's proposal and displaced answer, the decision on any status, its note, a decision a user hold set aside (still set aside), an open row's accept that a revised recommendation set aside (still set aside, as of the text it accepted) and every commitment with its confirmed or unconfirmed mark; an unknown field or a contradictory pair is refused, naming the field. Two held-row cases have no field and are not restored: a seeded `blocked` row comes back `deferred` (still USER-RESERVED), and the seed text of an open or superseded-by-plan row is lost when its hold set aside an accept or an alternative, whose own note takes `note`. Every older grammar still imports, read as it always was: `waits on::`, `awaiting user::`, `confirmed::`, `plan proposes::` and `answer::` rows, a settled row's `; confirmed::` tail, and the unescaped `waits on:`, `confirmed:` and `; confirmed:` forms. A row's tail of register tags (`kind::`, `problem::`, `depends::`, `reopened::`, `recheck::`, after the answer and before `stale::`) is peeled before any of this parsing and kept in `meta.seededFrom.tags`; `export-ledger` writes it back unchanged, and a merge into a ledger keeps that ledger's own tail | | `next-id --ledger L [--count N]` | Print the next N question ids (default 1), one per line: the highest number L holds plus 1, never a reused or gap-filling id. L holds each register row's `Q`, each `[Q]` lead a renumbered row carries, and the high end of each `- gap \| Q-Q \| ` line that covers no row (one past the last row counts); with `--dir`, also every number the page's merged export writes, every page question's own `Q` id and every pending proposal's `Q`. Needs no data dir; refuses an L without exactly one open-question register heading and a count below 1 | | `reopen --ledger L Q --round R --fact-file P` | Set L's answered row `Q` back to `open` under its own number with `reopened:: round R`, move its answer and the fact in file P (UTF-8, whitespace runs made one space; an empty one is refused) to L's `## Reopened answers` section as `- Q \| round R \| was: \| fact: ` (the section is created after the register's section, its `### Deferred questions` included, when L has none), keeping every row's other tags and `stale::` mark, and add `recheck:: Q` to every row whose `depends::` reaches `Q` at any depth, printing `marked Q` for each. `reopen --ledger L --rechecked Q...` removes `recheck::` from those rows once the user has reconfirmed or changed each; it refuses a row with none. Needs no data dir; same backup and verdict as `supersede`; refuses a row that is not `answered` | -| `show --level full\|standard\|minimal R` | Print the saved round file R at that detail level. `full` prints R byte for byte; `standard` drops `Origin:`, `Basis:`, `Checked against:` and the `Source:` quote's body (its links stay) and puts each alternative on one line; `minimal` also drops `Context:` and `Source:` and prints `Alternatives: (say "more Q")`. The header (the lines before the first question), every other line, the blank lines and the `---` rules stay at every level; a run of blanks a dropped field leaves becomes one. A label may follow a marker emoji and sit in bold. Needs no data dir | +| `show --level full\|standard\|minimal R` | Print the saved round file R at that detail level. `full` prints R byte for byte; `standard` drops `Origin:`, `Basis:`, `Checked against:` and the `Source:` quote's body (its links stay) and puts each alternative on one line; `minimal` also drops `Context:` and `Source:` and prints `Alternatives: (say "more Q")` for listed alternatives (inline ones stay as written). The header (the lines before the first question), every other line, the blank lines and the `---` rules stay at every level, and each run of blank lines becomes one. A line that starts with a protected label word in any form (`Flips if`, `Commits you to`, `Withheld`, `Discovery`, `Second view`, `Departs from`) is never dropped, whatever field it follows. A label may follow a marker (characters outside printable ASCII, such as an emoji) and sit in bold; a list item never opens a question; fences follow `check-round-codes.sh` (a ``` or ~~~ run, closed by a run of the same character at least as long). Needs no data dir | | `scope-check --ledger L --round N --limit M [--rounds-dir D]` | Print the counts half of round N's `Scope check:` line: ` of M questions registered`, then for each of the two rounds before N `round added , settled `, where `a` is the rows whose round cell is n and `s` those of them `answered`, `deferred`, `withdrawn` or `blocked`. With D, the two rounds are the highest below N that have a `round-.md` in D. Needs no data dir | -| `measures --ledger L --rounds-dir D --level V` | Print `rounds= median_lines= reopened= after_lookup=`: the `round-.md` files in D, the median line count of `show --level V` over them, the entries under L's `## Reopened answers`, and those whose fact names a lookup (a `/:` or the word `lookup`). Needs no data dir | +| `measures --ledger L --rounds-dir D --level V` | Print `rounds= median_lines= reopened= after_lookup=`: the `round-.md` files in D, the median line count of `show --level V` over them, the entries under L's `## Reopened answers`, and those whose fact names a lookup (a `/:` not inside a path, or `lookup`, `lookups` or `looked up`). Needs no data dir | | `doctor --ledger L` | Report, writing nothing, what the running version needs and L or the page lacks: a `missing:` line for each of L's `## Constraint ledger` and `## Open-question register` sections and for open questions without a Basis or a `Checked against:` line in `facts`, and a `note:` when L's `Planning version:` line or `meta.pluginVersion` (the version that first wrote the data dir) names another version; exit 1 on any `missing:` line. `--dir` need not exist yet | | `lease [--release \| --takeover]` | Print the watcher holding the lease, or `no lease`; `--release` clears it; `--takeover` clears it and runs `watch.sh` for this session (run it as the background task after `/clear`), and the old watcher exits 3 with a `handed over, not a failure` message | diff --git a/plugins/planning/surface/exporters.py b/plugins/planning/surface/exporters.py index 1999445ff63..090dbc44f58 100644 --- a/plugins/planning/surface/exporters.py +++ b/plugins/planning/surface/exporters.py @@ -157,6 +157,7 @@ # Register statuses that leave a question unresolved; superseded-by-plan is a plan's # displacement of a user answer, waiting on the user's explicit reply. UNSETTLED = ("open", "superseded-by-plan") +SETTLED = ("answered", "deferred", "withdrawn", "blocked") # The Brief's field labels, per context/problem-framing.md "Problem fields" and "Work size". PROBLEM_FIELDS = ("Gap", "Who / how many", "Frequency", "Trend", "Cost", "Evidence") WORK_SIZE_FIELDS = ("Appetite", "Size", "Uncertainty", "Split signals") @@ -1506,7 +1507,7 @@ def export_brief(d, text=None): sections = dict(sections, decisions=ticked_recap(sections["decisions"], rows)) count = { s: sum(1 for r in rows if r["status"] == s) - for s in ("answered", "deferred", "blocked", "withdrawn", *UNSETTLED) + for s in (*SETTLED, *UNSETTLED) } answered = [r for r in rows if r["status"] == "answered"] replaced = replacements(rows) @@ -2187,6 +2188,31 @@ def section_span(heads, start, total): return start, next((h for h, lv, _ in heads if h > start and lv <= level), total) +REOPENED_ENTRY = re.compile( + r"^- (Q[1-9][0-9]*) \| round ([1-9][0-9]*) \| was: (.*) \| fact: (.*)$" +) + + +def reopened_entry(qid, rnd, was, fact): + """One `## Reopened answers` entry, the line REOPENED_ENTRY reads back.""" + return f"- {qid} | round {rnd} | was: {was} | fact: {fact}" + + +def reopened_entries(text): + """[(Q, round, was, fact)] of each entry under the ledger's `## Reopened answers`.""" + bare = text.splitlines() + heads = headings(bare) + out = [] + for h, _, title in heads: + if title == "reopened answers": + start, end = section_span(heads, h, len(bare)) + for line in bare[start + 1 : end]: + m = REOPENED_ENTRY.match(line) + if m: + out.append((m.group(1), int(m.group(2)), m.group(3), m.group(4))) + return out + + def add_reopened(lines, entry): """Append `entry` to the ledger's `## Reopened answers` section. A ledger with none gets it where the checklist template puts it: after the register's whole level-2 section, its @@ -2267,7 +2293,7 @@ def add(tail): if row in marked: rewrite_row(lines, j, m.start(2), lambda s, r: (s, r), recheck(row)) order.append(row) - add_reopened(lines, f"- {n} | round {rnd} | was: {was[0]} | fact: {fact}") + add_reopened(lines, reopened_entry(n, rnd, was[0], fact)) return "".join(lines), order @@ -2579,7 +2605,7 @@ def import_ledger(doc, text, ledger, at): added = ADDED.search(cell) if added and int(added.group(1)) > rnd: added_in[qid] = int(added.group(1)) - if status not in (*UNSETTLED, "answered", "deferred", "withdrawn", "blocked"): + if status not in (*UNSETTLED, *SETTLED): refuse(f"unknown status {status!r} for Q{n}", ledger) named = parse_named(res, ledger) if named: diff --git a/plugins/planning/surface/round.py b/plugins/planning/surface/round.py index cee73f97a4e..1af75fd68b6 100644 --- a/plugins/planning/surface/round.py +++ b/plugins/planning/surface/round.py @@ -2961,8 +2961,15 @@ def cmd_next_id(d, a): ROUND_LABEL = re.compile( rf"^[^ -~]*[ \t]*(?:\*\*|__)?({'|'.join(ROUND_LABELS)})(?:\*\*|__)?:" ) +LIST_ITEM = re.compile(r"^(?:[ \t]*>)*[ \t]*(?:[-*+]|[0-9]+[.)])[ \t]") QUESTION_LINE = re.compile(r"^[^A-Za-z0-9]*(Q[0-9]+)(?:[:)]|\.(?![A-Za-z0-9]))") -ROUND_FENCE = re.compile(r"^\s*(```|~~~)") +PROTECTED = re.compile( + r"^[^A-Za-z0-9]*(?:flips if|commits you to|withheld|discovery|second view|departs from)\b", + re.IGNORECASE, +) +ROUND_FENCE = re.compile( + r"^(?:[ \t]*>)*[ \t]*(?:(?:[-*+]|[0-9]+[.)])[ \t]+)?(`{3,}|~{3,})(.*)$" +) DROPPED = { "full": (), "standard": ("Origin", "Basis", "Checked against"), @@ -2970,28 +2977,50 @@ def cmd_next_id(d, a): } MD_LINK = re.compile(r"\[[^\]]*\]\([^)\s]+\)|]+>?") ROUND_FILE = re.compile(r"^round-([1-9][0-9]*)\.md$") -SETTLED = ("answered", "deferred", "withdrawn", "blocked") -REOPENED_ENTRY = re.compile( - r"^- Q[1-9][0-9]* \| round [1-9][0-9]* \| was: .* \| fact: (.*)$" +LOOKUP = re.compile( + r"(?` quote and list marker) and closes on a run of its character at least as long with + nothing after it; fenced is True for every line inside or on a fence.""" + m = ROUND_FENCE.match(bare) + if fence is None: + return ((m.group(1)[0], len(m.group(1))), True) if m else (None, False) + run, rest = (m.group(1), m.group(2)) if m else ("", "x") + if run[:1] == fence[0] and len(run) >= fence[1] and not rest.strip(): + return None, True + return fence, True + + +def question_id(bare): + """The Q a question line opens, or None; a list item never opens a question.""" + if LIST_ITEM.match(bare): + return None + q = QUESTION_LINE.match(bare) + return q.group(1) if q else None def round_segments(lines): """[(label, qid, lines)] of a round file's lines: the header (label "header"), then per question its title, each labelled field with the lines that continue it up to the next blank - line or label, and each blank or other line on its own. Fenced lines stay in their segment.""" - segs, qid, fenced = [["header", None, []]], None, False + line or label, and each blank or other line on its own. A line that starts with a protected + label word in any form starts a segment of its own, kept at every level. Fenced lines stay in + their segment.""" + segs, qid, fence = [["header", None, []]], None, None for line in lines: bare = line.rstrip("\r\n") - if fenced or ROUND_FENCE.match(bare): - fenced = fenced != bool(ROUND_FENCE.match(bare)) + fence, fenced = fence_step(fence, bare) + if fenced: segs[-1][2].append(line) continue - q = QUESTION_LINE.match(bare) + q = question_id(bare) label = ROUND_LABEL.match(bare) if q: - qid = q.group(1) + qid = q segs.append(["title", qid, [line]]) elif segs[-1][0] == "header": segs[-1][2].append(line) @@ -2999,7 +3028,7 @@ def round_segments(lines): segs.append(["blank", qid, [line]]) elif label: segs.append([label.group(1), qid, [line]]) - elif segs[-1][0] == "blank": + elif segs[-1][0] == "blank" or PROTECTED.match(bare): segs.append(["text", qid, [line]]) else: segs[-1][2].append(line) @@ -3013,9 +3042,10 @@ def eol_of(line): def render_round(text, level): """The round file `text` at `level`: full is the text itself; standard drops Origin:, Basis:, Checked against: and the Source: quote's body (keeping its links) and puts each - alternative on one line; minimal also drops Context: and Source: and replaces the - alternatives with their count and `say "more Q"`. The header, every other line and the - blank lines are kept, a run of blanks a dropped field leaves collapsed to one.""" + alternative on one line; minimal also drops Context: and Source: and replaces listed + alternatives with their count and `say "more Q"` (inline ones stay as written). The + header, every other line and the blank lines are kept, each run of blank lines collapsed + to one.""" if level == "full": return text out = [] @@ -3033,7 +3063,9 @@ def render_round(text, level): items[-1] += " " + line.strip() else: items.append(line.rstrip("\r\n")) - if level == "minimal": + if not items: + pass + elif level == "minimal": seg = [f'Alternatives: {len(items)} (say "more {qid}"){eol}'] else: seg = [seg[0], *(item + eol for item in items)] @@ -3089,24 +3121,11 @@ def cmd_scope_check(d, a): parts = [f"{len(rows)} of {a.limit} questions registered"] for n in list(earlier)[:2]: mine = [status for _, status, rnd, _, _, _ in rows if rnd == n] - settled = sum(status in SETTLED for status in mine) + settled = sum(status in exporters.SETTLED for status in mine) parts.append(f"round {n} added {len(mine)}, settled {settled}") print("; ".join(parts)) -def reopened_facts(text): - """The fact of each `- Q | round | was: ... | fact: ...` entry under the ledger's - `## Reopened answers` heading.""" - bare = text.splitlines() - heads = exporters.headings(bare) - facts = [] - for h, _, title in heads: - if title == "reopened answers": - start, end = exporters.section_span(heads, h, len(bare)) - facts += [m.group(1) for x in bare[start + 1 : end] if (m := REOPENED_ENTRY.match(x))] - return facts - - def cmd_measures(d, a): text = read_input(a.ledger, "ledger") exporters.check_ledger(text, a.ledger) @@ -3122,7 +3141,7 @@ def cmd_measures(d, a): else: total = counts[mid - 1] + counts[mid] median = str(total // 2) if total % 2 == 0 else f"{total / 2:.1f}" - facts = reopened_facts(text) + facts = [fact for _, _, _, fact in exporters.reopened_entries(text)] after = sum(bool(LOOKUP.search(f)) for f in facts) print( f"rounds={len(counts)} median_lines={median} reopened={len(facts)} " @@ -3677,7 +3696,7 @@ def main(argv=None): "measures", help="print rounds (round-.md files in --rounds-dir), the median lines of those " "rounds shown at --level, the entries under the ledger's '## Reopened answers', and " - "those whose fact names a lookup (a /: or the word lookup); no data dir " + "those whose fact names a lookup (a /:, lookup, lookups or looked up); no data dir " "needed", ) s.add_argument("--ledger", required=True, help="ledger markdown file") diff --git a/plugins/planning/surface/test_exporters.py b/plugins/planning/surface/test_exporters.py index fcc3ceea151..46563c2f365 100644 --- a/plugins/planning/surface/test_exporters.py +++ b/plugins/planning/surface/test_exporters.py @@ -5316,6 +5316,37 @@ def test_the_tag_grammar_matches_the_gate(self): named = set(re.findall(r"(Q\d+): answered row has an empty resolution", out)) self.assertEqual(named, empty) + def test_the_gate_settles_exactly_the_settled_statuses(self): + settled = set() + for status in (*exporters.SETTLED, *exporters.UNSETTLED, "discussing"): + ledger = self.tmp / f"status-{status}.md" + ledger.write_bytes( + ( + "# Interview ledger\n\n## Open-question register\n\n" + f"- Q1 | {status} | round 1 | One? | an answer\n" + ).encode() + ) + rc, out = self.check("--ledger", ledger) + self.assertIn(rc, (0, 1), out) + if rc == 0: + settled.add(status) + self.assertEqual(settled, set(exporters.SETTLED)) + + def test_the_gate_skips_the_entry_reopen_writes(self): + text = ( + "# Interview ledger\n\n## Open-question register\n\n" + "- Q1 | answered | round 1 | One? | three tries\n" + ) + new, _ = exporters.reopen_row(text, "Q1", 3, "the lookup | found otherwise", "the test") + self.assertEqual( + exporters.reopened_entries(new), + [("Q1", 3, "three tries", "the lookup / found otherwise")], + ) + ledger = self.tmp / "reopened.md" + ledger.write_bytes(new.encode()) + rc, out = self.check("--ledger", ledger) + self.assertEqual(rc, 1, out) + def test_a_repeated_tag_is_refused_naming_its_row(self): with self.assertRaises(SystemExit) as raised: exporters.parse_tags("depends:: Q1; depends:: Q2", "Q7", "the ledger") diff --git a/plugins/planning/surface/test_round.py b/plugins/planning/surface/test_round.py index 286e9059ac3..f441bdbd323 100644 --- a/plugins/planning/surface/test_round.py +++ b/plugins/planning/surface/test_round.py @@ -5971,6 +5971,52 @@ def test_a_context_bullet_that_starts_with_a_label_word_is_not_a_label(self): at = out.index("- No reader is named yet.") self.assertEqual(out[at + 1 : at + 3], bullets) + def show_variant(self, level, *edits): + """show at `level` of the fixture with each (anchor line, added lines) inserted.""" + text = ROUND_FULL.read_text(encoding="utf-8") + for anchor, added in edits: + self.assertIn(anchor + "\n", text) + text = text.replace(anchor + "\n", anchor + "\n" + "".join(x + "\n" for x in added), 1) + with tempfile.TemporaryDirectory(prefix="iv-show-") as tmp: + path = Path(tmp) / "round-3.md" + path.write_bytes(text.encode("utf-8")) + return self.show(level, path).decode("utf-8").splitlines() + + def test_a_protected_line_under_a_dropped_field_is_kept(self): + protected = [ + "*Flips if:* the guide changes its batching rule.", + "Second view (recommender): log both failures.", + "- Commits you to: no second store", + "Departs from: Q4 (one write per job), once more.", + "Withheld: until the queue lookup returns.", + "**Discovery** first: who reads the entries.", + ] + edits = ( + ("Basis: verified, `src/retry.ts:12` and the audit guide's Writes section", protected[:2]), + ("- The store depends on the queue lookup.", protected[2:]), + ) + for level in ("full", "standard", "minimal"): + with self.subTest(level=level): + out = self.show_variant(level, *edits) + for line in protected: + self.assertIn(line, out) + + def test_a_source_with_no_link_prints_nothing_at_standard(self): + out = self.show_variant( + "standard", ("- No reader is named yet.", ["", "Source: the team wiki, quoted", "> No reader yet."]) + ) + self.assertNotIn("Source: the team wiki, quoted", out) + self.assertNotIn("> No reader yet.", out) + self.assertIn("- No reader is named yet.", out) + + def test_inline_alternatives_are_kept_as_written(self): + inline = "Alternatives: (a) a dashboard; (b) a weekly email." + anchor = "Discovery: who reads the entries settles whether they need a dashboard." + for level in ("standard", "minimal"): + with self.subTest(level=level): + out = self.show_variant(level, (anchor, ["", inline])) + self.assertIn(inline, out) + def test_an_unknown_level_is_refused(self): rc, out, err = run_bytes("show", "--level", "brief", str(ROUND_FULL)) self.assertNotEqual(rc, 0) @@ -6046,19 +6092,62 @@ class TestMeasures(unittest.TestCase): """measures prints rounds, median lines per shown round, reopened rows and those reopened after a lookup.""" + FACTS = { + "Q1": "/discovery:research found the queue already persists jobs", + "Q2": "two lookups found the host has no Redis", + "Q3": "we looked up the vendor page: batches flush every five seconds", + "Q4": "the user said the budget is four tries, per docs/build:3", + "Q5": "the user changed their mind", + } + + @classmethod + def setUpClass(cls): + """One ledger whose `## Reopened answers` entries `reopen` itself wrote.""" + cls.base = Path(tempfile.mkdtemp(prefix="iv-measures-base-")) + cls.addClassCleanup(shutil.rmtree, cls.base, ignore_errors=True) + ledger = cls.base / "interview-checklist.md" + ledger.write_bytes( + ( + "# Interview ledger\n\n## Open-question register\n\n" + + "".join(f"- Q{n} | answered | round 1 | Q {n}? | yes\n" for n in range(1, 6)) + ).encode("utf-8") + ) + for qid, fact in cls.FACTS.items(): + fact_file = cls.base / f"fact-{qid}.txt" + fact_file.write_bytes(fact.encode("utf-8")) + rc, out, err = run_bytes( + "reopen", "--ledger", str(ledger), qid, "--round", "3", + "--fact-file", str(fact_file), + ) + assert rc == 0, err + cls.ledger_bytes = ledger.read_bytes() + def setUp(self): self.tmp = Path(tempfile.mkdtemp(prefix="iv-measures-")) self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) self.rounds = self.tmp / "slice" self.rounds.mkdir() self.ledger = self.rounds / "interview-checklist.md" - self.ledger.write_bytes( - ( - SCOPE_LEDGER + "\n## Reopened answers\n\n" - "- Q2 | round 3 | was: yes | fact: the /discovery:research lookup found the " - "queue already persists jobs\n" - "- Q4 | round 3 | was: yes | fact: the user said the budget is four tries\n" - ).encode("utf-8") + self.ledger.write_bytes(self.ledger_bytes) + + def round_of(self, n, count): + text = "".join(f"line {k}\n" for k in range(count)) + (self.rounds / f"round-{n}.md").write_bytes(text.encode("utf-8")) + + def test_the_reopen_writer_and_the_measures_reader_agree(self): + self.assertEqual( + self.measures("full"), "rounds=0 median_lines=0 reopened=5 after_lookup=3\n" + ) + + def test_an_even_count_median_is_the_mean_of_the_middle_two(self): + self.round_of(1, 4) + self.round_of(2, 6) + self.assertEqual( + self.measures("full"), "rounds=2 median_lines=5 reopened=5 after_lookup=3\n" + ) + self.round_of(2, 7) + self.assertEqual( + self.measures("full"), "rounds=2 median_lines=5.5 reopened=5 after_lookup=3\n" ) def measures(self, level): @@ -6076,22 +6165,21 @@ def measures(self, level): def test_rounds_median_lines_and_reopened_rows(self): for n, count in ((1, 4), (2, 10), (3, 7)): - text = "".join(f"line {k}\n" for k in range(count)) - (self.rounds / f"round-{n}.md").write_bytes(text.encode("utf-8")) + self.round_of(n, count) (self.rounds / "round-notes.md").write_bytes(b"not a round\n") self.assertEqual( self.measures("full"), - "rounds=3 median_lines=7 reopened=2 after_lookup=1\n", + "rounds=3 median_lines=7 reopened=5 after_lookup=3\n", ) def test_median_lines_count_the_round_at_the_shown_level(self): (self.rounds / "round-3.md").write_bytes(ROUND_FULL.read_bytes()) self.assertEqual( - self.measures("full"), "rounds=1 median_lines=51 reopened=2 after_lookup=1\n" + self.measures("full"), "rounds=1 median_lines=58 reopened=5 after_lookup=3\n" ) self.assertEqual( self.measures("minimal"), - f"rounds=1 median_lines={len(MINIMAL)} reopened=2 after_lookup=1\n", + f"rounds=1 median_lines={len(MINIMAL)} reopened=5 after_lookup=3\n", ) diff --git a/plugins/planning/surface/tests/fixtures/round-file/round-full.md b/plugins/planning/surface/tests/fixtures/round-file/round-full.md index b04998ca01b..e175344bc80 100644 --- a/plugins/planning/surface/tests/fixtures/round-file/round-full.md +++ b/plugins/planning/surface/tests/fixtures/round-file/round-full.md @@ -7,6 +7,7 @@ Held: Q12 (queue choice) until the queue lookup returns Context: - Each retry runs in its own worker (`src/retry.ts:12`). +- Q4: you chose one write per job in round 1. - The audit log takes one write per job today ([audit guide](https://example.com/audit)). Source: [Audit log guide, Writes](https://example.com/audit#writes) @@ -35,6 +36,12 @@ Alternatives: Context: - No reader is named yet. +- The draft entry shape: +````text +Q9: a fenced line, not a question +``` +Basis: a fenced line, not a label +```` Discovery: who reads the entries settles whether they need a dashboard. From 4d13f6a8e851d4eba06de62a500cb9fccbbd99b3 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 21 Nov 2026 08:13:35 -0500 Subject: [PATCH 14/30] feat(planning): interview layout at detail levels, opening lines and one home per rule Phase 8 of the interview redesign (Brief AC5, AC6, AC13, AC14), plus the four IMPORTANT findings and the cheap SUGGESTIONs of the Phase 7 code-design review (.work/reviews/feat-interview-redesign/ 20261121T1300Z-code-reviewer-phase7.md). - SKILL.md: the per-question template is the saved full block, with a blank line after the title and a `---` line between questions. "Keep the layout whole" owns Context first and the `---` rule (product.md is a declared exception); new "Save the full block; print it at the detail level": the round sent is `round.sh show --level ` output, "more Q" and "full round", the protected lines, and the no-Bash standard print. Origin:/Basis: bind the full block; the printed reason ends with its evidence label and source at every level. New "Open the round header with its scope check". The bare-code paragraph describes the structure checks and how to fix each missing= line (scope-check, single-scope-check, recommendation, single-recommendation). "One question at a time" is renamed "Question pacing". Round order, ids, the shared lane's cap, commitment parts per lane, scenario ids, Context terms, held questions and lookups point at their homes. New "Three glossaries, and who writes each"; the closing summary group is renamed "Project glossary changes". Step 4 gains "When the contract is persisted". The Question lanes resolver call passes --pacing and --ledger and prints `line` as the opening lines. - Review fixes: a term's restatement is registered like any question (finding 2); the glossary plugin offer is in offers/offerable.json (finding 3); a routing suggestion in an intake round is one plain line, never a `My recommendation:` block, in lookups.md and routing.md (finding 4); interview-recommender.md says it inherits the session's model (finding 1); a withheld question shows no Second view: line, a basis fact missing from the Context is added to it, the quality bar's three bars sit in its Context, discovery questions may arise in any round, the sweep's "no severity, confidence or blocking cut" matches assumption-sweep.md, the hold and reopen rules are stated once in lookups.md (SUGGESTIONs 5-10). Skipped: 11 (unattended rung for an open discovery row: loop.md Unattended path is a pinned ladder and the change is a design call), 12 (interview-person-deferred-document grader: an eval grader edit outside Phase 8's files). - lookups.md: one home for A1-A4, A6, A7: a released held question is asked in the next message that presents questions; a recommendation is never shown before its facts (held, else withheld); new "Passes around the interview" (before round 1, during the rounds, after confirmation; no context clear between passes). - loop.md: Frontier rounds item 2 is the one statement of round order (item 4 and SKILL.md point at it); the header names Scope check:; item 5 points at SKILL.md Step 4 for when the Brief is written; the counter points at question-shape.md; the shared cap is the one exception to "never capped"; the degrade table's column is headed `Q`; the glossary is read at the start of every interview; terminology restatements are registered. - question-shape.md: the `id` row (one counter per topic, assigned by next-id when first shown); the alternatives count in every lane; the Context terms rule is the one home. - lane-shared.md: "Round cap" (8), the id rule via next-id, gaps declared by a gap line, terminal-only layout rules. lane-page.md, lane-document.md and surface.md: id and gap pointers, terminal-only layout rules. - product.md, scenario-comparison.md: the declared exception; the scenario id home. templates/document.md: a blank line between label blocks. templates/checklist.md: the sweep's seven categories. - composition.md, handoff.md: pointers to lookups.md and the renamed group. - reference/config.md, schemas/planning.schema.json: question_detail names every line no level hides. - surfaces/registry.mjs: `line` is the round-8 opening, one sentence group per line: `Asking here, one round|question at a time. A local page or a shared session is also available; say which to switch.`, `Questions are saved to .` (with --ledger), then one `Optional:` line per installed planner and per offer. Entries gain optional `where` (terminal: here) and `offerName` (page: local page). Readers of `line`: SKILL.md "Question lanes" only (the lane spokes print notices, not it). - offers/offer-source.mjs: the offer text is ` . Say "install " to add it`; offerable.json adds domain-driven-design. - Legacy skills/interview/evals/evals.json: "next contiguous Q" becomes "next id from round.sh next-id" (no digested case touched). - session-flow steps.md and pre-implementation-order.md: pointers only. Tests - surface-registry.test.mjs: the opening lines asserted exactly against the Q38 wording and the eval's first-line regex; pacing single; no ledger, no saved-to line; planner and offer lines; the offer name is an argument word. The null-line, unavailable-surface, document and kiosk assertions are kept. - offer-source.test.mjs: the owner-approved offer text; the glossary entry is listed by this marketplace at ./plugins/domain-driven-design. Re-pins in tests/interview-defenses.test.sh (sha literals only; no pin added, removed or loosened; every phrase and pin_exact pin passes): - SKILL.md Stance: 7d7f2372cef7aaa76d271c3e57e6a9a6208727c94b4dce3c6d33357f2d2926e4 -> 2f45a095d33c22488286ffd2c5b0248b1f3f9123084245b4c2fbc9916fd94406: layout, detail levels, scope-check header, pointers to the rules' homes and term restatements registered; partial rounds still never resolve an unanswered question, and only where it re-surfaces moved to loop.md item 2. - SKILL.md Step 4: 3fee6385252f2f476aef4e00797c9f416e7a7c7f89e3ea6ebdabad267177a640 -> 6e6577f57f115fa749e6fb5ec9bef7bf860cfdc07d36aa7aa064563f7f02b981: adds "When the contract is persisted"; the cross-check, the UNCONFIRMED stamp and the scope path are unchanged and nothing acts on an unconfirmed Brief. - SKILL.md Action Router: 9ec05ae465fc8e693fde59be1e81d386cd350f63021787ecc29179fc23eff758 -> 368743e50f46c68695014cc10793b89a1cf2c0b2e84258798579c4fbe0e5c36d: the resolver passes --pacing and --ledger, the opening prints as lines, and the cap names the shared lane's exception; the lock row and STOP-on-gap are byte-identical. - SKILL.md Step 3: 101586e9916d476589a8519aa93cf89121ad517d46278a480aab737027563419 -> d1aedd09a3cc54f12763212381e7ef7b0731165303cf4b18ba6e1da2c9526533: the Glossary group is renamed "Project glossary changes"; the gates and the confirmation exemption are unchanged. - SKILL.md "does NOT do": 261ea805a3e429229840ef9c86bacdae791acedfa9aedbfb2d95c5e949273d84 -> 6fbbb933036f9c1f2b060ec0e16099fa04bfb8082a8f429e396a462cd16856de: "in-flight lookup" becomes "a lookup" with its link; the fudge prohibition is byte-identical. - SKILL.md Step 2: d9c84aac832447d47e2f50d5008a449d378bfa3282291d0334c9bd65129a8744 -> 4f1230d595f67a615b884320885dcc743dd42080822d3197a2f4a823ab7cb95d: a failing consequential recommendation is held for a lookup instead of shown "not yet researched" (stricter), the recommender's withheld and missing-fact clauses, the sweep's full no-cut wording; constraint ledger, candidates, tripwire and sweep rules hold. - loop.md Step 2: cd3481b6f2d547f82f33ee14cc69e22b38a5c936f857461f9e8538e18cdfd3d1 -> 221af05eb4cb7c1f58a756f9218fc7f23ad420e5266d569ca2db6ca379ffd416: item 2 is the one round order, the header names Scope check:, item 5 points at Step 4, restatements are registered; the constraint rows and the process-change definition are byte-identical. - loop.md relentless mechanics: 53ad96eee9dc6133be50f497d4228b725006506df953ad9d24bedbc087ea9c9d -> 713bf5f475597226013d471a5cc1a41a4579e256b844c69d8d5c9b28aea15a01: counter and cap pointers, the `Q` column, the restored registration of a term restatement; the late commitment row still takes the next free id. Not changed: the Gates section, since no text may be added above the Action Router; its "Full rule: Step 3/Step 4" pointers already lead to the G1 home. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015XWwmeDkiEnVmSx4dtkkfh --- .../feat-interview-redesign-07635b38.md | 1 + .../feat-interview-redesign-3c9e1a52.md | 7 ++ .../planning/agents/interview-recommender.md | 8 +- plugins/planning/offers/offer-source.mjs | 2 +- plugins/planning/offers/offerable.json | 7 ++ plugins/planning/reference/config.md | 2 +- plugins/planning/schemas/planning.schema.json | 2 +- plugins/planning/skills/interview/SKILL.md | 73 +++++++++++-------- .../skills/interview/context/composition.md | 4 +- .../skills/interview/context/handoff.md | 2 +- .../skills/interview/context/lane-document.md | 5 +- .../skills/interview/context/lane-page.md | 4 +- .../skills/interview/context/lane-shared.md | 14 +++- .../skills/interview/context/lookups.md | 39 ++++++++-- .../planning/skills/interview/context/loop.md | 24 +++--- .../skills/interview/context/product.md | 4 + .../interview/context/question-shape.md | 18 ++++- .../skills/interview/context/routing.md | 6 +- .../interview/context/scenario-comparison.md | 2 +- .../skills/interview/context/surface.md | 2 +- .../skills/interview/evals/evals.json | 20 ++--- .../skills/interview/templates/checklist.md | 2 +- .../skills/interview/templates/document.md | 5 +- plugins/planning/surfaces/entries/page.mjs | 1 + .../planning/surfaces/entries/terminal.mjs | 1 + plugins/planning/surfaces/registry.mjs | 28 ++++--- .../planning/tests/interview-defenses.test.sh | 16 ++-- plugins/planning/tests/offer-source.test.mjs | 11 +++ .../planning/tests/surface-registry.test.mjs | 64 ++++++++++------ .../context/pre-implementation-order.md | 9 +-- .../skills/workflow/context/steps.md | 8 +- 31 files changed, 253 insertions(+), 138 deletions(-) create mode 100644 .changes/session-flow/feat-interview-redesign-3c9e1a52.md diff --git a/.changes/planning/feat-interview-redesign-07635b38.md b/.changes/planning/feat-interview-redesign-07635b38.md index b44da531a22..1f3e544d952 100644 --- a/.changes/planning/feat-interview-redesign-07635b38.md +++ b/.changes/planning/feat-interview-redesign-07635b38.md @@ -8,6 +8,7 @@ bump: minor - **Register tags and `round.sh reopen`.** A register row's resolution can end in `kind::`, `problem::`, `depends::`, `reopened::` and `recheck::` tags, which import and export carry unchanged. `round.sh reopen --ledger Q --round --fact-file ` reopens an answered row under its own number, keeps the old answer and the fact in `## Reopened answers`, and flags every row built on it with `recheck::`, which blocks `check-open-questions.sh` until `reopen --rechecked` clears it. - **Round detail levels, `scope-check` and `measures`.** `round.sh show --level full|standard|minimal ` prints a saved round at that detail level (full byte for byte; no level hides the recommendation with its reason, `Withheld:`, `Discovery:`, `Second view:`, `Departs from:`, `Flips if:` or `Commits you to:`), set by the new `question_detail` setting (default `standard`). `round.sh scope-check` prints the counts for a round header's `Scope check:` line, and `round.sh measures` prints rounds, median lines per shown round and reopened answers. `check-round-codes.sh` also fails a question with no single `My recommendation:`, `Withheld:`, `Discovery:` or `Strong answer:` line, and a round header without exactly one `Scope check:` line. - **Interview flow: gap triage, held questions, a blind recommender and stable ids.** Before round 1, `/planning:interview` sorts every unknown into a lookup, a discovery question or a decision question; when discovery questions exist, round 1 asks only those, open, each with a `Discovery:` line and no recommendation (new `context/lookups.md`). A question a running lookup could change is held on the `Held:` line, at most one dispatch runs per lookup skill, and a fact that contradicts an answer runs `round.sh reopen`. Each round's recommendations come from the new read-only `interview-recommender` agent, which sees the facts, constraints and settled decisions but not the conversation; a differing draft shows as `Second view:` and a departure from a settled decision as `Departs from:`. New ids come from `round.sh next-id`, problems are numbered P1, P2 and tagged on their questions, the round header's `Scope check:` line comes from `round.sh scope-check`, the end sweep reports every finding for the disposition pass to filter, the glossary is read at the start of every interview and a term is written only after the user confirms its restatement, and `/planning:audit-answers` never auto-accepts a `kind:: discovery` row. +- **Interview layout, opening lines and one home per rule.** The round file holds the full question block and the round you see is `round.sh show` output at `question_detail`, with a blank line after the title, the recommendation block and the alternatives, a `---` line between questions and the header's `Scope check:` line; at every level the recommendation's reason ends with its evidence label and source, and "more Q" or "full round" shows the rest. The opening lines now read `Asking here, one round at a time. A local page or a shared session is also available; say which to switch.`, `Questions are saved to .` and one `Optional:` line per planner or plugin offer (`surfaces/registry.mjs resolve` gains `--pacing` and `--ledger`), and the glossary plugin is offered when it is not installed. Each rule stated two ways now has one home that the other files link to: round order, ids and gaps, the shared lane's 8-question round cap, when lookups and the passes around the interview run, the three glossaries, scenario ids and when the Brief is persisted. A term's restatement is registered like any question, and a routing suggestion in an intake round is one line, never a `My recommendation:` block. ### Removed diff --git a/.changes/session-flow/feat-interview-redesign-3c9e1a52.md b/.changes/session-flow/feat-interview-redesign-3c9e1a52.md new file mode 100644 index 00000000000..64e0fb139b8 --- /dev/null +++ b/.changes/session-flow/feat-interview-redesign-3c9e1a52.md @@ -0,0 +1,7 @@ +--- +bump: patch +--- + +### Changed + +- **The workflow navigator points at `/planning:interview` for its own rules.** The interview stage in `context/steps.md` and the ladder in `context/pre-implementation-order.md` no longer restate how the interview paces its questions or when explore, research, a PRD or a prototype runs around it; both name `/planning:interview` as the owner of those rules. diff --git a/plugins/planning/agents/interview-recommender.md b/plugins/planning/agents/interview-recommender.md index 84e131676c7..b8177cb733e 100644 --- a/plugins/planning/agents/interview-recommender.md +++ b/plugins/planning/agents/interview-recommender.md @@ -30,8 +30,8 @@ Q | recommend: | basis: | withheld: ``` -**Model.** We set no `model` and no `effort`: the recommender is meant to judge at the session's -own level, and this file names no model. +**Model.** We set `model: inherit` and no `effort`: the recommender inherits the session's model, +so it judges at the session's own level, and this file names no model. -- **Pointer:** for what a subagent with no `model` or `effort` field runs on, see - [create custom subagents: supported frontmatter fields](https://code.claude.com/docs/en/sub-agents#supported-frontmatter-fields). +- **Pointer:** for what `inherit` and an absent `effort` field mean, see + [create custom subagents: choose a model](https://code.claude.com/docs/en/sub-agents#choose-a-model). diff --git a/plugins/planning/offers/offer-source.mjs b/plugins/planning/offers/offer-source.mjs index f12923c8b1f..7e899ef33dd 100644 --- a/plugins/planning/offers/offer-source.mjs +++ b/plugins/planning/offers/offer-source.mjs @@ -106,7 +106,7 @@ export function freshness(lastCommit, now) { } const offerText = (e, date) => - `${e.name}, an optional ${e.kind} that ${e.helps}${date ? `, last updated ${date}` : ''}, is not installed; say "install ${e.name}" to add it` + `${e.name} ${e.helps}${date ? ` (last updated ${date})` : ''}. Say "install ${e.name}" to add it` export async function findOffers({ entries = loadOfferable(), diff --git a/plugins/planning/offers/offerable.json b/plugins/planning/offers/offerable.json index 6162731dcbd..5df03ee454c 100644 --- a/plugins/planning/offers/offerable.json +++ b/plugins/planning/offers/offerable.json @@ -7,6 +7,13 @@ "kind": "planner", "invoke": "/html-plan", "helps": "writes the plan as one interactive HTML page" + }, + { + "name": "domain-driven-design", + "marketplace": "melodic-software", + "repo": "melodic-software/claude-code-plugins", + "kind": "glossary", + "helps": "keeps the project glossary these questions use" } ] } diff --git a/plugins/planning/reference/config.md b/plugins/planning/reference/config.md index 9d77c07200f..a0d6b15c3ac 100644 --- a/plugins/planning/reference/config.md +++ b/plugins/planning/reference/config.md @@ -151,7 +151,7 @@ layer. | Key | Values | Default | Reader | |---|---|---|---| | `question_pacing` | `round`, `single`, `auto` | `round` | `/planning:interview`, first round: `round` asks the frontier as one set, `single` one question per message, `auto` picks `single` for a broad, open-ended topic and `round` otherwise. A request in the session ("one at a time") overrides it for that session | -| `question_detail` | `full`, `standard`, `minimal` | `standard` | `/planning:interview`, every round: how much of each question `round.sh show` prints from the saved round file, which always holds the full block. `standard` drops `Origin:`, `Basis:`, `Checked against:` and the `Source:` quote's body; `minimal` also drops `Context:` and the source and counts the alternatives; no level hides `Flips if:` or `Commits you to:`. A request in the session ("full round") overrides it for that session | +| `question_detail` | `full`, `standard`, `minimal` | `standard` | `/planning:interview`, every round: how much of each question `round.sh show` prints from the saved round file, which always holds the full block. `standard` drops `Origin:`, `Basis:`, `Checked against:` and the `Source:` quote's body; `minimal` also drops `Context:` and the source and counts the alternatives; no level hides the recommendation with its reason, `Withheld:`, `Discovery:`, `Second view:`, `Departs from:`, `Flips if:` or `Commits you to:` (the interview skill's "Save the full block; print it at the detail level"). A request in the session ("full round") overrides it for that session | | `domain_docs` | `auto`, `ask`, `off` | `auto` | `/planning:interview`, engineering sessions, Domain-aware behaviors: `auto` writes a resolved term through the glossary skill and hands each qualifying decision to `/architecture:record-decision` at round step 8, "Decision record"; `ask` asks before each write; `off` writes neither and records them in the register. A request in the session overrides it for that session | | `answer_audit_rounds` | an integer, `0` or more | `2` | `/planning:interview`, after each round's read-back: offer `/planning:audit-answers` once when this many rounds were accepted whole; `0` never offers | | `wayfind_suggest_questions` | an integer, `0` or more | `27` | `/planning:interview`, at each round boundary: suggest `/planning:wayfind chart` once when the ledger's register holds more than this many questions; `0` never suggests | diff --git a/plugins/planning/schemas/planning.schema.json b/plugins/planning/schemas/planning.schema.json index 60e27c05617..7638ae91da4 100644 --- a/plugins/planning/schemas/planning.schema.json +++ b/plugins/planning/schemas/planning.schema.json @@ -47,7 +47,7 @@ "default": "round" }, "question_detail": { - "description": "How much of each question /planning:interview prints from the saved round file, which always holds the full block. full: every line. standard: drops Origin:, Basis:, Checked against: and the Source: quote's body (its link stays), one line per alternative. minimal: the title, the recommendation or Withheld: or Discovery: line, Second view:, Departs from:, Flips if: and Commits you to:, with a count of the alternatives. No level hides Flips if: or Commits you to:. A request in the session overrides it for that session.", + "description": "How much of each question /planning:interview prints from the saved round file, which always holds the full block. full: every line. standard: drops Origin:, Basis:, Checked against: and the Source: quote's body (its link stays), one line per alternative. minimal: the title, the recommendation or Withheld: or Discovery: line, Second view:, Departs from:, Flips if: and Commits you to:, with a count of the alternatives. No level hides the recommendation with its reason, Withheld:, Discovery:, Second view:, Departs from:, Flips if: or Commits you to:. A request in the session overrides it for that session.", "enum": ["full", "standard", "minimal"], "default": "standard" }, diff --git a/plugins/planning/skills/interview/SKILL.md b/plugins/planning/skills/interview/SKILL.md index 9445f4ba5f5..3f45c11337b 100644 --- a/plugins/planning/skills/interview/SKILL.md +++ b/plugins/planning/skills/interview/SKILL.md @@ -83,28 +83,28 @@ Unknown actions route to `auto`; surface the unrecognized request as a one-line **Default action leans to `me` (relentless prose rounds).** When invoked with no args (or by proactive auto-trigger), bias toward `me`-mode. Drive open decisions through frontier-rounds prose Q&A. Fall back to direct synthesis (`lock`-style) ONLY when context heavily informs against asking: intent already crystal-clear with no open decisions, OR the user signaled "just lock it / stop asking". Auto-detect's synthesize-directly path is for the genuinely-clear case, not the default posture. -**Question surface: inline prose by default.** Rounds render as numbered inline prose. Dictation-friendly, no per-question cap, and each question carries its recommendation, reasoning, and probe in one readable block. `AskUserQuestion` is an opt-in surface, enabled via the plugin's `use_ask_user_question` user config (`${user_config.use_ask_user_question}`, default off). When opted in, use it ONLY for a round of mutually independent questions that fits one `AskUserQuestion` call and whose questions are **simple selections or binary confirms** (the per-call question limit: ). A card carries options, not a recommendation's reasoning or a constraint-surfacing probe, so any question needing its basis argued stays prose. Fall back to prose when the frontier exceeds that limit, any question in the round depends on another, or a question needs more than a pick. The card cannot express a dependency or a rationale, and chunking a round across multiple cards fragments it. When in doubt, prose. +**Question surface: inline prose by default.** Rounds render as numbered inline prose. Dictation-friendly, no cap on questions per round except the shared lane's ([`context/lane-shared.md`](context/lane-shared.md) "Round cap"), and each question carries its recommendation, reasoning, and probe in one readable block. `AskUserQuestion` is an opt-in surface, enabled via the plugin's `use_ask_user_question` user config (`${user_config.use_ask_user_question}`, default off). When opted in, use it ONLY for a round of mutually independent questions that fits one `AskUserQuestion` call and whose questions are **simple selections or binary confirms** (the per-call question limit: ). A card carries options, not a recommendation's reasoning or a constraint-surfacing probe, so any question needing its basis argued stays prose. Fall back to prose when the frontier exceeds that limit, any question in the round depends on another, or a question needs more than a pick. The card cannot express a dependency or a rationale, and chunking a round across multiple cards fragments it. When in doubt, prose. -**Question lanes.** The surface registry picks the lane; this skill names none. Before the first round, run `node "${CLAUDE_PLUGIN_ROOT}/surfaces/registry.mjs" resolve --surface '${user_config.surface}' --data-dir "${CLAUDE_PLUGIN_DATA}"`, adding `--argument ''` when the user asks for a surface in this session or states which one is configured (the `share` action passes `share`), and `--unattended` when the caller declared the run unattended (Step 1.5). It prints one JSON line: `surface`, `decidedBy`, `available`, `alternatives`, `read`, `handle`, `line`, `planners`, `offers` and `notices`. When `line` is not null, print it verbatim, once, as the first line of the message that asks round 1, and proceed without waiting for a surface choice; print each notice in one line. The line may also name an installed optional planner and offer an optional plugin that is not installed; neither is ever required, and the interview goes on either way. When the user says yes to an offer, run that entry's `offers[].install` command, which installs at user scope, then tell the user to run `/reload-plugins` to load it. Read each `read` path under `${CLAUDE_PLUGIN_ROOT}` and nothing else of another lane, and follow `handle.rules`; a later request for another surface runs the resolver again with that argument. When the resolver cannot run (no node or no Bash), say so in one line and ask inline prose rounds as below. Every lane carries one question shape, [`context/question-shape.md`](context/question-shape.md); read it before writing a question anywhere but this conversation. `use_emoji_question_markers`: `${user_config.use_emoji_question_markers}`; `question_pacing`: `${user_config.question_pacing}`; `shared_session_open_browser`: `${user_config.shared_session_open_browser}`. A value still shown as its `${user_config.KEY}` token is unset and takes that key's declared default: `use_emoji_question_markers` `true`, `shared_session_open_browser` `true`. `question_pacing` takes the value its cascade resolves ("One question at a time" below), default `round`. Spokes arrive unexpanded, so put these resolved values in place of their `${user_config.KEY}` tokens. +**Question lanes.** The surface registry picks the lane; this skill names none. Before the first round, once `question_pacing` has resolved ("Question pacing" below), run `node "${CLAUDE_PLUGIN_ROOT}/surfaces/registry.mjs" resolve --surface '${user_config.surface}' --pacing '' --ledger '//interview-checklist.md' --data-dir "${CLAUDE_PLUGIN_DATA}"`, adding `--argument ''` when the user asks for a surface in this session or states which one is configured (the `share` action passes `share`), and `--unattended` when the caller declared the run unattended (Step 1.5). It prints one JSON line: `surface`, `decidedBy`, `available`, `alternatives`, `read`, `handle`, `line`, `planners`, `offers` and `notices`. When `line` is not null, print it verbatim, once, as the opening lines of the message that asks round 1, keeping each of its lines on its own line, and proceed without waiting for a surface choice; print each notice in one line. Its later lines may name an installed optional planner and offer an optional plugin that is not installed; neither is ever required, and the interview goes on either way. When the user says yes to an offer, run that entry's `offers[].install` command, which installs at user scope, then tell the user to run `/reload-plugins` to load it. Read each `read` path under `${CLAUDE_PLUGIN_ROOT}` and nothing else of another lane, and follow `handle.rules`; a later request for another surface runs the resolver again with that argument. When the resolver cannot run (no node or no Bash), say so in one line and ask inline prose rounds as below. Every lane carries one question shape, [`context/question-shape.md`](context/question-shape.md); read it before writing a question anywhere but this conversation. `use_emoji_question_markers`: `${user_config.use_emoji_question_markers}`; `question_pacing`: `${user_config.question_pacing}`; `shared_session_open_browser`: `${user_config.shared_session_open_browser}`. A value still shown as its `${user_config.KEY}` token is unset and takes that key's declared default: `use_emoji_question_markers` `true`, `shared_session_open_browser` `true`. `question_pacing` takes the value its cascade resolves ("Question pacing" below), default `round`; `question_detail` (`${user_config.question_detail}`) the value "Keep the layout whole" below resolves, default `standard`. Spokes arrive unexpanded, so put these resolved values in place of their `${user_config.KEY}` tokens. ## Stance: supportive, depth-first, opinionated The Q&A path of this skill is one engine wrapped in a stop condition and an output contract. Four working principles drive it: -1. **Frontier-rounds loop**. A question is ready once every decision it rests on has an answer, and the set of ready questions is the **frontier**. On a webhook-retry task, the retry budget is ready on day one, while the alert threshold rests on that budget and so sits out until the budget is answered. Each round numbers every ready question, orders them by how much later work each answer would reshape, largest first, sends them together (one per message under `single` pacing, "One question at a time" below), and waits; the replies decide what is ready next time. One ready question still makes a round +1. **Frontier-rounds loop**. A question is ready once every decision it rests on has an answer, and the set of ready questions is the **frontier**. On a webhook-retry task, the retry budget is ready on day one, while the alert threshold rests on that budget and so sits out until the budget is answered. Each round numbers every ready question, orders them as [`context/loop.md`](context/loop.md) "Frontier rounds" item 2 says (new questions by how much later work each answer would reshape, largest first), sends them together (one per message under `single` pacing, "Question pacing" below), and waits; the replies decide what is ready next time. One ready question still makes a round 2. **Survey-then-deep**. Before asking blind, do a fast breadth pass (repo files, recent commits, existing skills, relevant project rules) so questions land in real context 3. **Climb-to-anchor**. Find the nearest `CLAUDE.md`, `AGENTS.md`, domain-vocabulary file, or module README by walking UP from the relevant directory toward repo root; let those shape questions instead of asking what is already documented 4. **The project's terms from the start**. Once the interview starts, past the quick-change skip in [`context/routing.md`](context/routing.md), invoke `/domain-driven-design:ubiquitous-language lookup` via the Skill tool, with one line saying why ("Reading the project glossary so these questions use its terms."); when that skill is not listed, read the glossary file as loop.md "Step 1. - Survey before you ask" says, and the opening line offers its install once ("Question lanes"). + Survey before you ask" says, and the opening lines offer its plugin's install once ("Question lanes"). Agree each term in the question where it first appears and write it only after the user confirms its restatement ("Agree a term where it first appears"). Route decisions, gotchas, and conventions to their proper homes (ADR, project rules, side note) in the same response. A general session writes no repo docs. It drives to a shared-understanding summary only -**Intake the starting point.** Early in the loop (or before it), establish where the user is. One discovery question in the intake round ([`context/lookups.md`](context/lookups.md) "The intake round") that discloses their starting point; questions and recommendations calibrate to that disclosure. Route by what it shows; recommend, never auto-switch: +**Intake the starting point.** Early in the loop (or before it), establish where the user is. One discovery question in the intake round ([`context/lookups.md`](context/lookups.md) "The intake round") that discloses their starting point; questions and recommendations calibrate to that disclosure. Route by what it shows; recommend, never auto-switch, in the one-line shape [`context/lookups.md`](context/lookups.md) "The intake round" gives a routing suggestion: - **Idea stage** (a half-formed idea, "I just had this thought", the user does not yet know what they want): recommend `/planning:brainstorm`, which explores approaches and returns here with a handoff. If the user stays, follow [`problem-framing.md`](../../context/problem-framing.md) "Idea-stage requests". - **User-facing product work** whose users or success measures are unsettled: recommend `/planning:prd` first. @@ -114,7 +114,7 @@ The Q&A path of this skill is one engine wrapped in a stop condition and an outp Every other skill the interview hands to or detours through: [`context/composition.md`](context/composition.md). -**Ask the quality bar in round 1.** When the request does not already state it, round 1 asks, as a discovery question, which bar the work is for: a throwaway prototype, an internal tool, or production. When the request states it, never re-ask it; treat it as settled. Interview depth scales with the bar. A prototype gets the scoping questions only, with production concerns (accounts, scaling, monitoring, CI, coverage targets) named as deferred rather than asked. Production gets the full depth, including failure handling, data correctness, security, monitoring, and rollback; an internal tool sits between the two. When the user cannot judge which bar applies ("I don't know", "you tell me"), recommend one from evidence in the next round: who runs the result, what data it touches, and whether it ships to users. Ask them to confirm it like any recommendation. Never fall back to the prototype bar unasked. +**Ask the quality bar in round 1.** When the request does not already state it, round 1 asks, as a discovery question, which bar the work is for, its Context naming the three bars: a throwaway prototype, an internal tool, or production. When the request states it, never re-ask it; treat it as settled. Interview depth scales with the bar. A prototype gets the scoping questions only, with production concerns (accounts, scaling, monitoring, CI, coverage targets) named as deferred rather than asked. Production gets the full depth, including failure handling, data correctness, security, monitoring, and rollback; an internal tool sits between the two. When the user cannot judge which bar applies ("I don't know", "you tell me"), recommend one from evidence in the next round: who runs the result, what data it touches, and whether it ships to users. Ask them to confirm it like any recommendation. Never fall back to the prototype bar unasked. **Open with the problem when the request has none.** When the request names no problem (an idea, or a solution with no stated why), round 1's first question asks what problem it solves and why the user wants it solved; keep that answer in the user's words for the Goal's Intent line. @@ -124,7 +124,7 @@ Every other skill the interview hands to or detours through: [`context/compositi A new effort starts here, however big. When the interview shows the effort will outrun a single session AND is still too foggy to phrase as sharp questions, the user can't yet list the decisions, let alone lock them. Name `/planning:wayfind` to the user (it charts the fog as a map of decision items and works the frontier down decision by decision, graduating to a Brief once it clears); recommend, never auto-switch. The user can run it in this same session: its chart step reads this topic's ledger, so settled answers become decisions so far and open questions become decision items. -**Question budget scales with what's already settled.** Upstream artifacts, research findings, exploration output, a PRD, a design resolution, count as settled prerequisites: an interview invoked after them starts with a smaller tree and fewer rounds; never re-ask what an artifact already answers. There is no numeric question cap, but a frontier that keeps *ballooning* (each round opens more branches than it closes) is the wayfind signal above, not a license for a marathon session. Surface the routing recommendation instead of grinding on. +**Question budget scales with what's already settled.** Upstream artifacts, research findings, exploration output, a PRD, a design resolution, count as settled prerequisites: an interview invoked after them starts with a smaller tree and fewer rounds; never re-ask what an artifact already answers. There is no numeric question cap (the shared lane's round cap, [`context/lane-shared.md`](context/lane-shared.md) "Round cap", limits a round, not the interview), but a frontier that keeps *ballooning* (each round opens more branches than it closes) is the wayfind signal above, not a license for a marathon session. Surface the routing recommendation instead of grinding on. **Bulk application work is not a decision set. The interview settles it once, it never executes it.** When the task applies a settled shape across a corpus (every document in a set, every file matching a pattern, every page of a spec), the interview's output is TWO things and no more: the small set of genuinely contested decisions, driven to decisions the normal way, plus an **execution contract**. One line in the session's output artifact, routed by domain like every other output: the Brief's `### Acceptance criteria` in an engineering session, the shared-understanding summary in a general one. Naming the per-unit close-out loop (one source unit at a time: apply, verify, close) and what *closed* means for a unit. NEVER enumerate the per-unit applications as decision rows: an interview that becomes the execution container turns every application step into a decision needing its own adoption ceremony, and the corpus's size becomes the session's length. **Tripwire. Count the candidates against the corpus:** when the candidate question count scales with the number of source units rather than with the number of genuine forks, that is execution masquerading as decisions. Collapse the per-unit rows into the loop, keep only the forks, and say what you collapsed. Distinct from the ballooning frontier above and remedied differently: ballooning routes to `/planning:wayfind`, unit-scaling collapses into the execution contract and hands off to execution. @@ -136,10 +136,11 @@ Tone is collaborative but opinionated. You are not interrogating; you are helpin Read [`context/relentless-mode.md`](context/relentless-mode.md) before the first round when the action resolves to `me`. It holds what relentless mode means and its canonical framing. The round mechanics below apply to every mode that asks questions. -**Ask each round inline as one numbered set.** Per-question shape within the round: +**Ask each round inline as one numbered set.** Per-question shape within the round, as the saved round file holds it (the full block; "Keep the layout whole" says what each detail level prints): ```text Q: **** + Context: - () - () @@ -161,49 +162,57 @@ Basis: verified, Alternatives: - (a)