diff --git a/.changes/planning/feat-interview-redesign-07635b38.md b/.changes/planning/feat-interview-redesign-07635b38.md new file mode 100644 index 00000000000..22d16072b26 --- /dev/null +++ b/.changes/planning/feat-interview-redesign-07635b38.md @@ -0,0 +1,18 @@ +--- +bump: minor +--- + +### Added + +- **Stable question ids.** `round.sh next-id --ledger [--dir ] [--count ]` prints the next free ids after the highest one held, and the ledger export keeps every number a ledger row already has instead of renumbering. `check-open-questions.sh` accepts a gap in ids only where a hand-written `- gap | Q-Q | ` line declares it, and `export-ledger --ledger` keeps those lines. `export-ledger` and `sync-ledger` write `- gap | Q-Q | never assigned` for ids nothing ever held, never for one the ledger held, and a gap line past the last row reserves those ids. +- **Register tags and `round.sh reopen`.** A register row's resolution can end in `kind::`, `problem::`, `depends::`, `reopened::` and `recheck::` tags, which import and export carry unchanged. `round.sh reopen --ledger Q --round --fact-file ` reopens an answered row under its own number, keeps the old answer and the fact in `## Reopened answers`, and flags every row built on it with `recheck::`, which blocks `check-open-questions.sh` until `reopen --rechecked` clears it. +- **Round detail levels, `scope-check` and `measures`.** `round.sh show --level full|standard|minimal ` prints a saved round at that detail level (full byte for byte; no level hides the recommendation with its reason, `Withheld:`, `Discovery:`, `Second view:`, `Departs from:`, `Flips if:` or `Commits you to:`), set by the new `question_detail` setting (default `standard`). `round.sh scope-check` prints the counts for a round header's `Scope check:` line, and `round.sh measures` prints rounds, median lines per shown round and reopened answers. `check-round-codes.sh` also fails a question with no single `My recommendation:`, `Withheld:`, `Discovery:` or `Strong answer:` line, and a round header without exactly one `Scope check:` line. +- **Interview flow: gap triage, held questions, a blind recommender and stable ids.** Before round 1, `/planning:interview` sorts every unknown into a lookup, a discovery question or a decision question; when discovery questions exist, round 1 asks only those, open, each with a `Discovery:` line and no recommendation (new `context/lookups.md`). A question a running lookup could change is held on the `Held:` line, at most one dispatch runs per lookup skill, and a fact that contradicts an answer runs `round.sh reopen`. Each round's recommendations come from the new read-only `interview-recommender` agent, which sees the facts, constraints and settled decisions but not the conversation; a differing draft shows as `Second view:` and a departure from a settled decision as `Departs from:`. New ids come from `round.sh next-id`, problems are numbered P1, P2 and tagged on their questions, the round header's `Scope check:` line comes from `round.sh scope-check`, the end sweep reports every finding for the disposition pass to filter, the glossary is read at the start of every interview and a term is written only after the user confirms its restatement, and `/planning:audit-answers` never auto-accepts a `kind:: discovery` row. +- **Interview layout, opening lines and one home per rule.** The round file holds the full question block and the round you see is `round.sh show` output at `question_detail`, with a blank line after the title, the recommendation block and the alternatives, a `---` line between questions and the header's `Scope check:` line; at every level the recommendation's reason ends with its evidence label and source, and "more Q" or "full round" shows the rest. The opening lines now read `Asking here, one round at a time. A local page or a shared session is also available; say which to switch.`, `Questions are saved to .` and one `Optional:` line per planner or plugin offer (`surfaces/registry.mjs resolve` gains `--pacing` and `--ledger`), and the glossary plugin is offered when it is not installed. Each rule stated two ways now has one home that the other files link to: round order, ids and gaps, the shared lane's 8-question round cap, when lookups and the passes around the interview run, the three glossaries, scenario ids and when the Brief is persisted. A term's restatement is registered like any question, and a routing suggestion in an intake round is one line, never a `My recommendation:` block. Unattended, an open discovery question is `blocked` with arbiter USER-RESERVED. +- **Interview fixes from the eval run.** `round.sh next-id --round --limit ` also prints the round's `Scope check:` line, counting the new rows, so one call serves a round's ids and its header; the skill now states that `--limit` takes the resolved `wayfind_suggest_questions` integer, `27` when no layer sets it. The opening `line` is printed exactly as given, the checklist template is read and written rather than copied by shell, and under `single` pacing the `Later:` line never previews the frontier. A prescribed solution's rethink-or-accept question and a reported problem's instance question go out in the intake round, a doubt about a stated fix is a captured assumption, a rejected technical recommendation gets one crux question, a control read only in a document is `documented, unverified`, facts from the request cite `user, round 1`, unattended runs carry the coverage line word for word, and a scenario walk holds only scenarios the facts settle. +- **Interview fixes from the second eval run.** Every lane, inline rounds included, reads `context/question-shape.md` before its first question, so its Context rules reach the conversation. A Context claim cites its own source, never another question, and facts in a message that resumes past round N cite the round it asks for. A recommendation that leans on a control read only in a document carries `documented, unverified`, a reason resting on a `judgment` line is researched or withheld, a fact a `Second view:` line raises goes on the affected question's `Flips if:` line, and the quality bar is never settled by evidence. The register row's question is the title as shown, and a scenario walk puts each scenario in its own bullet under `Context:` with its own person and amount, day or date, and states no definition first. +- **The round check enforces four interview rules.** `scripts/check-round-codes.sh` fails a `## Terminology matrix` term that a question uses without mentioning it in its own Context (`undefined=`), and with `--pacing single` it fails any header line other than the round title, `Scope check:`, `Later:`, `Held:` and `Settled by evidence:` (`extra=`). It also prints `check=:` self-checks (`own-context-terms`, `flips-if-fact`, `existing-control`, `single-pacing`). They never change the exit code, and the skill sends a round only after each one is answered. Two eval graders now accept what the skill prescribes: the intent question in its yes-accepts form, and the `Pacing:` line beside the surface offer. + +### Removed + +- **The interview's per-round contradiction pass.** `/planning:interview` no longer dispatches a sub-agent before each round to check the draft against earlier answers. The end sweep still finds contradictions and asks each as one question naming both sides, and an answer that conflicts with an earlier one is still caught when it is given. The legacy eval case `pre-round-contradiction-pass` is removed with it. diff --git a/.changes/session-flow/feat-interview-redesign-3c9e1a52.md b/.changes/session-flow/feat-interview-redesign-3c9e1a52.md new file mode 100644 index 00000000000..64e0fb139b8 --- /dev/null +++ b/.changes/session-flow/feat-interview-redesign-3c9e1a52.md @@ -0,0 +1,7 @@ +--- +bump: patch +--- + +### Changed + +- **The workflow navigator points at `/planning:interview` for its own rules.** The interview stage in `context/steps.md` and the ladder in `context/pre-implementation-order.md` no longer restate how the interview paces its questions or when explore, research, a PRD or a prototype runs around it; both name `/planning:interview` as the owner of those rules. diff --git a/.changes/work-items/feat-interview-redesign-559d7912.md b/.changes/work-items/feat-interview-redesign-559d7912.md new file mode 100644 index 00000000000..c087268f021 --- /dev/null +++ b/.changes/work-items/feat-interview-redesign-559d7912.md @@ -0,0 +1,5 @@ +--- +bump: none +--- + +No release: the eval case `decompose-decisions-no-coined-ids` gains the `no-network` tag, which only the eval validity check reads; no shipped file changes. diff --git a/docs/contracts/verification-level/README.md b/docs/contracts/verification-level/README.md index 23304d8b8be..371a4200ca9 100644 --- a/docs/contracts/verification-level/README.md +++ b/docs/contracts/verification-level/README.md @@ -77,8 +77,8 @@ deliverable is not governed. These skills do not read the level: `/planning:audi `/performance:verify`, `/discipline:scrutinize-dont-coast`, `/review:quality-gate`, `/review:fanout`, `/planning:devils-advocate`, `/review:security-review` and `/planning:interview`. Each is itself the independent check when invoked, except -`/planning:interview`, whose contradiction pass, assumption sweep and quote check produce the -questions it exists to ask. +`/planning:interview`, whose assumption sweep and quote check produce the questions it exists to +ask. `/coupling:reduce` and the conventions lens of `/architecture:improve` do not read it either: reproducing a scan finding produces the finding they apply or propose for recording, so it runs on diff --git a/plugins/planning/.claude-plugin/plugin.json b/plugins/planning/.claude-plugin/plugin.json index 331f865ce02..f1926e5fb04 100644 --- a/plugins/planning/.claude-plugin/plugin.json +++ b/plugins/planning/.claude-plugin/plugin.json @@ -23,6 +23,17 @@ "auto" ] }, + "question_detail": { + "type": "string", + "title": "Interview question detail", + "description": "How much of each question /planning:interview prints: standard (default; hides Origin, Basis, Checked against and the source quote), full (the whole block) or minimal (answer lines and an alternatives count). The round file keeps the full block; a session request or planning.yaml overrides it.", + "default": "standard", + "options": [ + "full", + "standard", + "minimal" + ] + }, "use_ask_user_question": { "type": "boolean", "title": "AskUserQuestion question rounds", diff --git a/plugins/planning/README.md b/plugins/planning/README.md index 0aef2a82505..6cc481b7e46 100644 --- a/plugins/planning/README.md +++ b/plugins/planning/README.md @@ -144,6 +144,7 @@ these and hands them to the plugin, see [User configuration](https://code.claude | --- | --- | --- | --- | | `surface` | string | `"terminal"` | Where /planning:interview asks its rounds: terminal (default, inline) or page (a local 127.0.0.1 page the session watches). Unset: terminal, and the interview names the other available surfaces in one line. | | `question_pacing` | string | `"round"` | How /planning:interview asks a frontier: round (default, one message), single (one question per message, largest blast radius first) or auto (single for a broad, open-ended topic, else round). A session request overrides it there; a repository's planning.yaml overrides this value. | +| `question_detail` | string | `"standard"` | How much of each question /planning:interview prints: standard (default; hides Origin, Basis, Checked against and the source quote), full (the whole block) or minimal (answer lines and an alternatives count). The round file keeps the full block; a session request or planning.yaml overrides it. | | `use_ask_user_question` | boolean | `false` | When enabled, the planning skills' question rounds (interview, prd, design, plan) render a round of independent questions that fits one AskUserQuestion call instead of inline prose. Default: inline prose (dictation-friendly). | | `use_emoji_question_markers` | boolean | `true` | When on, each interview round question leads with a ❓ anchor on its Q line and its My recommendation line leads with ➡️, in the terminal and on the page surface. Purely presentational: persisted artifacts never carry the emoji. Default: on; turn off for plain text. | | `shared_session_open_browser` | boolean | `true` | When on, starting a /planning:interview shared session opens the owner's page in the default browser. Turn off to open it yourself. Default: on. | diff --git a/plugins/planning/agents/interview-recommender.md b/plugins/planning/agents/interview-recommender.md new file mode 100644 index 00000000000..b8177cb733e --- /dev/null +++ b/plugins/planning/agents/interview-recommender.md @@ -0,0 +1,37 @@ +--- +name: interview-recommender +description: "Blind recommender for a /planning:interview round: given the round's decision questions with their facts, the constraint ledger and the settled decisions, but none of the conversation, returns one recommendation per question with its basis and any departure from a settled decision. Read-only. Dispatched by the interview skill's Step 2; not intended for direct ad-hoc use." +tools: "Read, Grep, Glob" +model: inherit # reason: the recommendation should come from the session's own model, so no model is named here +maxTurns: 15 +--- + +You recommend an answer to each decision question of one interview round. You start with no +conversation history, by design: you never see what the user said, how the session reasoned, or +which answer the session would pick, so your recommendation rests only on the evidence below. + +Your prompt carries: + +- the round's decision questions, each with its id, its wording, its options (unranked) and its + Context facts, every fact with its source; +- the constraint ledger rows; +- the settled decisions: each answered question with its answer; +- the self-answered facts, each with its source. + +Read a cited source when a fact's wording leaves the answer open; read nothing the prompt does not +point at. Recommend from these facts only, never from recall or convention. Treat a settled +decision as an input you may depart from, not as closed: when the facts point away from one, say +so. When no fact settles a question, withhold the recommendation and name the evidence that would. + +Return one line per question, in the prompt's order, and nothing else: no reasoning, no preamble. + +```text +Q | recommend: | basis: | flips if: | departs from: : one sentence, or none> +Q | withheld: +``` + +**Model.** We set `model: inherit` and no `effort`: the recommender inherits the session's model, +so it judges at the session's own level, and this file names no model. + +- **Pointer:** for what `inherit` and an absent `effort` field mean, see + [create custom subagents: choose a model](https://code.claude.com/docs/en/sub-agents#choose-a-model). diff --git a/plugins/planning/context/problem-framing.md b/plugins/planning/context/problem-framing.md index 934911723e9..48202f66a28 100644 --- a/plugins/planning/context/problem-framing.md +++ b/plugins/planning/context/problem-framing.md @@ -47,18 +47,21 @@ the Brief. A request that names a solution and no problem ("add a retry wrapper") gets its problem worked back out: what failure or gap would this solution fix? State that problem with its evidence rating, then ask one question: rethink the approach against the problem, or accept the prescribed -solution. An attended run always asks, however obvious the problem looks. A request that states -its problem is restated and confirmed, and gets no rethink-or-accept question: its named solution -is accepted. No question reopens it, including one that asks to choose or settle the mechanism +solution. An attended run always asks, however obvious the problem looks; in `/planning:interview` +outside `lock` this is the intake round's problem question, asked open in discovery shape +([`../skills/interview/context/lookups.md`](../skills/interview/context/lookups.md) "The intake +round"). A request that states its problem is restated and confirmed, and gets no +rethink-or-accept question: its named solution is accepted. No question reopens it, including one that asks to choose or settle the mechanism against an alternative (a durable queue instead of the named wrapper); questions about the solution stay on its parameters. A concern that the solution may not cover the problem is recorded -under `### Captured assumptions` with the condition that would revisit it, and stated in one line, -never asked. +under `### Captured assumptions` with the condition that would revisit it, never asked; a round +states it in at most one line, which names no other cause and no other approach. ## Reported-only evidence -A stated problem whose evidence is only `reported` gets one instance question, asked with the -restatement: the last concrete instance, what it cost, and who hit it. It is one question, never +A stated problem whose evidence is only `reported` gets one instance question, paired with the +restatement's confirmation ask: one question that asks the user to confirm the restated problem +and to give the last concrete instance, what it cost, and who hit it. It is one question, never a why chain, and it asks about the problem's evidence only: it never reopens the named solution. Under `lock` it joins the one confirmation question. The answer rewrites the Evidence line (an artifact the user points at is `observed`; an instance the user recounts stays `reported`), and no diff --git a/plugins/planning/context/recommendation-basis.md b/plugins/planning/context/recommendation-basis.md index 9fb2064266d..9e7c92f2ffa 100644 --- a/plugins/planning/context/recommendation-basis.md +++ b/plugins/planning/context/recommendation-basis.md @@ -25,7 +25,11 @@ convention, with its boundary and adopters, is bar; any other may rest on judgment if its label says so. - **Three outcomes.** Every recommendation ends in exactly one: - **verified**: presented with `Basis: verified` plus what verified it (a - `file:line`, tool output, or URL fetched this session); + `file:line`, tool output, or URL fetched this session). A source verifies + what it holds, not what it describes: a control (a protection, ruleset, + check or policy) read only in a document, not in its enforcing source (the + ruleset, the workflow, the setting a tool reports), is labelled + `documented, unverified` with the document's `file:line`, never `verified`; - **judgment**: presented with `Basis: judgment`; only for a recommendation that is not consequential; - **withheld**: a consequential recommendation that cannot be settled. It diff --git a/plugins/planning/evals/brainstorm-spec-stage-no-candidates/case.yaml b/plugins/planning/evals/brainstorm-spec-stage-no-candidates/case.yaml index 2206b449b73..47100049fb1 100644 --- a/plugins/planning/evals/brainstorm-spec-stage-no-candidates/case.yaml +++ b/plugins/planning/evals/brainstorm-spec-stage-no-candidates/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: brainstorm-spec-stage-no-candidates description: A spec-stage interview request that states its problem and a solution gets no candidate list and no reopened mechanism, because the idea-stage rule does not apply to it. Needs --allow-tools Write. -tags: [interview, brainstorm, problem-framing, stated-problem, scaffold] +tags: [interview, brainstorm, problem-framing, stated-problem, scaffold, no-network] expected_outcome: The reply lists no alternative approaches, asks no question that reopens streaming the export as the fix, does not run the brainstorm skill, and writes no PLAN.md; it may restate the problem and ask about the streaming change's details. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-adr-verdict-overruled/case.yaml b/plugins/planning/evals/interview-adr-verdict-overruled/case.yaml index 23402957cda..12c32182255 100644 --- a/plugins/planning/evals/interview-adr-verdict-overruled/case.yaml +++ b/plugins/planning/evals/interview-adr-verdict-overruled/case.yaml @@ -1,10 +1,10 @@ schema_version: "1.1" name: interview-adr-verdict-overruled description: "The closing restatement gave a 'no ADR' verdict on a decision and the user overrules it in the reply that confirms the rest. Why hard: the reply reads as a plain confirmation, so the easy path writes the Brief and drops the overrule, or argues the verdict again instead of handing the decision over." -tags: [interview, confirmation, adr, closing-summary] +tags: [interview, confirmation, adr, closing-summary, no-network] expected_outcome: "The overruled decision (where failed deliveries are stored) goes to /architecture:record-decision with its rationale and alternatives, or, when that skill is not available, is recorded as an ADR candidate in its register row with that said; the verdict is not argued again." execution: - prompt: "/planning:interview Add retries to the partner webhook sender.\n\nWhere we are: an engineering session, domain_docs is auto. Every question is answered and the assumption sweep added nothing. Your closing restatement listed these ADR verdicts: Q1 retry count: no ADR, no real alternative weighed. Q2 backoff schedule: no ADR, no real alternative weighed. Q3 failed-delivery storage: you recommended a Postgres table over a Redis stream and I agreed; verdict no ADR, weighed but easy to reverse. Q4 alert channel: no ADR, a process decision.\n\nMy reply: Q3: record it. The rest is confirmed." + prompt: "Use the interview skill from the planning plugin: Add retries to the partner webhook sender.\n\nWhere we are: an engineering session, domain_docs is auto. Every question is answered and the assumption sweep added nothing. Your closing restatement listed these ADR verdicts: Q1 retry count: no ADR, no real alternative weighed. Q2 backoff schedule: no ADR, no real alternative weighed. Q3 failed-delivery storage: you recommended a Postgres table over a Redis stream and I agreed; verdict no ADR, weighed but easy to reverse. Q4 alert channel: no ADR, a process decision.\n\nMy reply: Q3: record it. The rest is confirmed." allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit] max_turns: 20 timeout_seconds: 900 diff --git a/plugins/planning/evals/interview-bare-vague-build-asks/case.yaml b/plugins/planning/evals/interview-bare-vague-build-asks/case.yaml index 8033af97211..fdc264f2e80 100644 --- a/plugins/planning/evals/interview-bare-vague-build-asks/case.yaml +++ b/plugins/planning/evals/interview-bare-vague-build-asks/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-bare-vague-build-asks description: A bare vague build request with no slash command gets a question about the problem or scope before any source is written -tags: [interview, bare-prompt, vague-build, scaffold] +tags: [interview, bare-prompt, vague-build, scaffold, no-network] expected_outcome: The reply asks the user about the problem, the users or the scope of notifications (which events, which channel, who receives them) before building; no Write or Edit touches a file under src/. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-bare-vague-explore-no-extra/case.yaml b/plugins/planning/evals/interview-bare-vague-explore-no-extra/case.yaml index 3dfa87fdf2a..e7114ee833f 100644 --- a/plugins/planning/evals/interview-bare-vague-explore-no-extra/case.yaml +++ b/plugins/planning/evals/interview-bare-vague-explore-no-extra/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-bare-vague-explore-no-extra description: A bare vague exploratory question gets an answer, not scoping questions or an interview -tags: [interview, bare-prompt, negative, scaffold] +tags: [interview, bare-prompt, negative, scaffold, no-network] expected_outcome: The reply explains what the retry code in src/ does (retries a call up to 3 times with a doubling delay on ConnectionError) without asking the user scoping questions and without starting an interview. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-brief-keeps-adr-verdicts/case.yaml b/plugins/planning/evals/interview-brief-keeps-adr-verdicts/case.yaml index 61d210e47a5..aa03e834918 100644 --- a/plugins/planning/evals/interview-brief-keeps-adr-verdicts/case.yaml +++ b/plugins/planning/evals/interview-brief-keeps-adr-verdicts/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-brief-keeps-adr-verdicts description: "The user confirms a closing restatement whose ADR verdicts say no decision gets an ADR, and the interview writes the Brief. Why hard: the verdicts look like chat bookkeeping and the Brief template's familiar sections hold everything else, so the easy path writes the Brief without them and the one record of why the storage choice got no ADR is lost." -tags: [interview, brief, adr, closing-summary, scaffold] +tags: [interview, brief, adr, closing-summary, scaffold, no-network] expected_outcome: ".work/webhook-retries/PLAN.md has a ### ADR verdicts section under ## Brief with one line per decision Q1 to Q4, each carrying its verdict, including Q3 (failed-delivery storage in a Postgres table, Redis stream weighed) with the verdict ADR candidate, not checked." context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-closing-probe-something-else/prompt.md b/plugins/planning/evals/interview-closing-probe-something-else/prompt.md index e633376d027..33d6d4d61f3 100644 --- a/plugins/planning/evals/interview-closing-probe-something-else/prompt.md +++ b/plugins/planning/evals/interview-closing-probe-something-else/prompt.md @@ -1,9 +1,9 @@ --- description: "Eval-first case for the closing probe wording. Why hard: 'Anything else?' is the habitual close, and it invites a 'no'; 'Is there something else...?' left fewer concerns unmet in a primary-care trial (Heritage et al. 2007)." -tags: [interview, closing-probe, wording] +tags: [interview, closing-probe, wording, no-network] runs: 3 -max_turns: 12 -timeout_seconds: 600 +max_turns: 30 +timeout_seconds: 900 allowed_tools: [Read, Glob, Grep, Skill] expected_outcome: "Round 1 closes with a probe worded 'something else' (for example 'Is there something else that would change these recommendations?') and does not use 'anything else'" --- diff --git a/plugins/planning/evals/interview-coded-terms-round-answerable/graders/answerable-without-session.md b/plugins/planning/evals/interview-coded-terms-round-answerable/graders/answerable-without-session.md index 690b1e836f9..587a95d516f 100644 --- a/plugins/planning/evals/interview-coded-terms-round-answerable/graders/answerable-without-session.md +++ b/plugins/planning/evals/interview-coded-terms-round-answerable/graders/answerable-without-session.md @@ -18,7 +18,9 @@ PASS only if all of these hold: substitute (for example "merge robot" for the pr-merge lane, or "latest version" for the head SHA) is not. 3. The unanswered rerun-cap question is shown again with its context, recommendation, basis and - alternatives, not as a one-line reminder. + alternatives, not as a one-line reminder. In this check and the FAIL line below, a question + whose recommendation is withheld meets "recommendation and basis" with a `Withheld:` line that + names the evidence that would settle it; it still needs its context and alternatives. FAIL if any question needs the ledger or the session to be understood, if a technical term was replaced by a metaphor or lay wording, if the rerun-cap question is restated without its diff --git a/plugins/planning/evals/interview-coded-terms-round-answerable/prompt.md b/plugins/planning/evals/interview-coded-terms-round-answerable/prompt.md index e98e4445a15..0d209ff5f59 100644 --- a/plugins/planning/evals/interview-coded-terms-round-answerable/prompt.md +++ b/plugins/planning/evals/interview-coded-terms-round-answerable/prompt.md @@ -1,6 +1,6 @@ --- description: "A resumed interview whose ledger names every constraint, answer and coined term by code (C1 to C3, Q1 to Q4, R-cap). Why hard: the ledger is in the agent's context, so 'Checked against: C1, C3' or 'rests on Q3' reads as complete to it, while a reader with none of the session's knowledge cannot answer; the easy repair, a lay rewording, drops the technical terms the owner uses." -tags: [interview, plain-names, scaffold] +tags: [interview, plain-names, scaffold, no-network] runs: 3 max_turns: 20 timeout_seconds: 900 diff --git a/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/context-claims-referenced.md b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/context-claims-referenced.md index b0ba135a6f5..40d7e827c2c 100644 --- a/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/context-claims-referenced.md +++ b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/context-claims-referenced.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '^(?![\s\S]*?(?:(?:^|\n)[ \t>*_-]*(?:\*\*)?Context(?:\*\*)?[ \t]*:(?:\*\*)?(?:(?![ \t]*(?:\n|$))(?:(?!(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d))[^\n])+(?=\n|$)|(?:[ \t]*\n(?:[ \t]*\n)?|[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d)[^\n]*\n)(?:(?![ \t>*_-]*(?:\*\*)?(?:❓|➡|My recommendation|Recommendation\b|Proposal\b|Basis\b|Checked against|Alternatives\b|Source[ \t]*:|Inference\b|Flips if|Withheld\b|Commits you to|Q\d+[.):]))[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d)[^\n]*\n)*(?![ \t>*_-]*(?:\*\*)?(?:❓|➡|My recommendation|Recommendation\b|Proposal\b|Basis\b|Checked against|Alternatives\b|Source[ \t]*:|Inference\b|Flips if|Withheld\b|Commits you to|Q\d+[.):]))(?![ \t]*(?:\n|$))(?:(?!(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d))[^\n])+(?=\n|$))|(?:^|\n)[^A-Za-z0-9\n]*Q\d+[.):](?:(?!\n[ \t>*_-]*(?:\*\*)?Context(?:\*\*)?[ \t]*:)[\s\S])*?\n[ \t>*_-]*(?:\*\*)?(?:➡|My recommendation)))[\s\S]*?(?:^|\n)[ \t>*_-]*(?:\*\*)?Context(?:\*\*)?[ \t]*:(?:\*\*)?(?:[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d)|[ \t]*\n(?:[ \t]*\n)?[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d))' +pattern: '^(?![\s\S]*?(?:(?*_-]*(?:\*\*)?Context(?:\*\*)?[ \t]*:(?:\*\*)?(?:(?![ \t]*(?:\n|$))(?![^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d))|(?:[ \t]*\n(?:[ \t]*\n)?|(?=[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d))[^\n]*\n)(?=((?:(?![ \t>*_-]*(?:\*\*)?(?:❓|➡|My recommendation|Recommendation\b|Proposal\b|Basis\b|Checked against|Alternatives\b|Source[ \t]*:|Inference\b|Flips if|Withheld\b|Discovery\b|Held\b|Second view|Departs from|Scope check|Commits you to|Q\d+[.):]))(?=[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d))[^\n]*\n)*))\1(?![ \t>*_-]*(?:\*\*)?(?:❓|➡|My recommendation|Recommendation\b|Proposal\b|Basis\b|Checked against|Alternatives\b|Source[ \t]*:|Inference\b|Flips if|Withheld\b|Discovery\b|Held\b|Second view|Departs from|Scope check|Commits you to|Q\d+[.):]))(?![ \t]*(?:\n|$))(?![^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d)))|(?*_-]*(?:\*\*)?Context(?:\*\*)?[ \t]*:)[^\n]*)*?\n[ \t>*_-]*(?:\*\*)?(?:➡|My recommendation)))[\s\S]*?(?*_-]*(?:\*\*)?Context(?:\*\*)?[ \t]*:(?:\*\*)?(?:(?=[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d))|[ \t]*\n(?:[ \t]*\n)?(?=[^\n]*?(?:https?://|\]\(|[\w./-]+\.[A-Za-z]\w*:\d|[\d._-]*[A-Za-z][\w.-]*/[\w./-]*:\d|\b(?:README|LICENSE|CHANGELOG|NOTICE|CODEOWNERS|[A-Za-z]*file)\b:\d|\bjudgment\b|\buser, round \d)))' flags: i arm: both --- diff --git a/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/scope-check-line.md b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-context-unfamiliar-term-upstream/prompt.md b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/prompt.md index 7eaa7096807..6e07c9084bd 100644 --- a/plugins/planning/evals/interview-context-unfamiliar-term-upstream/prompt.md +++ b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/prompt.md @@ -1,11 +1,15 @@ --- -description: "First interview round on absorbing an upstream triage tool's cluster step, which rests on a coined upstream term (dedupe fingerprint) and on what the vendored README says. Why hard: the agent has read both files, so a question that names the cluster step and the fingerprint without defining them, or a decision question that does not say what upstream does and what the repository has today, reads as complete to it while the user would have to ask before answering. The prompt states the quality bar, so round 1 asks no bar question, which has no upstream gap to show." -tags: [interview, context-block, scaffold] +description: "First interview round on absorbing an upstream triage tool's cluster step, which rests on a coined upstream term (dedupe fingerprint) and on what the vendored README says. Why hard: the agent has read both files, so a question that names the cluster step and the fingerprint without defining them, or a decision question that does not say what upstream does and what the repository has today, reads as complete to it while the user would have to ask before answering. The prompt states the quality bar and the round-1 answers, so the first decision round asks no bar question, which has no upstream gap to show, and carries a Scope check: header line." +tags: [interview, context-block, scaffold, no-network] runs: 3 max_turns: 20 timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Write] -expected_outcome: "Every question in round 1 opens with a Context block that defines the dedupe fingerprint and the cluster step wherever the question uses them, says in each decision question what the upstream README does and what triage/intake.py does today, and cites a file:line or link on each line, so the user can answer every question without a follow-up" +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write] +expected_outcome: "Every question in round 2 opens with a Context block that defines the dedupe fingerprint and the cluster step wherever the question uses them, says in each decision question what the upstream README does and what triage/intake.py does today, and cites a file:line or link on each line, so the user can answer every question without a follow-up" --- -/planning:interview me Should our triage intake gain a cluster step like the upstream triage tool's? Its docs are vendored at vendor/upstream-triage/README.md, and our intake is triage/intake.py. This is a throwaway prototype. Ask me your first round, then stop for my answers. +/planning:interview me Should our triage intake gain a cluster step like the upstream triage tool's? Its docs are vendored at vendor/upstream-triage/README.md, and our intake is triage/intake.py. This is a throwaway prototype. + +Round 1 is answered: Q1, the problem is that one crash reported by many users opens many duplicate issues; Q2, the users are the two maintainers who triage those issues. + +Ask me round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-context-unfamiliar-term-upstream/samples/scope-check-line.json b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-context-unfamiliar-term-upstream/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-coverage-prompt-asked/prompt.md b/plugins/planning/evals/interview-coverage-prompt-asked/prompt.md index 34341113346..049ff7a9357 100644 --- a/plugins/planning/evals/interview-coverage-prompt-asked/prompt.md +++ b/plugins/planning/evals/interview-coverage-prompt-asked/prompt.md @@ -1,11 +1,11 @@ --- -description: "A fully specified task whose acceptance criteria are all event-driven: while capturing them, the interview asks the one coverage prompt about a missing unwanted-behavior (IF/THEN) case and a missing state-driven (WHILE) case" -tags: [interview, acceptance-criteria, coverage-prompt] +description: "A fully specified task whose acceptance criteria are all event-driven: while capturing them, the interview asks the one coverage prompt about a missing unwanted-behavior (IF/THEN) case and a missing state-driven (WHILE) case. The prompt starts past the intake round, which asks discovery questions only, so the prompt is graded in round 2" +tags: [interview, acceptance-criteria, coverage-prompt, no-network] runs: 3 max_turns: 20 timeout_seconds: 900 allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply asks one question covering both a missing unwanted-behavior (IF , THEN ) criterion and a missing state-driven (WHILE , ) criterion" +expected_outcome: "The round 2 reply asks one question covering both a missing unwanted-behavior (IF , THEN ) criterion and a missing state-driven (WHILE , ) criterion" --- /planning:interview auto @@ -22,3 +22,7 @@ Acceptance criteria: - AC1: When `--dry-run` is passed, the tool prints each branch it would delete, one per line. - AC2: When `--dry-run` is passed, no branch is deleted. + +Round 1, the intake round, is answered: Q1, the quality bar is production, since the flag ships in the tool's next release; Q2, the users are this repository's maintainers, who run the tool on their own machines. + +Ask me round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-coverage-unattended-unexamined/prompt.md b/plugins/planning/evals/interview-coverage-unattended-unexamined/prompt.md index 93c499f00b7..11c3d06e5a3 100644 --- a/plugins/planning/evals/interview-coverage-unattended-unexamined/prompt.md +++ b/plugins/planning/evals/interview-coverage-unattended-unexamined/prompt.md @@ -1,6 +1,6 @@ --- description: "An unattended lock run with nobody to answer: the interview skips the acceptance-criteria coverage prompt and states that unwanted-behavior and state-driven coverage went unexamined, instead of asking it or dropping it silently" -tags: [interview, acceptance-criteria, coverage-prompt, unattended] +tags: [interview, acceptance-criteria, coverage-prompt, unattended, no-network] runs: 3 max_turns: 20 timeout_seconds: 900 diff --git a/plugins/planning/evals/interview-crux-on-rejection/prompt.md b/plugins/planning/evals/interview-crux-on-rejection/prompt.md index d9dc5ef62e5..332b9fef3fb 100644 --- a/plugins/planning/evals/interview-crux-on-rejection/prompt.md +++ b/plugins/planning/evals/interview-crux-on-rejection/prompt.md @@ -1,9 +1,9 @@ --- description: "Eval-first case for the disagreement move. Why hard: the user rejects two recommendations, one technical (Postgres queue versus Kafka) and one personal (the command's name). Re-arguing both, or silently giving up on both, are the easy replies; the target is one crux question on the technical one and plain deference on the name." -tags: [interview, disagreement, crux] +tags: [interview, disagreement, crux, no-network] runs: 3 -max_turns: 12 -timeout_seconds: 600 +max_turns: 30 +timeout_seconds: 900 allowed_tools: [Read, Glob, Grep, Skill] expected_outcome: "The reply asks one crux question about Q2 (what would have to be true for Kafka, or for the Postgres queue, to be right), or proposes one check that would settle it, and accepts `ship-docs` for Q3 without arguing" --- diff --git a/plugins/planning/evals/interview-discuss-sets-discussing/prompt.md b/plugins/planning/evals/interview-discuss-sets-discussing/prompt.md index 0cffc4b8f7f..d4b9e69a2f8 100644 --- a/plugins/planning/evals/interview-discuss-sets-discussing/prompt.md +++ b/plugins/planning/evals/interview-discuss-sets-discussing/prompt.md @@ -1,6 +1,6 @@ --- description: "A reply that asks to talk Q4 through instead of answering it, beside an answer to Q5 and silence on Q6. Why hard: the reply reads like engagement, so the easy moves are to record Q4 at its recommendation once the concern is addressed, or to leave its row `open` with no record that a discussion is under way." -tags: [interview, register, discussing, scaffold] +tags: [interview, register, discussing, scaffold, no-network] runs: 3 max_turns: 20 timeout_seconds: 900 diff --git a/plugins/planning/evals/interview-document-dba-replica/case.yaml b/plugins/planning/evals/interview-document-dba-replica/case.yaml index c12fbfa71db..2f1f47cdb31 100644 --- a/plugins/planning/evals/interview-document-dba-replica/case.yaml +++ b/plugins/planning/evals/interview-document-dba-replica/case.yaml @@ -1,13 +1,13 @@ schema_version: "1.1" name: interview-document-dba-replica description: With the recipient and the needed answers given up front, the interview's document surface writes one template-shaped document for the DBA into the topic's memory slice, covering every item the user named, and reports its path without quizzing the user on the subject -tags: [interview, document, outcome] +tags: [interview, document, outcome, no-network] expected_outcome: One Write creates .work/reporting-replica/document-.md holding Context, How to answer and Anything else sections, questions headed `### Q:` with answer stubs, and a question on each of the maintenance window, tolerable replica lag and backup retention. No document is written outside .work/, and the reply names the path. execution: prompt: "Use the interview skill from the planning plugin on its document surface, topic reporting-replica. We want to point the reporting dashboards at a Postgres read replica, but three things are our DBA's call, not mine. It goes to Priya, the DBA who owns the production Postgres cluster; she knows replication well but hasn't seen the dashboard work. I need back from her: the nightly maintenance window, how much replica lag the replica can run behind before she'd page someone, and how long backups are retained. Write the document." allowed_tools: [Read, Glob, Grep, Skill, Write, Bash] - max_turns: 15 - timeout_seconds: 600 + max_turns: 30 + timeout_seconds: 900 graders: - name: skill-fired type: tool_used diff --git a/plugins/planning/evals/interview-ears-convention-tags-criteria/prompt.md b/plugins/planning/evals/interview-ears-convention-tags-criteria/prompt.md index e3ed2b6d0f4..270998047a8 100644 --- a/plugins/planning/evals/interview-ears-convention-tags-criteria/prompt.md +++ b/plugins/planning/evals/interview-ears-convention-tags-criteria/prompt.md @@ -1,6 +1,6 @@ --- description: "The repository's convention home sets acceptance_criteria_format to ears: the locked Brief's acceptance criteria carry the bracketed EARS pattern tags, resolved through the plugin's authoring-formats ladder" -tags: [interview, conventions, ears, acceptance-criteria, scaffold] +tags: [interview, conventions, ears, acceptance-criteria, scaffold, no-network] runs: 3 max_turns: 30 timeout_seconds: 900 diff --git a/plugins/planning/evals/interview-ears-convention-tags/prompt.md b/plugins/planning/evals/interview-ears-convention-tags/prompt.md index fcd32df47b3..13519fa161d 100644 --- a/plugins/planning/evals/interview-ears-convention-tags/prompt.md +++ b/plugins/planning/evals/interview-ears-convention-tags/prompt.md @@ -1,6 +1,6 @@ --- description: "The repository's convention home sets acceptance_criteria_format to ears: the locked Brief's acceptance criteria carry the bracketed EARS pattern tag after each AC id" -tags: [interview, acceptance-criteria, ears, convention] +tags: [interview, acceptance-criteria, ears, convention, no-network] runs: 3 max_turns: 25 timeout_seconds: 900 diff --git a/plugins/planning/evals/interview-fact-looked-up-not-asked/prompt.md b/plugins/planning/evals/interview-fact-looked-up-not-asked/prompt.md index b6375e23256..86c3088688a 100644 --- a/plugins/planning/evals/interview-fact-looked-up-not-asked/prompt.md +++ b/plugins/planning/evals/interview-fact-looked-up-not-asked/prompt.md @@ -1,9 +1,9 @@ --- description: "The alert threshold turns on how many times the webhook sender retries today, which the user does not know and the repository's config file states. Why hard: the user's 'I don't know' invites a question back, or a request to go and check, when the fact is the interview's to look up." -tags: [interview, self-answer, facts, scaffold] +tags: [interview, self-answer, facts, scaffold, no-network] runs: 3 -max_turns: 16 -timeout_seconds: 600 +max_turns: 30 +timeout_seconds: 900 allowed_tools: [Read, Glob, Grep, Skill] expected_outcome: "The reply states the current retry count (4 retries, from config/webhooks.json) as a looked-up fact with its source and builds its alert-threshold recommendation on it; it does not ask the user how many retries there are or ask them to go and find out" --- diff --git a/plugins/planning/evals/interview-final-brief-changes-only/prompt.md b/plugins/planning/evals/interview-final-brief-changes-only/prompt.md index d492a45dbea..2eb2bc76339 100644 --- a/plugins/planning/evals/interview-final-brief-changes-only/prompt.md +++ b/plugins/planning/evals/interview-final-brief-changes-only/prompt.md @@ -1,6 +1,6 @@ --- description: "Eval-first case for the final restatement. Why hard: eight answers are on record and listing all of them looks thorough, but seven were already read back and accepted, so the list buries the one changed answer and the two assumptions the user has never seen." -tags: [interview, confirmation, restatement] +tags: [interview, confirmation, restatement, no-network] runs: 3 max_turns: 12 timeout_seconds: 600 diff --git a/plugins/planning/evals/interview-final-confirm-active-reply/prompt.md b/plugins/planning/evals/interview-final-confirm-active-reply/prompt.md index dab9b0f61cd..7a2e42a2ab8 100644 --- a/plugins/planning/evals/interview-final-confirm-active-reply/prompt.md +++ b/plugins/planning/evals/interview-final-confirm-active-reply/prompt.md @@ -1,6 +1,6 @@ --- description: "Eval-first case for the final confirmation wording. Why hard: every answer is in and the sweep is clean, so closing with 'Sound good?' reads as polite and natural, but a bare 'yes' to it confirms nothing specific." -tags: [interview, confirmation, wording] +tags: [interview, confirmation, wording, no-network] runs: 3 max_turns: 12 timeout_seconds: 600 diff --git a/plugins/planning/evals/interview-goal-keeps-user-why/prompt.md b/plugins/planning/evals/interview-goal-keeps-user-why/prompt.md index 88cafb6e64a..6e04de18a71 100644 --- a/plugins/planning/evals/interview-goal-keeps-user-why/prompt.md +++ b/plugins/planning/evals/interview-goal-keeps-user-why/prompt.md @@ -1,6 +1,6 @@ --- description: "Unattended lock whose request states the user's reason and where the work heads, and no tiebreak between goals: the Brief's Goal carries the Intent line in the user's own words, the Vision line, and 'When goals conflict, prefer: none stated'" -tags: [interview, brief, goal, intent, vision] +tags: [interview, brief, goal, intent, vision, no-network] runs: 3 max_turns: 30 timeout_seconds: 900 diff --git a/plugins/planning/evals/interview-handoff-no-design-question/prompt.md b/plugins/planning/evals/interview-handoff-no-design-question/prompt.md index ccf6aaf249c..d90708e5cff 100644 --- a/plugins/planning/evals/interview-handoff-no-design-question/prompt.md +++ b/plugins/planning/evals/interview-handoff-no-design-question/prompt.md @@ -1,6 +1,6 @@ --- description: "Contract case for the interview hand-off when the Brief is locked, the codebase and externals are understood, and the change has no design question (no new types, interface contracts, modules or data model) but is not trivially small. Why hard: the usual reflex is to route into design rounds or straight into code." -tags: [interview, handoff] +tags: [interview, handoff, no-network] runs: 3 max_turns: 12 timeout_seconds: 600 diff --git a/plugins/planning/evals/interview-handoff-small-contract-is-plan/prompt.md b/plugins/planning/evals/interview-handoff-small-contract-is-plan/prompt.md index 3b08694f565..0f9943d5ad1 100644 --- a/plugins/planning/evals/interview-handoff-small-contract-is-plan/prompt.md +++ b/plugins/planning/evals/interview-handoff-small-contract-is-plan/prompt.md @@ -1,6 +1,6 @@ --- description: "Contract case for the interview hand-off when the task is small, not design-significant, and the acceptance criteria already are the plan. Why hard: the natural reflex is to add a planning step that has nothing left to decide." -tags: [interview, handoff] +tags: [interview, handoff, no-network] runs: 3 max_turns: 12 timeout_seconds: 600 diff --git a/plugins/planning/evals/interview-handoff-unsure-midflow/prompt.md b/plugins/planning/evals/interview-handoff-unsure-midflow/prompt.md index 8e15513c021..3a28b83ee3b 100644 --- a/plugins/planning/evals/interview-handoff-unsure-midflow/prompt.md +++ b/plugins/planning/evals/interview-handoff-unsure-midflow/prompt.md @@ -1,6 +1,6 @@ --- description: "Contract case for the interview skill when the user arrives mid-flow after an interview and is unsure where that leaves the work. Why hard: guessing one specific pipeline step is wrong when the user has not said what is settled." -tags: [interview, handoff] +tags: [interview, handoff, no-network] runs: 3 max_turns: 12 timeout_seconds: 600 diff --git a/plugins/planning/evals/interview-held-until-lookup-returns/graders/held-line.md b/plugins/planning/evals/interview-held-until-lookup-returns/graders/held-line.md new file mode 100644 index 00000000000..6d44964e23d --- /dev/null +++ b/plugins/planning/evals/interview-held-until-lookup-returns/graders/held-line.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}(?:\*\*)?Held(?:\*\*)?:[^\n]*\b(?:who|alerted|alerting|routing|channel|page[sd]?|paging|PagerDuty|on-call)\b' +flags: i +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-held-until-lookup-returns/graders/held-question-not-asked.md b/plugins/planning/evals/interview-held-until-lookup-returns/graders/held-question-not-asked.md new file mode 100644 index 00000000000..6d36ed6aff2 --- /dev/null +++ b/plugins/planning/evals/interview-held-until-lookup-returns/graders/held-question-not-asked.md @@ -0,0 +1,16 @@ +--- +type: llm +arm: both +focus: last_message +--- + +PASS only if all of these hold: + +1. The reply does not ask who is alerted (or which channel alerts go to) as a question to answer + now, and gives no recommendation for it. +2. It says that question waits for the PagerDuty lookup, on a `Held:` line or an equivalent line in + the round header. +3. It asks the question about what an alert message includes. + +FAIL if the reply asks the who-is-alerted question now (with or without a recommendation), asks +nothing, or stops to wait for the lookup without asking the alert-content question. diff --git a/plugins/planning/evals/interview-held-until-lookup-returns/prompt.md b/plugins/planning/evals/interview-held-until-lookup-returns/prompt.md new file mode 100644 index 00000000000..2ba679601f2 --- /dev/null +++ b/plugins/planning/evals/interview-held-until-lookup-returns/prompt.md @@ -0,0 +1,19 @@ +--- +description: "Round 3 of a webhook-alerting interview with two open decisions, one of which rests on a lookup that runs in another session and reports back tomorrow, so it cannot return inside this run. Why hard: the easy moves are to ask both questions now, recommending a channel the lookup may rule out, or to wait and ask nothing; the skill holds the dependent question on the round header's Held: line and asks the other." +tags: [interview, lookups, held, no-network] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "Round 3 names the who-is-alerted question on a Held: line until the PagerDuty lookup returns and does not ask it, and asks the question about what an alert message includes" +--- + +/planning:interview me Add retries and failure alerting to our outbound webhook sender. + +Rounds 1 and 2 are answered: Q1, the problem is that failed deliveries are lost and nobody hears about them; Q2, the users are the receiving teams and our on-call engineers; Q3, the quality bar is production; Q4 to Q7 settled five attempts with exponential backoff from 30 seconds, a dead-letter table after the last attempt, and receivers deduplicating on `event_id`. + +Two decisions are open: who is alerted when deliveries to one endpoint keep failing, and what an alert message includes (the endpoint, its last response status, how many events are waiting). + +A lookup I started in another session is checking whether our PagerDuty account has the Events API enabled; it reports back tomorrow, and the answer decides whether alerts can page on-call or must go to the #webhooks Slack channel. + +Ask round 3 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-held-until-lookup-returns/samples/held-line.json b/plugins/planning/evals/interview-held-until-lookup-returns/samples/held-line.json new file mode 100644 index 00000000000..cbfbc0c4106 --- /dev/null +++ b/plugins/planning/evals/interview-held-until-lookup-returns/samples/held-line.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "**Round 3.** Alerting for failed deliveries.\nScope check: 9 of 12 questions registered\nHeld: Q8 (who is alerted) until the PagerDuty Events API lookup returns\n\n❓ Q9: **What does an alert message include?**", + "why": "the held question is named on the Held: line" + } + ], + "fail": [ + { + "answer": "**Round 3.** Alerting.\n\n❓ Q8: **Who is alerted when one endpoint keeps failing?**\n\n❓ Q9: **What does an alert message include?**", + "why": "no Held: line; both are asked" + }, + { + "answer": "**Round 3.**\nHeld: nothing\n\n❓ Q8: **What does an alert include?**", + "why": "the Held: line names no alert-routing question" + } + ] +} diff --git a/plugins/planning/evals/interview-idea-routes-brainstorm/case.yaml b/plugins/planning/evals/interview-idea-routes-brainstorm/case.yaml index 50b916c457f..bb08b31ad78 100644 --- a/plugins/planning/evals/interview-idea-routes-brainstorm/case.yaml +++ b/plugins/planning/evals/interview-idea-routes-brainstorm/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-idea-routes-brainstorm description: A half-formed idea with no problem or direction gets a recommendation to run /planning:brainstorm first, not a spec interview or a Brief -tags: [interview, intake, idea-stage, brainstorm, scaffold] +tags: [interview, intake, idea-stage, brainstorm, scaffold, no-network] expected_outcome: The reply recommends /planning:brainstorm as the next step for the idea (it may also ask what problem the idea solves or why the user wants it); it does not auto-invoke brainstorm, does not ask scoping or acceptance-criteria questions as if the idea were settled, and no Write or Edit touches PLAN.md. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-intake-discovery-only/graders/discovery-line.md b/plugins/planning/evals/interview-intake-discovery-only/graders/discovery-line.md new file mode 100644 index 00000000000..42fbd4f8ec1 --- /dev/null +++ b/plugins/planning/evals/interview-intake-discovery-only/graders/discovery-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}(?:\*\*)?Discovery(?:\*\*)?:[^\S\n]*\S' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-intake-discovery-only/graders/no-recommendation.md b/plugins/planning/evals/interview-intake-discovery-only/graders/no-recommendation.md new file mode 100644 index 00000000000..e7c6842ee17 --- /dev/null +++ b/plugins/planning/evals/interview-intake-discovery-only/graders/no-recommendation.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}(?:\*\*)?My recommendation(?:\*\*)?:' +flags: i +match: not_contains +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-intake-discovery-only/graders/only-open-discovery.md b/plugins/planning/evals/interview-intake-discovery-only/graders/only-open-discovery.md new file mode 100644 index 00000000000..41087817013 --- /dev/null +++ b/plugins/planning/evals/interview-intake-discovery-only/graders/only-open-discovery.md @@ -0,0 +1,20 @@ +--- +type: llm +arm: both +focus: last_message +--- + +PASS only if all of these hold: + +1. The reply asks at least one numbered question about what the request leaves unknown: the + problem notifications would solve, who they are for, or what is in scope. +2. Every question in the reply is open: none proposes or recommends an answer, offers lettered + alternatives to pick from, or states an assumed answer for the user to confirm. +3. The reply asks no decision about how to build notifications (channels, delivery mechanism, + settings screens). + +A line suggesting another skill (for example brainstorming) is not a question and does not fail +check 2. + +FAIL if any question carries a recommended or assumed answer, if the reply asks a build decision, +or if it asks no question. diff --git a/plugins/planning/evals/interview-intake-discovery-only/prompt.md b/plugins/planning/evals/interview-intake-discovery-only/prompt.md new file mode 100644 index 00000000000..74a575a1a4f --- /dev/null +++ b/plugins/planning/evals/interview-intake-discovery-only/prompt.md @@ -0,0 +1,11 @@ +--- +description: "A first request that names a feature but not its problem, users or scope. Why hard: the skill can draft plausible answers for all of them, so the easy move is a first round of recommended answers that puts words in the user's mouth; the gap triage sorts these unknowns into discovery questions, and round 1 asks only those, open, each with a Discovery: line and no recommendation." +tags: [interview, intake, discovery, no-network] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "Round 1 asks only open discovery questions (the problem, who it is for, what is in scope), each with a Discovery: line and none with a My recommendation: line, then stops for the answers" +--- + +/planning:interview Add notifications to our team task app. I'm here and will answer; ask me your first round, then stop. diff --git a/plugins/planning/evals/interview-intake-discovery-only/samples/discovery-line.json b/plugins/planning/evals/interview-intake-discovery-only/samples/discovery-line.json new file mode 100644 index 00000000000..0fd7ed7609e --- /dev/null +++ b/plugins/planning/evals/interview-intake-discovery-only/samples/discovery-line.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "❓ Q1: **What problem should notifications solve?**\n\nContext:\n- The request names the feature only (user, round 1).\n\nDiscovery: the answer sets what the feature must do.", + "why": "a Discovery: line" + } + ], + "fail": [ + { + "answer": "❓ Q1: **What problem should notifications solve?**\n\nWithheld: only you know why you want this.", + "why": "the old withheld shape, no Discovery: line" + }, + { + "answer": "This round is for discovery: we learn the problem first.\n\nQ1: **What problem?**", + "why": "the word mid-sentence, lowercase, is not the label" + } + ] +} diff --git a/plugins/planning/evals/interview-intake-discovery-only/samples/no-recommendation.json b/plugins/planning/evals/interview-intake-discovery-only/samples/no-recommendation.json new file mode 100644 index 00000000000..54a0cac2d47 --- /dev/null +++ b/plugins/planning/evals/interview-intake-discovery-only/samples/no-recommendation.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 1.** Notifications for the team task app.\nScope check: 3 of 12 questions registered\n\n❓ Q1: **What problem should notifications solve?**\n\nContext:\n- The request names the feature, not the problem it solves (user, round 1).\n\nDiscovery: the answer sets what the feature must do.\n\n---\n\n❓ Q2: **Who are they for?**\n\nDiscovery: the answer sets whose workflow the design follows.", + "why": "open discovery questions, no recommendation line" + }, + { + "answer": "Q1: **What problem should this solve?**\nI have no recommendation yet; your answer decides it.", + "why": "the word inside a sentence is not the label" + } + ], + "fail": [ + { + "answer": "❓ Q1: **What problem should notifications solve?**\n\n➡️ My recommendation: **Missed task assignments**.", + "why": "a recommendation in the intake round" + }, + { + "answer": "Q1: **Who are they for?**\n**My recommendation:** the assignees", + "why": "a bold recommendation label" + } + ] +} diff --git a/plugins/planning/evals/interview-intent-asked-recorded/graders/asks-second-variant.md b/plugins/planning/evals/interview-intent-asked-recorded/graders/asks-second-variant.md index 5bf50c5313a..1fd0fc89d41 100644 --- a/plugins/planning/evals/interview-intent-asked-recorded/graders/asks-second-variant.md +++ b/plugins/planning/evals/interview-intent-asked-recorded/graders/asks-second-variant.md @@ -1,5 +1,5 @@ --- type: regex -pattern: '\b(?:is|are|will|would|do|does|should|could|might|can|has|have|any|what|which|how)\b(?:[^\n.!?]|\.(?=\S)){0,120}\b(?:second|another|other|more|additional|different|multiple)\b(?:[^\n.!?]|\.(?=\S)){0,120}\b(?:providers?|vendors?|variants?|country|countries|clients?|tenants?)\b(?:[^\n.!?]|\.(?=\S)){0,200}\?' +pattern: '\b(?:is|are|will|would|do|does|should|could|might|can|has|have|any|what|which|how)\b(?:[^\n.!?]|\.(?=\S)){0,120}\b(?:second|another|other|more|additional|different|multiple)\b(?:[^\n.!?]|\.(?=\S)){0,120}\b(?:providers?|vendors?|variants?|country|countries|clients?|tenants?)\b(?:[^\n.!?]|\.(?=\S)){0,200}\?|\b(?:should|will|would|is|are|do|does|can)\b(?:[^\n.!?]|\.(?=\S)){0,120}\bonly\b(?:[^\n.!?]|\.(?=\S)){0,120}\b(?:no|without)\b(?:[^\n.!?]|\.(?=\S)){0,120}\b(?:providers?|vendors?|variants?|clients?|tenants?)\b(?:[^\n.!?]|\.(?=\S)){0,200}\?' flags: i --- diff --git a/plugins/planning/evals/interview-intent-asked-recorded/graders/registers-intent-row.md b/plugins/planning/evals/interview-intent-asked-recorded/graders/registers-intent-row.md index 71b897bfaaf..1dfc7d805c8 100644 --- a/plugins/planning/evals/interview-intent-asked-recorded/graders/registers-intent-row.md +++ b/plugins/planning/evals/interview-intent-asked-recorded/graders/registers-intent-row.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '(?:^|\n)- Q\d+ \| open \| round \d+ \|[^|\n]*\b(?:second|another|other|variants?|providers?|generali[sz]e|specific)\b' +pattern: '(?:^|\n)- Q\d+ \| open \| round \d+ \|[^|\n]*\b(?:second|another|other|variants?|variations?|providers?|generali[sz]e|specific)\b' flags: i target: { source: file, path: .work/address-validation/interview-checklist.md } --- diff --git a/plugins/planning/evals/interview-intent-asked-recorded/graders/scope-check-line.md b/plugins/planning/evals/interview-intent-asked-recorded/graders/scope-check-line.md new file mode 100644 index 00000000000..d4655bb06bd --- /dev/null +++ b/plugins/planning/evals/interview-intent-asked-recorded/graders/scope-check-line.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +--- diff --git a/plugins/planning/evals/interview-intent-asked-recorded/prompt.md b/plugins/planning/evals/interview-intent-asked-recorded/prompt.md index 692ee14be42..28bac3ffd35 100644 --- a/plugins/planning/evals/interview-intent-asked-recorded/prompt.md +++ b/plugins/planning/evals/interview-intent-asked-recorded/prompt.md @@ -1,11 +1,15 @@ --- -description: "Engineering task with one external provider and nothing said about others: the interview asks the intent question (is a second variant known, a category, or speculative) with a recommendation, and registers it in the ledger at ask-time" -tags: [interview, intent, engineering-defaults] +description: "Engineering task with one external provider and nothing said about others, past the intake round: the interview asks the intent question (is a second variant known, a category, or speculative) with a recommendation in the first decision round, registers it in the ledger at ask-time, and heads the round with a Scope check: line" +tags: [interview, intent, engineering-defaults, no-network] runs: 3 max_turns: 30 timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit] -expected_outcome: "Round 1 asks whether a second address-validation provider or per-client variation is known, expected as a category, or speculative, with a recommendation, and the ledger .work/address-validation/interview-checklist.md carries that question as an open register row" +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] +expected_outcome: "Round 2 asks whether a second address-validation provider or per-client variation is known, expected as a category, or speculative, with a recommendation; the round header carries a Scope check: line; and the ledger .work/address-validation/interview-checklist.md carries that question as an open register row" --- /planning:interview Add delivery-address validation to checkout using the Loqate API, so an undeliverable address is caught before payment. This goes to production. Use the topic slug `address-validation`. + +Round 1 is answered: Q1, the problem is orders shipped to addresses the carrier cannot deliver to; Q2, the users are shoppers at checkout and the support team that handles returned parcels. + +Ask round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-intent-asked-recorded/samples/asks-second-variant.json b/plugins/planning/evals/interview-intent-asked-recorded/samples/asks-second-variant.json index 21c65bb13be..afe00bc6e27 100644 --- a/plugins/planning/evals/interview-intent-asked-recorded/samples/asks-second-variant.json +++ b/plugins/planning/evals/interview-intent-asked-recorded/samples/asks-second-variant.json @@ -7,6 +7,10 @@ { "answer": "Q3: Will other clients need a different validation vendor, or is Loqate the only one?", "why": "asks about per-client variation" + }, + { + "answer": "❓ Q10: **Should this be built for Loqate and delivery addresses only, with no layer for swapping providers?**\n\n➡️ My recommendation: **Yes: build for Loqate and delivery addresses only.** No second provider appears anywhere.", + "why": "the yes-accepts form of the intent question: yes accepts the recommendation that no second provider is expected" } ], "fail": [ @@ -25,6 +29,10 @@ { "answer": "No other provider is coming, so should the Pay button stay disabled?", "why": "states the intent answer, then asks a different question" + }, + { + "answer": "Q4: Should the check run only on delivery addresses at launch, with no billing address check yet?", + "why": "a scope question in the yes-accepts form, but it asks nothing about a second provider or variant" } ] } diff --git a/plugins/planning/evals/interview-intent-asked-recorded/samples/registers-intent-row.json b/plugins/planning/evals/interview-intent-asked-recorded/samples/registers-intent-row.json index 62c664bf924..a46cf697a22 100644 --- a/plugins/planning/evals/interview-intent-asked-recorded/samples/registers-intent-row.json +++ b/plugins/planning/evals/interview-intent-asked-recorded/samples/registers-intent-row.json @@ -3,6 +3,10 @@ { "answer": "## Open-question register\n\n- Q1 | open | round 1 | What should checkout show for an undeliverable address? |\n- Q2 | open | round 1 | Is a second address-validation provider known, a category, or speculative? |\n", "why": "the intent question is registered open at ask-time" + }, + { + "answer": "## Open-question register\n\n- Q7 | open | round 2 | Should mechanism decisions wait for planning? | problem:: P1 depends:: Q6\n- Q8 | open | round 2 | Should this change build against Loqate only (variation intent)? | problem:: P1\n", + "why": "the intent row named by the skill's own term, variation intent" } ], "fail": [ diff --git a/plugins/planning/evals/interview-intent-asked-recorded/samples/scope-check-line.json b/plugins/planning/evals/interview-intent-asked-recorded/samples/scope-check-line.json new file mode 100644 index 00000000000..246aa17cb9c --- /dev/null +++ b/plugins/planning/evals/interview-intent-asked-recorded/samples/scope-check-line.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "**Round 2.** A production address check before payment, for shoppers and support.\nScope check: 3 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Is a second address-validation provider known, a category, or speculative?**", + "why": "the round header carries the Scope check: line" + } + ], + "fail": [ + { + "answer": "**Round 2.** A production address check before payment.\n\n❓ Q3: **Is a second address-validation provider known, a category, or speculative?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Is a second provider coming?**", + "why": "the words appear mid-sentence, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-intent-prototype-one-line/graders/asks-a-round.md b/plugins/planning/evals/interview-intent-prototype-one-line/graders/asks-a-round.md index c7219c03746..9b0dc5bc6a1 100644 --- a/plugins/planning/evals/interview-intent-prototype-one-line/graders/asks-a-round.md +++ b/plugins/planning/evals/interview-intent-prototype-one-line/graders/asks-a-round.md @@ -1,5 +1,5 @@ --- type: regex -pattern: '(?:^|\n)[ \t]*(?:#{1,6}[ \t]*)?(?:\*\*)?(?:Q1\b|1[.)])[^?]{0,400}\?' +pattern: '(?:^|\n)[ \t]*(?:#{1,6}[ \t]*)?(?:❓\s*)?(?:\*\*)?(?:Q[1-9]\d*\b|[1-9]\d*[.)])[^?]{0,400}\?' flags: i --- diff --git a/plugins/planning/evals/interview-intent-prototype-one-line/prompt.md b/plugins/planning/evals/interview-intent-prototype-one-line/prompt.md index 1356dc6af48..681a5c8d675 100644 --- a/plugins/planning/evals/interview-intent-prototype-one-line/prompt.md +++ b/plugins/planning/evals/interview-intent-prototype-one-line/prompt.md @@ -1,11 +1,15 @@ --- -description: "Stated throwaway prototype that reads one external store: the intent question is one line, and no quality requirement (performance, availability, security, scaling, monitoring) is asked; they are named as deferred" -tags: [interview, intent, quality-bar, prototype, engineering-defaults] +description: "Stated throwaway prototype that reads one external store: the intent question is one line, and no quality requirement (performance, availability, security, scaling, monitoring) is asked; they are named as deferred. The prompt starts past the intake round, so the variation-intent question, a decision question, comes in round 2" +tags: [interview, intent, quality-bar, prototype, engineering-defaults, scaffold, no-network] runs: 3 max_turns: 20 timeout_seconds: 600 allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit] -expected_outcome: "Round 1 asks one one-line question on whether other stores or platforms are expected (or confirms keeping it specific), asks no quality-requirement question, and names quality requirements as deferred for a prototype" +expected_outcome: "Round 2 asks one one-line question on whether other stores or platforms are expected (or confirms keeping it specific), asks no quality-requirement question, and names quality requirements as deferred for a prototype" --- /planning:interview I want a throwaway prototype that pulls last month's Shopify orders into a CSV, so I can show finance a report idea on Friday. It will never ship and only I will run it. Use the topic slug `orders-csv`. + +Round 1, the intake round, is answered and recorded in .work/orders-csv/interview-checklist.md: Q1, the report must show finance last month's order count and revenue per day; Q2, I start from nothing in this repository and already have a Shopify Admin API access token. + +Ask me round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-intent-prototype-one-line/samples/asks-a-round.json b/plugins/planning/evals/interview-intent-prototype-one-line/samples/asks-a-round.json index 30cf9ac213d..f340dccb176 100644 --- a/plugins/planning/evals/interview-intent-prototype-one-line/samples/asks-a-round.json +++ b/plugins/planning/evals/interview-intent-prototype-one-line/samples/asks-a-round.json @@ -3,6 +3,10 @@ { "answer": "Here is what I see: an empty reports package.\n\nQ1: Which order fields should the CSV carry?\nMy recommendation: **id, date, total**.", "why": "a numbered question" + }, + { + "answer": "**Round 2.** A throwaway CSV of last month's Shopify orders for Friday's demo.\nScope check: 3 of 27 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Keep the code Shopify-only?**\n\n➡️ My recommendation: **Yes, Shopify only**. It is the only store you named (verified, user, round 1: last month's Shopify orders).", + "why": "round 2 opens at Q3 behind the emoji question marker" } ], "fail": [ @@ -13,6 +17,10 @@ { "answer": "I wrote the Brief and locked it.", "why": "no question asked" + }, + { + "answer": "Plan:\n1. Pull the orders\n2. Write the CSV\n3. Show finance", + "why": "a numbered list with no question" } ] } diff --git a/plugins/planning/evals/interview-intent-prototype-one-line/scaffold.sh b/plugins/planning/evals/interview-intent-prototype-one-line/scaffold.sh index 4140b8d7167..5f0b40427a0 100755 --- a/plugins/planning/evals/interview-intent-prototype-one-line/scaffold.sh +++ b/plugins/planning/evals/interview-intent-prototype-one-line/scaffold.sh @@ -32,3 +32,17 @@ PY git add . git commit -q -m "chore: reports package" + +mkdir -p .work/orders-csv +cat >.work/orders-csv/interview-checklist.md <<'MD' +# /planning:interview Checklist + +## Constraint ledger + +- C1 | confirmed | a throwaway prototype that never ships; only the user runs it | user, round 1 + +## Open-question register + +- Q1 | answered | round 1 | What must the report show finance? | last month's order count and revenue per day; kind:: discovery +- Q2 | answered | round 1 | Where do you start from? | nothing in this repository yet; a Shopify Admin API access token is in hand; kind:: discovery +MD diff --git a/plugins/planning/evals/interview-intent-unattended-blocked/prompt.md b/plugins/planning/evals/interview-intent-unattended-blocked/prompt.md index 3feed12537c..0b53980e1d2 100644 --- a/plugins/planning/evals/interview-intent-unattended-blocked/prompt.md +++ b/plugins/planning/evals/interview-intent-unattended-blocked/prompt.md @@ -1,6 +1,6 @@ --- description: "Unattended lock where only a triage bot's issue note says more providers are coming: the repeat-or-vary part stays unstated, so the Brief is still written, the intent row is blocked and deferred with arbiter USER-RESERVED, and the run names that blocker" -tags: [interview, intent, unattended, engineering-defaults] +tags: [interview, intent, unattended, engineering-defaults, no-network] runs: 3 max_turns: 40 timeout_seconds: 900 diff --git a/plugins/planning/evals/interview-lock-restates-problem/case.yaml b/plugins/planning/evals/interview-lock-restates-problem/case.yaml index d3452084e60..7ed5ffcf13c 100644 --- a/plugins/planning/evals/interview-lock-restates-problem/case.yaml +++ b/plugins/planning/evals/interview-lock-restates-problem/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-lock-restates-problem description: Under lock, a task that states its problem still gets a one-line problem restatement and one confirmation question before any Brief is written -tags: [interview, problem-framing, lock, scaffold] +tags: [interview, problem-framing, lock, scaffold, no-network] expected_outcome: The reply restates the problem (the nightly invoice export timing out for accounts with more than 10,000 invoices) in one line and asks the user to confirm it with a yes, and no Write or Edit touches PLAN.md. context: scaffold_script: scaffold.sh @@ -18,7 +18,7 @@ graders: arm: with-only - name: restates-problem-and-asks type: regex - pattern: '^(?=[\s\S]*\b(?:time[sd]?[ -]?outs?|timing out|large accounts?|10,?000)\b)(?=[\s\S]*(?:\b(?:is|does|did|was)\s+(?:that|this)\b[^\n?]{0,80}\?|\b(?:got|have)\s+(?:that|this)\s+right\b[^\n?]{0,40}\?|\bconfirm\s+(?:that|this|it|the\s+(?:problem|restatement))\b[^\n]{0,160}\?|,\s*(?:right|correct)\s*\?))' + pattern: '^(?=[\s\S]*\b(?:time[sd]?[ -]?outs?|timing out|large accounts?|10,?000)\b)(?=[\s\S]*(?:\b(?:is|does|did|was)\s+(?:that|this)\b[^\n?]{0,300}\?|\b(?:got|have)\s+(?:that|this)\s+right\b[^\n?]{0,40}\?|\bconfirm\s+(?:that|this|it|the\s+(?:problem|restatement))\b[^\n]{0,160}\?|,\s*(?:right|correct)\s*\?))' flags: i arm: both - name: no-plan-write diff --git a/plugins/planning/evals/interview-lock-restates-problem/samples/restates-problem-and-asks.json b/plugins/planning/evals/interview-lock-restates-problem/samples/restates-problem-and-asks.json index c1d92f4d575..a78c6689ab3 100644 --- a/plugins/planning/evals/interview-lock-restates-problem/samples/restates-problem-and-asks.json +++ b/plugins/planning/evals/interview-lock-restates-problem/samples/restates-problem-and-asks.json @@ -3,7 +3,8 @@ {"answer": "Problem: the nightly invoice export times out for accounts with more than 10,000 invoices (observed: logs/export.log).\n\nQ1: Is that the problem? Reply yes, or correct it.", "why": "problem line, then a yes question"}, {"answer": "**Problem:** large accounts (over 10,000 invoices) hit the 1800s timeout in the nightly export.\n\nDoes this capture the problem? (yes / correct it)", "why": "question mark followed by the yes option"}, {"answer": "The problem, in one line: exports time out for large accounts. Can you confirm that before I write the Brief?", "why": "confirm phrasing"}, - {"answer": "Restating: the nightly export times out for accounts over 10,000 invoices. Is that whats going on? yes/no", "why": "restatement without the word problem, confirmed by a yes/no question about it"} + {"answer": "Restating: the nightly export times out for accounts over 10,000 invoices. Is that whats going on? yes/no", "why": "restatement without the word problem, confirmed by a yes/no question about it"}, + {"answer": "❓ Q1: **Is this the problem: the nightly invoice export times out for accounts with more than 10,000 invoices (observed, logs/export.log), and support has logged three tickets about it this month, so large accounts get no export at all?**", "why": "a long one-line problem title, past the old 80-character window"} ], "fail": [ {"answer": "", "why": "empty reply"}, diff --git a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/scope-check-line.md b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/prompt.md b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/prompt.md index c5d1e697c95..e7bafe08895 100644 --- a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/prompt.md +++ b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/prompt.md @@ -1,11 +1,15 @@ --- -description: "Hard case. Why hard: the prompt floats a side idea (rewriting the whole admin page in React) whose natural recommendation is negative, so the question about it is easily worded 'Should we rewrite it?' with a recommended 'No', where a reply of 'yes' rejects the recommendation. Upstream mattpocock/skills#706." -tags: [interview, question-wording, negative-recommendation] +description: "Hard case. Why hard: the prompt floats a side idea (rewriting the whole admin page in React) whose natural recommendation is negative, so the question about it is easily worded 'Should we rewrite it?' with a recommended 'No', where a reply of 'yes' rejects the recommendation. Upstream mattpocock/skills#706. The prompt starts past the intake round, so the first decision round carries recommendations and its Scope check: header line." +tags: [interview, question-wording, negative-recommendation, no-network] runs: 3 -max_turns: 12 -timeout_seconds: 600 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "Round 1 asks about the React rewrite, recommends keeping it out of this change, and words that question so that answering 'yes' accepts the recommendation" +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "Round 2, headed by a Scope check: line, asks about the React rewrite, recommends keeping it out of this change, and words that question so that answering 'yes' accepts the recommendation" --- /planning:interview Add a CSV export button to the admin orders page so ops can download the filtered order list. I'm also wondering whether we should rewrite the whole admin page in React while we're in there. + +Round 1 is answered: Q1, the problem is that ops copy the filtered order list by hand into a spreadsheet each morning; Q2, the users are the ops team. + +Ask round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/scope-check-line.json b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-negative-recommendation-yes-accepts/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-nfr-quality-requirement/prompt.md b/plugins/planning/evals/interview-nfr-quality-requirement/prompt.md index a164dc1394b..e7dfc0fb5f8 100644 --- a/plugins/planning/evals/interview-nfr-quality-requirement/prompt.md +++ b/plugins/planning/evals/interview-nfr-quality-requirement/prompt.md @@ -1,6 +1,6 @@ --- description: "Unattended production lock whose request states a latency target, a degraded-provider behavior and a logging restriction: the Brief records each under Constraints as a Quality requirement line, and records the Intent and Quality bar lines under Captured assumptions" -tags: [interview, quality-requirements, nfr, engineering-defaults] +tags: [interview, quality-requirements, nfr, engineering-defaults, no-network] runs: 3 max_turns: 30 timeout_seconds: 900 diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/case.yaml b/plugins/planning/evals/interview-numbering-stray-id-not-reused/case.yaml new file mode 100644 index 00000000000..7d3108b6210 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: interview-numbering-stray-id-not-reused +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/first-new-id-q601.md b/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/first-new-id-q601.md new file mode 100644 index 00000000000..3f51a4c373f --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/first-new-id-q601.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '^(?:(?!(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?(?:^|\n)[^\n]{0,6}Q601:[^\S\n]*\*\*' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/ledger-keeps-ids.md b/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/ledger-keeps-ids.md new file mode 100644 index 00000000000..4382956fecc --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/ledger-keeps-ids.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)- Q1 \| answered \|[^\n]*\n- Q2 \| answered \|[\s\S]*\n- Q600 \| answered \| round 1 \|[\s\S]*\n- Q601 \| open \|' +target: { source: file, path: .work/invoice-export/interview-checklist.md } +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/no-gap-filled.md b/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/no-gap-filled.md new file mode 100644 index 00000000000..5ea5a8aa9d1 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/graders/no-gap-filled.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q(?:[3-9]|[1-9]\d|[1-5]\d\d):[^\S\n]*\*\*' +match: not_contains +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/prompt.md b/plugins/planning/evals/interview-numbering-stray-id-not-reused/prompt.md new file mode 100644 index 00000000000..8ff6bd99e03 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/prompt.md @@ -0,0 +1,15 @@ +--- +description: "The ledger holds Q1, Q2 and a stray Q600 left by an earlier session, with a gap line declaring Q3-Q599. Why hard: the natural move is to number the next question Q3, filling the gap, or to renumber Q600 down to Q3; the skill keeps every id and takes the next one from round.sh next-id, so new questions start at Q601." +tags: [interview, numbering, register, scaffold, no-network] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] +expected_outcome: "The reply's new questions start at Q601; no new question takes Q3 to Q599; and the ledger still holds Q1, Q2 and Q600 under their numbers, with the new rows from Q601" +--- + +/planning:interview Resume the invoice-export interview; the ledger is .work/invoice-export/interview-checklist.md and round 1 is answered. + +Still open: which date range one export covers (finance closes its books on the calendar month), and how amounts are written (invoices store integer cents with a currency code, and customers are billed in USD, EUR and GBP). + +Ask round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/first-new-id-q601.json b/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/first-new-id-q601.json new file mode 100644 index 00000000000..891bca6d113 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/first-new-id-q601.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "**Round 2.** A production month-end export for finance.\nScope check: 5 of 12 questions registered\n\n❓ Q601: **Should one export cover one calendar month?**\n\nContext:\n- Finance closes on the calendar month (your notes).\n\n➡️ My recommendation: **Yes**.\n\n---\n\n❓ Q602: **How are amounts written?**", + "why": "new ids start at Q601" + } + ], + "fail": [ + { + "answer": "**Round 2.** A production month-end export for finance.\nScope check: 5 of 12 questions registered\n\n❓ Q3: **Should one export cover one calendar month?**\n\nContext:\n- Finance closes on the calendar month (your notes).\n\n➡️ My recommendation: **Yes**.\n\n---\n\n❓ Q4: **How are amounts written?**", + "why": "the gap is filled" + }, + { + "answer": "**Round 2.** A production month-end export for finance.\nScope check: 5 of 12 questions registered\n\n❓ Q602: **Should one export cover one calendar month?**\n\nContext:\n- Finance closes on the calendar month (your notes).\n\n➡️ My recommendation: **Yes**.\n\n---\n\n❓ Q601: **How are amounts written?**", + "why": "Q601 is not the first question shown" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/ledger-keeps-ids.json b/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/ledger-keeps-ids.json new file mode 100644 index 00000000000..9047fc39049 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/ledger-keeps-ids.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "## Open-question register\n\n- Q1 | answered | round 1 | What problem? | manual rebuild\n- Q2 | answered | round 1 | Who uses it? | finance\n- gap | Q3-Q599 | ids skipped by an earlier session\n- Q600 | answered | round 1 | What quality bar? | production\n- Q601 | open | round 2 | Should one export cover one calendar month? |\n", + "why": "old ids kept, new row Q601" + } + ], + "fail": [ + { + "answer": "## Open-question register\n\n- Q1 | answered | round 1 | What problem? | manual rebuild\n- Q2 | answered | round 1 | Who uses it? | finance\n- gap | Q3-Q599 | ids skipped by an earlier session\n- Q3 | answered | round 1 | What quality bar? | production\n- Q4 | open | round 2 | Should one export cover one calendar month? |\n", + "why": "Q600 renumbered down" + }, + { + "answer": "## Open-question register\n\n- Q1 | answered | round 1 | What problem? | manual rebuild\n- Q2 | answered | round 1 | Who uses it? | finance\n- gap | Q3-Q599 | ids skipped by an earlier session\n- Q600 | answered | round 1 | What quality bar? | production\n- Q3 | open | round 2 | Date range? |\n", + "why": "the new row fills the gap" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/no-gap-filled.json b/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/no-gap-filled.json new file mode 100644 index 00000000000..80df7441577 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/samples/no-gap-filled.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** A production month-end export for finance.\nScope check: 5 of 12 questions registered\n\n❓ Q601: **Should one export cover one calendar month?**\n\nContext:\n- Finance closes on the calendar month (your notes).\n\n➡️ My recommendation: **Yes**.\n\n---\n\n❓ Q602: **How are amounts written?**", + "why": "no title in Q3-Q599" + }, + { + "answer": "Context:\n- Q2 said the users are finance (round 1).\n\n❓ Q601: **Date range?**", + "why": "a cited id is not a title" + } + ], + "fail": [ + { + "answer": "❓ Q3: **Date range?**", + "why": "Q3 fills the gap" + }, + { + "answer": "❓ Q599: **Date range?**", + "why": "inside the declared gap" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-stray-id-not-reused/scaffold.sh b/plugins/planning/evals/interview-numbering-stray-id-not-reused/scaffold.sh new file mode 100755 index 00000000000..9745bb5806e --- /dev/null +++ b/plugins/planning/evals/interview-numbering-stray-id-not-reused/scaffold.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +set -euo pipefail + +mkdir -p .work/invoice-export +cat >.work/invoice-export/interview-checklist.md <<'MD' +# /planning:interview Checklist + +## Open-question register + +- Q1 | answered | round 1 | What problem does this solve? | finance rebuilds the month's invoices by hand for reconciliation +- Q2 | answered | round 1 | Who uses it? | the finance team +- gap | Q3-Q599 | ids skipped by an earlier session +- Q600 | answered | round 1 | What quality bar does this work meet? | production +MD diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/case.yaml b/plugins/planning/evals/interview-numbering-tool-display-order/case.yaml new file mode 100644 index 00000000000..c99cf7fcd1d --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: interview-numbering-tool-display-order +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/graders/calls-next-id.md b/plugins/planning/evals/interview-numbering-tool-display-order/graders/calls-next-id.md new file mode 100644 index 00000000000..8caf45c7265 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/graders/calls-next-id.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Bash +input_match: 'round\.sh\W{0,4}\s+next-id\b' +min: 1 +arm: with-only +--- diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/graders/first-new-id-q8.md b/plugins/planning/evals/interview-numbering-tool-display-order/graders/first-new-id-q8.md new file mode 100644 index 00000000000..03cbb6e0ef0 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/graders/first-new-id-q8.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '^(?:(?!(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?(?:^|\n)[^\n]{0,6}Q8:[^\S\n]*\*\*' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/graders/ids-in-display-order.md b/plugins/planning/evals/interview-numbering-tool-display-order/graders/ids-in-display-order.md new file mode 100644 index 00000000000..1ed62f2937c --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/graders/ids-in-display-order.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q(?:8:[^\S\n]*\*\*(?:(?!\n[^\n]{0,6}Q\d+:)[\s\S])*?\n[^\n]{0,6}Q(?!9:)|9:[^\n]*(?:(?!\n[^\n]{0,6}Q\d+:)[\s\S])*?\n[^\n]{0,6}Q(?!10:)|10:[^\n]*(?:(?!\n[^\n]{0,6}Q\d+:)[\s\S])*?\n[^\n]{0,6}Q(?!11:)|11:[^\n]*(?:(?!\n[^\n]{0,6}Q\d+:)[\s\S])*?\n[^\n]{0,6}Q(?!12:))\d+:' +match: not_contains +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/graders/no-reused-id.md b/plugins/planning/evals/interview-numbering-tool-display-order/graders/no-reused-id.md new file mode 100644 index 00000000000..49c50e81c23 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/graders/no-reused-id.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q[1-7]:[^\S\n]*\*\*' +match: not_contains +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/graders/registers-q8-open.md b/plugins/planning/evals/interview-numbering-tool-display-order/graders/registers-q8-open.md new file mode 100644 index 00000000000..92060f2148f --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/graders/registers-q8-open.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)- Q8 \| open \| round 3 \|' +target: { source: file, path: .work/webhook-retries/interview-checklist.md } +arm: both +--- diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/prompt.md b/plugins/planning/evals/interview-numbering-tool-display-order/prompt.md new file mode 100644 index 00000000000..37e2a05590c --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/prompt.md @@ -0,0 +1,15 @@ +--- +description: "A round is due on a ledger whose register ends at Q7. Why hard: the model can see the last id and pick the next ones itself, which is how ids once jumped or were reused; the skill says ids come from round.sh next-id, in the order the round shows the questions." +tags: [interview, numbering, register, scaffold, no-network] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] +expected_outcome: "The run calls round.sh next-id; the reply's new questions start at Q8 and run upward in the order shown; no new question reuses Q1 to Q7; and the ledger gains open rows from Q8" +--- + +/planning:interview Resume the webhook-retries interview; the ledger is .work/webhook-retries/interview-checklist.md and every question so far is answered. + +The alerting side is still open: who is alerted when deliveries to one endpoint keep failing (the on-call rota is in PagerDuty, and the team also has a #webhooks Slack channel), and what an alert message includes. + +Ask round 3 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/samples/calls-next-id.json b/plugins/planning/evals/interview-numbering-tool-display-order/samples/calls-next-id.json new file mode 100644 index 00000000000..b3b92d7c092 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/samples/calls-next-id.json @@ -0,0 +1,28 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Bash", + "input": { + "command": "bash \"/plugins/planning/surface/round.sh\" next-id --ledger .work/webhook-retries/interview-checklist.md --count 2" + } + } + ], + "why": "ids come from the tool" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/webhook-retries/interview-checklist.md" + } + } + ], + "why": "the ledger is read and ids picked by hand" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/samples/first-new-id-q8.json b/plugins/planning/evals/interview-numbering-tool-display-order/samples/first-new-id-q8.json new file mode 100644 index 00000000000..0cd5c37257d --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/samples/first-new-id-q8.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "**Round 3.** Alerting for failed deliveries.\nScope check: 9 of 12 questions registered\n\n❓ Q8: **Who is alerted when one endpoint keeps failing?**\n\nContext:\n- The on-call rota is in PagerDuty (your notes).\n\n➡️ My recommendation: **Page on-call**. (verified, your notes)\n\n---\n\n❓ Q9: **What does an alert include?**\n\nContext:\n- x\n\n➡️ My recommendation: **The endpoint and its last status**.\n\nIs there something else that would change these recommendations?", + "why": "the first question shown is Q8" + } + ], + "fail": [ + { + "answer": "**Round 3.** Alerting for failed deliveries.\nScope check: 9 of 12 questions registered\n\n❓ Q10: **Who is alerted when one endpoint keeps failing?**\n\nContext:\n- The on-call rota is in PagerDuty (your notes).\n\n➡️ My recommendation: **Page on-call**. (verified, your notes)\n\n---\n\n❓ Q11: **What does an alert include?**\n\nContext:\n- x\n\n➡️ My recommendation: **The endpoint and its last status**.\n\nIs there something else that would change these recommendations?", + "why": "the first new id skips Q8" + }, + { + "answer": "❓ Q9: **What does an alert include?**\n\n❓ Q8: **Who is alerted?**", + "why": "Q8 is not the first question shown" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/samples/ids-in-display-order.json b/plugins/planning/evals/interview-numbering-tool-display-order/samples/ids-in-display-order.json new file mode 100644 index 00000000000..e265f830f16 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/samples/ids-in-display-order.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": "**Round 3.** Alerting for failed deliveries.\nScope check: 9 of 12 questions registered\n\n❓ Q8: **Who is alerted when one endpoint keeps failing?**\n\nContext:\n- The on-call rota is in PagerDuty (your notes).\n\n➡️ My recommendation: **Page on-call**. (verified, your notes)\n\n---\n\n❓ Q9: **What does an alert include?**\n\nContext:\n- x\n\n➡️ My recommendation: **The endpoint and its last status**.\n\nIs there something else that would change these recommendations?", + "why": "Q8 then Q9" + }, + { + "answer": "❓ Q8: **Who?**\n\n---\n\n❓ Q9: **What?**\n\n---\n\n❓ Q10: **When?**", + "why": "Q8, Q9, Q10 in order" + }, + { + "answer": "❓ Q8: **Who is alerted?**\n\n➡️ My recommendation: **Page on-call and post to `#webhooks`**.\n\nCommits you to:\n- Q9: Slack destination: the channel `#webhooks` (user, round 3)\n\nAlternatives:\n- (a) Slack only.\n\n---\n\n❓ Q10: **What does an alert include?**", + "why": "the Commits you to: part takes Q9, so the next question is Q10" + } + ], + "fail": [ + { + "answer": "❓ Q8: **Who?**\n\n❓ Q10: **What?**", + "why": "Q9 is skipped" + }, + { + "answer": "❓ Q9: **What?**\n\n❓ Q8: **Who?**", + "why": "shown out of order" + }, + { + "answer": "❓ Q8: **Who is alerted?**\n\nCommits you to:\n- Q11: Slack destination: the channel `#webhooks`\n\n---\n\n❓ Q12: **What does an alert include?**", + "why": "the commitment part skips Q9" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/samples/no-reused-id.json b/plugins/planning/evals/interview-numbering-tool-display-order/samples/no-reused-id.json new file mode 100644 index 00000000000..d423a76dc78 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/samples/no-reused-id.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 3.** Alerting for failed deliveries.\nScope check: 9 of 12 questions registered\n\n❓ Q8: **Who is alerted when one endpoint keeps failing?**\n\nContext:\n- The on-call rota is in PagerDuty (your notes).\n\n➡️ My recommendation: **Page on-call**. (verified, your notes)\n\n---\n\n❓ Q9: **What does an alert include?**\n\nContext:\n- x\n\n➡️ My recommendation: **The endpoint and its last status**.\n\nIs there something else that would change these recommendations?", + "why": "no new question takes Q1 to Q7" + }, + { + "answer": "Q8: **Who is alerted?**\nContext:\n- Q4 settled five attempts (round 2).", + "why": "citing an earlier id in Context is not a question title" + } + ], + "fail": [ + { + "answer": "❓ Q1: **Who is alerted?**", + "why": "the new question reuses Q1" + }, + { + "answer": "❓ Q7: **What does an alert include?**", + "why": "the new question reuses Q7" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/samples/registers-q8-open.json b/plugins/planning/evals/interview-numbering-tool-display-order/samples/registers-q8-open.json new file mode 100644 index 00000000000..a5a876f24f6 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/samples/registers-q8-open.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "## Open-question register\n\n- Q7 | answered | round 2 | Do receivers deduplicate? | yes\n- Q8 | open | round 3 | Who is alerted when one endpoint keeps failing? |\n", + "why": "Q8 is registered open in round 3" + } + ], + "fail": [ + { + "answer": "## Open-question register\n\n- Q7 | answered | round 2 | Do receivers deduplicate? | yes\n", + "why": "no new row" + }, + { + "answer": "- Q600 | open | round 3 | Who is alerted? |\n", + "why": "a jumped id" + } + ] +} diff --git a/plugins/planning/evals/interview-numbering-tool-display-order/scaffold.sh b/plugins/planning/evals/interview-numbering-tool-display-order/scaffold.sh new file mode 100755 index 00000000000..b9ff731c007 --- /dev/null +++ b/plugins/planning/evals/interview-numbering-tool-display-order/scaffold.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +set -euo pipefail + +mkdir -p .work/webhook-retries +cat >.work/webhook-retries/interview-checklist.md <<'MD' +# /planning:interview Checklist + +## Constraint ledger + +- C1 | confirmed | webhooks are delivered at least once | user, round 1 + +## Open-question register + +- Q1 | answered | round 1 | What problem does this solve? | failed deliveries are lost and nobody hears about them +- Q2 | answered | round 1 | Who uses it? | the receiving teams and our on-call engineers +- Q3 | answered | round 1 | What quality bar does this work meet? | production +- Q4 | answered | round 2 | How many delivery attempts before the sender gives up? | five +- Q5 | answered | round 2 | What delay between delivery attempts? | exponential backoff from 30 seconds +- Q6 | answered | round 2 | What happens to an event after its last failed attempt? | it moves to a dead-letter table +- Q7 | answered | round 2 | Do receivers deduplicate repeated deliveries? | yes, on the event_id header +MD diff --git a/plugins/planning/evals/interview-option-existing-control-drops/case.yaml b/plugins/planning/evals/interview-option-existing-control-drops/case.yaml index a2e8c059f87..d2be437782b 100644 --- a/plugins/planning/evals/interview-option-existing-control-drops/case.yaml +++ b/plugins/planning/evals/interview-option-existing-control-drops/case.yaml @@ -1,14 +1,14 @@ schema_version: "1.1" name: interview-option-existing-control-drops description: An interview over an option that adds a stable branch, a workflow, a ruleset and a bot app recommends dropping them for the protection and end-to-end gate main already has -tags: [interview, recommendation, existing-control, scaffold] -expected_outcome: The first round's recommendation on how consumers get the latest settings is to keep installing from main, naming main's existing protection (blocked direct pushes, required review, required consumer-e2e check) or the pull request end-to-end job as the control that already covers the risk, and it does not recommend building the stable branch, its workflow, its ruleset or the bot app. No Write or Edit touches PLAN.md. +tags: [interview, recommendation, existing-control, scaffold, no-network] +expected_outcome: The first decision round, headed by a Scope check line, gives a recommendation on how consumers get the latest settings is to keep installing from main, naming main's existing protection (blocked direct pushes, required review, required consumer-e2e check) or the pull request end-to-end job as the control that already covers the risk, and it does not recommend building the stable branch, its workflow, its ruleset or the bot app. No Write or Edit touches PLAN.md. context: scaffold_script: scaffold.sh execution: - prompt: "Use the interview skill from the planning plugin, topic latest-settings. Problem: consumers of this repository must always run the latest settings, and must never pick up a broken commit. Options I am weighing: (a) a long-lived `stable` branch consumers install from, a workflow that fast-forwards it to `main` after each merge, a ruleset blocking direct pushes to `stable`, and a bot app the workflow authenticates as to push past that ruleset; (b) something else you suggest. Settle mechanism decisions in this interview. I'm here and will answer; ask me your first round." - allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit] - max_turns: 20 + prompt: "Use the interview skill from the planning plugin, topic latest-settings. Problem: consumers of this repository must always run the latest settings, and must never pick up a broken commit. Options I am weighing: (a) a long-lived `stable` branch consumers install from, a workflow that fast-forwards it to `main` after each merge, a ruleset blocking direct pushes to `stable`, and a bot app the workflow authenticates as to push past that ruleset; (b) something else you suggest. Settle mechanism decisions in this interview. Round 1 is answered: Q1, the problem is as stated above; Q2, the users are the maintainers of the consumer repositories that install these settings. I'm here and will answer; ask me round 2." + allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] + max_turns: 30 timeout_seconds: 900 graders: - name: skill-fired @@ -20,7 +20,12 @@ graders: type: llm arm: both focus: trace - criteria: "PASS only if all hold: (1) a recommendation in the reply is to have consumers keep installing from main (or otherwise to drop the stable branch, its fast-forward workflow, its ruleset and the bot app); (2) its basis names an existing control that already covers the same risk: the protection on main (blocked direct pushes, required review, required consumer-e2e check) or the pull request CI job that installs the settings into a sample consumer; (3) no recommendation in the reply is to build any of the four proposed mechanisms; (4) where it relies on main's protection, it says that protection was read only from a document (docs/branch-protection.md) or labels it documented or unverified, never verified from an enforcing source. FAIL if the reply recommends building any of them, if it names no existing control, if it presents main's document-only protection as verified, or if it asks only about details of the proposed mechanisms (branch name, workflow trigger, app permissions)." + criteria: "PASS only if all hold: (1) a recommendation in the reply is to have consumers keep installing from main (or otherwise to drop the stable branch, its fast-forward workflow, its ruleset and the bot app); (2) its printed reason names an existing control that already covers the same risk: the protection on main (blocked direct pushes, required review, required consumer-e2e check) or the pull request CI job that installs the settings into a sample consumer; (3) no recommendation in the reply is to build any of the four proposed mechanisms; (4) where it relies on main's protection, it says that protection was read only from a document (docs/branch-protection.md) or labels it documented or unverified, never verified from an enforcing source. FAIL if the reply recommends building any of them, if it names no existing control, if it presents main's document-only protection as verified, or if it asks only about details of the proposed mechanisms (branch name, workflow trigger, app permissions)." + - name: scope-check-line + type: regex + pattern: '(?:^|\n)[^\n]{0,6}Scope check:' + target: last_message + arm: both - name: no-plan-write type: tool_used tool: Write diff --git a/plugins/planning/evals/interview-option-existing-control-drops/samples/scope-check-line.json b/plugins/planning/evals/interview-option-existing-control-drops/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-option-existing-control-drops/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-option-known-repetition-keeps/case.yaml b/plugins/planning/evals/interview-option-known-repetition-keeps/case.yaml index 9123892c4e1..83f18aed268 100644 --- a/plugins/planning/evals/interview-option-known-repetition-keeps/case.yaml +++ b/plugins/planning/evals/interview-option-known-repetition-keeps/case.yaml @@ -1,14 +1,14 @@ schema_version: "1.1" name: interview-option-known-repetition-keeps description: An interview over an option to add an interface whose second consumer is a stated requirement with the same send shape recommends keeping the interface rather than calling it speculative -tags: [interview, recommendation, known-repetition, scaffold] -expected_outcome: The first round's recommendation on how to structure the senders keeps the Notifier interface (or an equivalent shared abstraction), citing the stated SMS requirement R2 in docs/requirements.md as the second consumer and the shared send(recipient, message) shape. It does not call the interface speculative generality or YAGNI and does not ask to wait for a third channel. No Write or Edit touches PLAN.md. +tags: [interview, recommendation, known-repetition, scaffold, no-network] +expected_outcome: The first decision round, headed by a Scope check line, gives a recommendation on how to structure the senders keeps the Notifier interface (or an equivalent shared abstraction), citing the stated SMS requirement R2 in docs/requirements.md as the second consumer and the shared send(recipient, message) shape. It does not call the interface speculative generality or YAGNI and does not ask to wait for a third channel. No Write or Edit touches PLAN.md. context: scaffold_script: scaffold.sh execution: - prompt: "Use the interview skill from the planning plugin, topic order-confirmations. Problem: requirement R1 in docs/requirements.md, the email order confirmation, is next. Options I am weighing: (a) add a `Notifier` interface with one method, `send(recipient, message)`, now, with the email sender as its first implementation; (b) write the email sender directly and extract an interface later. Settle mechanism decisions in this interview. I'm here and will answer; ask me your first round." - allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit] - max_turns: 20 + prompt: "Use the interview skill from the planning plugin, topic order-confirmations. Problem: requirement R1 in docs/requirements.md, the email order confirmation, is next. Options I am weighing: (a) add a `Notifier` interface with one method, `send(recipient, message)`, now, with the email sender as its first implementation; (b) write the email sender directly and extract an interface later. Settle mechanism decisions in this interview. Round 1 is answered: Q1, the problem is as stated above; Q2, the users are shoppers who place orders and need a confirmation. I'm here and will answer; ask me round 2." + allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] + max_turns: 30 timeout_seconds: 900 graders: - name: skill-fired @@ -20,7 +20,12 @@ graders: type: llm arm: both focus: trace - criteria: "PASS only if all hold: (1) where the reply recommends how to structure the senders, the recommendation keeps the Notifier interface or an equivalent shared abstraction; (2) its basis cites the stated SMS requirement (R2 in docs/requirements.md) as a second consumer that already exists; (3) no recommendation drops the interface as speculative generality, YAGNI or premature abstraction, and none asks to wait for a third channel. FAIL if the reply recommends dropping or deferring the interface, or keeps it without citing the stated second consumer." + criteria: "PASS only if all hold: (1) where the reply recommends how to structure the senders, the recommendation keeps the Notifier interface or an equivalent shared abstraction; (2) its printed reason cites the stated SMS requirement (R2 in docs/requirements.md) as a second consumer that already exists; (3) no recommendation drops the interface as speculative generality, YAGNI or premature abstraction, and none asks to wait for a third channel. FAIL if the reply recommends dropping or deferring the interface, or keeps it without citing the stated second consumer." + - name: scope-check-line + type: regex + pattern: '(?:^|\n)[^\n]{0,6}Scope check:' + target: last_message + arm: both - name: no-plan-write type: tool_used tool: Write diff --git a/plugins/planning/evals/interview-option-known-repetition-keeps/samples/scope-check-line.json b/plugins/planning/evals/interview-option-known-repetition-keeps/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-option-known-repetition-keeps/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-owner-deferred-document/graders/document-beside-next-step.md b/plugins/planning/evals/interview-owner-deferred-document/graders/document-beside-next-step.md index f6e28c062f6..c78b9006954 100644 --- a/plugins/planning/evals/interview-owner-deferred-document/graders/document-beside-next-step.md +++ b/plugins/planning/evals/interview-owner-deferred-document/graders/document-beside-next-step.md @@ -9,7 +9,9 @@ PASS only if all of these hold: partner's ops lead to answer, naming `/planning:interview document`. 2. The reply also recommends a next step for the work itself (planning, design, exploration, research, or implementation), so the document runs beside the pipeline, not in place of it. -3. The reply does not choose an SSH key type or record one as an assumption. +3. The reply does not choose an SSH key type or record one as an assumption. Naming example key + types (such as the options the document will ask about) without choosing or recommending one + is not choosing. FAIL if the reply leaves the deferred question with no one asked to answer it, recommends only the document with no next step for the work, or picks a key type. diff --git a/plugins/planning/evals/interview-owner-deferred-document/prompt.md b/plugins/planning/evals/interview-owner-deferred-document/prompt.md index cc790f639ef..b3ac1b76cc6 100644 --- a/plugins/planning/evals/interview-owner-deferred-document/prompt.md +++ b/plugins/planning/evals/interview-owner-deferred-document/prompt.md @@ -1,6 +1,6 @@ --- description: "Contract case for the document hand-off on a `tenant` item whose owner is not the user (the partner endpoint's accepted SSH key type). Why hard: the Brief is locked and the natural handoff is /planning:plan alone, so the one question only the partner's ops lead can answer stays parked in the Brief with nobody asked to answer it." -tags: [interview, handoff, document] +tags: [interview, handoff, document, no-network] runs: 3 max_turns: 12 timeout_seconds: 600 diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/covers-d.md b/plugins/planning/evals/interview-pacing-round-arm/graders/covers-d.md deleted file mode 100644 index f76266389ee..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/covers-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-a.md b/plugins/planning/evals/interview-pacing-round-arm/graders/quality-a.md deleted file mode 100644 index 598ed376d33..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-b.md b/plugins/planning/evals/interview-pacing-round-arm/graders/quality-b.md deleted file mode 100644 index fc9c858a09c..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-c.md b/plugins/planning/evals/interview-pacing-round-arm/graders/quality-c.md deleted file mode 100644 index 92b82adfb7d..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-d.md b/plugins/planning/evals/interview-pacing-round-arm/graders/quality-d.md deleted file mode 100644 index 0ad035250ac..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/graders/quality-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-round-arm/prompt.md b/plugins/planning/evals/interview-pacing-round-arm/prompt.md deleted file mode 100644 index 42c6c0436f4..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/prompt.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -description: "Round arm of the round-versus-single pacing comparison (paired with the five interview-pacing-single-arm cases, same task and facts). Source: the consumer's own writing, synthetic. Why hard: the four open decisions for an invoice CSV export (date range, amount format across currencies, who may export, large exports) are not listed as a frontier; the skill must find each one in the stated facts and ask all of them in one round, each in the full layout. The easy failures are to assume one silently, or to drop the basis or alternatives as the round grows. Graders are regexes, one coverage and one quality slot per planted decision; the deciding rule is in ../interview-pacing.md." -tags: [interview, pacing, pacing-comparison, hard, no-network] -runs: 6 -max_turns: 30 -timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "Round 2 asks, in one message, a question on each of date range, amount format and currencies, who may export, and large exports, every question in the full layout (context citing a stated fact, recommendation with Basis or a Withheld line, alternatives), ending with the closing probe. A separate question deferring the large-export mechanism to planning is allowed and does not replace the large-export question" ---- - -/planning:interview me Add a CSV export of invoices to the billing admin page. - -Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. - -What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. - -Ask round 2 now, the whole round in one message, then stop for my answers. diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-a.json b/plugins/planning/evals/interview-pacing-round-arm/samples/quality-a.json deleted file mode 100644 index 37f9b860713..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-a.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one full-layout question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Exports need a scope.\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-b.json b/plugins/planning/evals/interview-pacing-round-arm/samples/quality-b.json deleted file mode 100644 index 5ee8637317c..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-b.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts need a format.\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-c.json b/plugins/planning/evals/interview-pacing-round-arm/samples/quality-c.json deleted file mode 100644 index 955c4a7135b..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Someone has to be allowed to export.\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-d.json b/plugins/planning/evals/interview-pacing-round-arm/samples/quality-d.json deleted file mode 100644 index 0290138ab91..00000000000 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/quality-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- Building the file takes a while for big months.\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "only the deferral question, D itself not asked" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/closing-probe.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/closing-probe.md deleted file mode 100644 index 1fd5f0c18bf..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/closing-probe.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: 'something else that would change' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-a.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-a.md deleted file mode 100644 index 765e72238ec..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-b.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-b.md deleted file mode 100644 index 7925aa6602e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-c.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-c.md deleted file mode 100644 index c5213709dcb..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-d.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-d.md deleted file mode 100644 index f76266389ee..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/covers-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-question.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-question.md deleted file mode 100644 index fef3f505ea7..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-question.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-recommendation.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-recommendation.md deleted file mode 100644 index 638d3b48888..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/one-recommendation.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'My recommendation:|Withheld:' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-a.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-a.md deleted file mode 100644 index 598ed376d33..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-b.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-b.md deleted file mode 100644 index fc9c858a09c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-c.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-c.md deleted file mode 100644 index 92b82adfb7d..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-d.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-d.md deleted file mode 100644 index 0ad035250ac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/graders/quality-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/prompt.md b/plugins/planning/evals/interview-pacing-single-arm-open-b/prompt.md deleted file mode 100644 index d80ecdfda4f..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/prompt.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -description: "Single arm of the round-versus-single pacing comparison with only the amount-and-currency decision (B) left open among the four planted ones (same task and facts as interview-pacing-round-arm). Source: the consumer's own writing, synthetic. The earlier single-pacing answers are stated in the prompt, because a case is one user turn and a resumed history carries no skill body. Why hard: the skill must ask the open decision in the full layout, one question only, without re-asking what is answered or deferred to planning. Graders are regexes; the deciding rule is in ../interview-pacing.md (Amendment 2)." -tags: [interview, pacing, pacing-comparison, hard, no-network] -runs: 6 -max_turns: 30 -timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply asks exactly one question, preferably on how amounts and the three currencies appear in the file, in the full layout with the closing probe, does not re-ask the date range, the month's date, who may export, the large-export approach or its mechanism, then stops" ---- - -/planning:interview me Add a CSV export of invoices to the billing admin page. - -Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. - -What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. - -We're doing round 2 one question at a time, and four are answered. Q4, how a large export completes: build the file in the background and offer it for download when it's ready, as you recommended; the job runner, file storage and ready notice wait for planning. Q5, what one export covers: one calendar month across all customer accounts, as you recommended. Q6, which date puts an invoice in a month: its issue date, with month boundaries in UTC. Q7, who may export: the `finance` role only. - -Ask me the next question, still one at a time, then stop for my answer. diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/closing-probe.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/closing-probe.json deleted file mode 100644 index e04406c4c79..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/closing-probe.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nReply when ready.", - "why": "no closing probe" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-a.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-a.json deleted file mode 100644 index 9a48f709d64..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-a.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing, one question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "date range appears only in a recap line, not as a question" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Which date puts an invoice into a calendar month's export?**\nContext:\n- Finance closes its books on the calendar month, and an invoice has issue, due and paid dates (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "which date places an invoice in a month is a sub-decision, not the date range one export covers" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-b.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-b.json deleted file mode 100644 index 368a0108dac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-b.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "amount and currency format appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-c.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-c.json deleted file mode 100644 index d316c5e9396..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing after A and D were answered, one question on who may export; the recap line asks nothing" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "who may export, phrased as who may start and download the file" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "who may export appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-d.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-d.json deleted file mode 100644 index cdb7c3a2087..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/covers-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be prepared after the click and downloaded once it's ready, instead of being built during the click like the page's other downloads?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D phrased as preparing the file after the click and downloading it once it's ready" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D appears only inside the question that defers the mechanism to planning; D itself is never asked" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nSettled by evidence: a large export runs as a background job, because one month is about 40,000 rows and in-request downloads time out after 30 seconds (your notes).\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D settled in a preface line with live alternatives left, and never asked" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "a question on who may export that mentions the finished file is not D" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-question.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-question.json deleted file mode 100644 index 1449335f937..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-question.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the Later line names the rest without asking them" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the recap line is not a question" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "a whole round" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-recommendation.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-recommendation.json deleted file mode 100644 index 822b3d3d89e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/one-recommendation.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "pass": [ - {"answer": "Q4: **Which role may export?**\nContext: support cannot see payment details (user, round 2).\nMy recommendation: `finance` only.\nBasis: verified, user, round 2\nAlternatives:\n- (a) Both roles\n\nIs there something else that would change this recommendation?", "why": "one recommendation"}, - {"answer": "Q4: Which date range does one export cover?\nWithheld: whether finance ever reconciles part-months, which only finance can confirm.\n\nIs there something else that would change this?", "why": "one withheld question"} - ], - "fail": [ - {"answer": "", "why": "empty"}, - {"answer": "Q4: Which role? My recommendation: finance.\n\nQ5: Which range? My recommendation: one month.", "why": "two questions"} - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-a.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-a.json deleted file mode 100644 index 37f9b860713..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-a.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one full-layout question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Exports need a scope.\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-b.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-b.json deleted file mode 100644 index 5ee8637317c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-b.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts need a format.\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-c.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-c.json deleted file mode 100644 index 955c4a7135b..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Someone has to be allowed to export.\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-d.json b/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-d.json deleted file mode 100644 index 0290138ab91..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-b/samples/quality-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- Building the file takes a while for big months.\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "only the deferral question, D itself not asked" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/closing-probe.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/closing-probe.md deleted file mode 100644 index 1fd5f0c18bf..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/closing-probe.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: 'something else that would change' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-a.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-a.md deleted file mode 100644 index 765e72238ec..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-b.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-b.md deleted file mode 100644 index 7925aa6602e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-c.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-c.md deleted file mode 100644 index c5213709dcb..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-d.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-d.md deleted file mode 100644 index f76266389ee..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/covers-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-question.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-question.md deleted file mode 100644 index fef3f505ea7..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-question.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-recommendation.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-recommendation.md deleted file mode 100644 index 638d3b48888..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/one-recommendation.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'My recommendation:|Withheld:' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-a.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-a.md deleted file mode 100644 index 598ed376d33..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-b.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-b.md deleted file mode 100644 index fc9c858a09c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-c.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-c.md deleted file mode 100644 index 92b82adfb7d..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-d.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-d.md deleted file mode 100644 index 0ad035250ac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/graders/quality-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/prompt.md b/plugins/planning/evals/interview-pacing-single-arm-open-d/prompt.md deleted file mode 100644 index 7781441eb9f..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/prompt.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -description: "Single arm of the round-versus-single pacing comparison with only the large-export decision (D) left open among the four planted ones (same task and facts as interview-pacing-round-arm). Source: the consumer's own writing, synthetic. The earlier single-pacing answers are stated in the prompt, because a case is one user turn and a resumed history carries no skill body. Why hard: the skill must ask how a 40,000-row export completes in the full layout, one question only, and not replace it with a question that only defers the mechanism to planning. Graders are regexes; the deciding rule is in ../interview-pacing.md (Amendment 2)." -tags: [interview, pacing, pacing-comparison, hard, no-network] -runs: 6 -max_turns: 30 -timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply asks exactly one question, preferably on how an export of about 40,000 rows completes when in-request downloads time out after 30 seconds, in the full layout with the closing probe, does not re-ask the date range, the month's date, the amount format or who may export, then stops" ---- - -/planning:interview me Add a CSV export of invoices to the billing admin page. - -Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. - -What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. - -We're doing round 2 one question at a time, and four are answered. Q4, what one export covers: one calendar month across all customer accounts, as you recommended. Q5, which date puts an invoice in a month: its issue date, with month boundaries in UTC. Q6, amounts: two-decimal numbers in the main currency unit with a separate currency-code column, no conversion and no total across currencies. Q7, who may export: the `finance` role only. - -Ask me the next question, still one at a time, then stop for my answer. diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/closing-probe.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/closing-probe.json deleted file mode 100644 index e04406c4c79..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/closing-probe.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nReply when ready.", - "why": "no closing probe" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-a.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-a.json deleted file mode 100644 index 9a48f709d64..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-a.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing, one question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "date range appears only in a recap line, not as a question" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Which date puts an invoice into a calendar month's export?**\nContext:\n- Finance closes its books on the calendar month, and an invoice has issue, due and paid dates (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "which date places an invoice in a month is a sub-decision, not the date range one export covers" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-b.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-b.json deleted file mode 100644 index 368a0108dac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-b.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "amount and currency format appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-c.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-c.json deleted file mode 100644 index d316c5e9396..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing after A and D were answered, one question on who may export; the recap line asks nothing" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "who may export, phrased as who may start and download the file" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "who may export appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-d.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-d.json deleted file mode 100644 index cdb7c3a2087..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/covers-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be prepared after the click and downloaded once it's ready, instead of being built during the click like the page's other downloads?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D phrased as preparing the file after the click and downloading it once it's ready" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D appears only inside the question that defers the mechanism to planning; D itself is never asked" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nSettled by evidence: a large export runs as a background job, because one month is about 40,000 rows and in-request downloads time out after 30 seconds (your notes).\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D settled in a preface line with live alternatives left, and never asked" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "a question on who may export that mentions the finished file is not D" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-question.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-question.json deleted file mode 100644 index 1449335f937..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-question.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the Later line names the rest without asking them" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the recap line is not a question" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "a whole round" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-recommendation.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-recommendation.json deleted file mode 100644 index 822b3d3d89e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/one-recommendation.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "pass": [ - {"answer": "Q4: **Which role may export?**\nContext: support cannot see payment details (user, round 2).\nMy recommendation: `finance` only.\nBasis: verified, user, round 2\nAlternatives:\n- (a) Both roles\n\nIs there something else that would change this recommendation?", "why": "one recommendation"}, - {"answer": "Q4: Which date range does one export cover?\nWithheld: whether finance ever reconciles part-months, which only finance can confirm.\n\nIs there something else that would change this?", "why": "one withheld question"} - ], - "fail": [ - {"answer": "", "why": "empty"}, - {"answer": "Q4: Which role? My recommendation: finance.\n\nQ5: Which range? My recommendation: one month.", "why": "two questions"} - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-a.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-a.json deleted file mode 100644 index 37f9b860713..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-a.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one full-layout question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Exports need a scope.\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-b.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-b.json deleted file mode 100644 index 5ee8637317c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-b.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts need a format.\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-c.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-c.json deleted file mode 100644 index 955c4a7135b..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Someone has to be allowed to export.\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-d.json b/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-d.json deleted file mode 100644 index 0290138ab91..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-open-d/samples/quality-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- Building the file takes a while for big months.\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "only the deferral question, D itself not asked" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/closing-probe.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/closing-probe.md deleted file mode 100644 index 1fd5f0c18bf..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/closing-probe.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: 'something else that would change' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-a.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-a.md deleted file mode 100644 index 765e72238ec..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-b.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-b.md deleted file mode 100644 index 7925aa6602e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-c.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-c.md deleted file mode 100644 index c5213709dcb..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-d.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-d.md deleted file mode 100644 index f76266389ee..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/covers-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-question.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-question.md deleted file mode 100644 index fef3f505ea7..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-question.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-recommendation.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-recommendation.md deleted file mode 100644 index 638d3b48888..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/one-recommendation.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'My recommendation:|Withheld:' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-a.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-a.md deleted file mode 100644 index 598ed376d33..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-b.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-b.md deleted file mode 100644 index fc9c858a09c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-c.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-c.md deleted file mode 100644 index 92b82adfb7d..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-d.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-d.md deleted file mode 100644 index 0ad035250ac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/graders/quality-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/prompt.md b/plugins/planning/evals/interview-pacing-single-arm-step-2/prompt.md deleted file mode 100644 index 2bd642d2b51..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/prompt.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -description: "Single arm, third message, of the round-versus-single pacing comparison (same task and facts as interview-pacing-round-arm). Source: the consumer's own writing, synthetic. The two earlier single-pacing answers are stated in the prompt, because a case is one user turn and a resumed history carries no skill body. Why hard: the user deferred the large-export mechanism to planning, so the skill must not re-ask it or treat it as open, and must ask one of the still-open decisions (amount format and currencies, or who may export) in the full layout without previewing the rest. Graders are regexes; the deciding rule is in ../interview-pacing.md." -tags: [interview, pacing, pacing-comparison, hard, no-network] -runs: 6 -max_turns: 30 -timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply asks exactly one question, preferably on amount format and currencies or on who may export, in the full layout with the closing probe, does not re-ask the date range, the large-export approach or its mechanism, then stops" ---- - -/planning:interview me Add a CSV export of invoices to the billing admin page. - -Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. - -What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. - -We're doing round 2 one question at a time, and two are answered. Q4, how a large export completes: build the file in the background and offer it for download when it's ready, as you recommended; the job runner, file storage and ready notice wait for planning. Q5, what one export covers: one calendar month across all customer accounts, as you recommended. - -Ask me the next question, still one at a time, then stop for my answer. diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/closing-probe.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/closing-probe.json deleted file mode 100644 index e04406c4c79..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/closing-probe.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nReply when ready.", - "why": "no closing probe" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-a.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-a.json deleted file mode 100644 index 9a48f709d64..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-a.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing, one question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "date range appears only in a recap line, not as a question" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Which date puts an invoice into a calendar month's export?**\nContext:\n- Finance closes its books on the calendar month, and an invoice has issue, due and paid dates (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "which date places an invoice in a month is a sub-decision, not the date range one export covers" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-b.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-b.json deleted file mode 100644 index 368a0108dac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-b.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "amount and currency format appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-c.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-c.json deleted file mode 100644 index d316c5e9396..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing after A and D were answered, one question on who may export; the recap line asks nothing" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "who may export, phrased as who may start and download the file" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "who may export appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-d.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-d.json deleted file mode 100644 index cdb7c3a2087..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/covers-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be prepared after the click and downloaded once it's ready, instead of being built during the click like the page's other downloads?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D phrased as preparing the file after the click and downloading it once it's ready" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D appears only inside the question that defers the mechanism to planning; D itself is never asked" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nSettled by evidence: a large export runs as a background job, because one month is about 40,000 rows and in-request downloads time out after 30 seconds (your notes).\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D settled in a preface line with live alternatives left, and never asked" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "a question on who may export that mentions the finished file is not D" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-question.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-question.json deleted file mode 100644 index 1449335f937..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-question.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the Later line names the rest without asking them" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the recap line is not a question" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "a whole round" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-recommendation.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-recommendation.json deleted file mode 100644 index 822b3d3d89e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/one-recommendation.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "pass": [ - {"answer": "Q4: **Which role may export?**\nContext: support cannot see payment details (user, round 2).\nMy recommendation: `finance` only.\nBasis: verified, user, round 2\nAlternatives:\n- (a) Both roles\n\nIs there something else that would change this recommendation?", "why": "one recommendation"}, - {"answer": "Q4: Which date range does one export cover?\nWithheld: whether finance ever reconciles part-months, which only finance can confirm.\n\nIs there something else that would change this?", "why": "one withheld question"} - ], - "fail": [ - {"answer": "", "why": "empty"}, - {"answer": "Q4: Which role? My recommendation: finance.\n\nQ5: Which range? My recommendation: one month.", "why": "two questions"} - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-a.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-a.json deleted file mode 100644 index 37f9b860713..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-a.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one full-layout question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Exports need a scope.\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-b.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-b.json deleted file mode 100644 index 5ee8637317c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-b.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts need a format.\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-c.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-c.json deleted file mode 100644 index 955c4a7135b..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Someone has to be allowed to export.\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-d.json b/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-d.json deleted file mode 100644 index 0290138ab91..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-2/samples/quality-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- Building the file takes a while for big months.\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "only the deferral question, D itself not asked" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/closing-probe.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/closing-probe.md deleted file mode 100644 index 1fd5f0c18bf..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/closing-probe.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: 'something else that would change' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-a.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-a.md deleted file mode 100644 index 765e72238ec..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-b.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-b.md deleted file mode 100644 index 7925aa6602e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-c.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-c.md deleted file mode 100644 index c5213709dcb..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-d.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-d.md deleted file mode 100644 index f76266389ee..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/covers-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-question.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-question.md deleted file mode 100644 index fef3f505ea7..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-question.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-recommendation.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-recommendation.md deleted file mode 100644 index 638d3b48888..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/one-recommendation.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'My recommendation:|Withheld:' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-a.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-a.md deleted file mode 100644 index 598ed376d33..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-b.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-b.md deleted file mode 100644 index fc9c858a09c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-c.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-c.md deleted file mode 100644 index 92b82adfb7d..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-d.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-d.md deleted file mode 100644 index 0ad035250ac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/graders/quality-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/prompt.md b/plugins/planning/evals/interview-pacing-single-arm-step-3/prompt.md deleted file mode 100644 index 5479bc6da5b..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/prompt.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -description: "Single arm, fourth message, of the round-versus-single pacing comparison (same task and facts as interview-pacing-round-arm). Source: the consumer's own writing, synthetic. The three earlier single-pacing answers are stated in the prompt, because a case is one user turn and a resumed history carries no skill body. Why hard: three planted decisions are settled, one with its mechanism deferred to planning, so the skill must ask the one still open (who may export) or another ready question, in the full layout, without re-asking a settled or deferred one. Graders are regexes; the deciding rule is in ../interview-pacing.md." -tags: [interview, pacing, pacing-comparison, hard, no-network] -runs: 6 -max_turns: 30 -timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply asks exactly one question, preferably on who may export, in the full layout with the closing probe, does not re-ask the date range, the amount format, the large-export approach or its mechanism, then stops" ---- - -/planning:interview me Add a CSV export of invoices to the billing admin page. - -Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. - -What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. - -We're doing round 2 one question at a time, and three are answered. Q4, how a large export completes: build the file in the background and offer it for download when it's ready, as you recommended; the job runner, file storage and ready notice wait for planning. Q5, what one export covers: one calendar month across all customer accounts, as you recommended. Q6, amounts: two-decimal numbers in the main currency unit with a separate currency-code column, no conversion and no total across currencies. - -Ask me the next question, still one at a time, then stop for my answer. diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/closing-probe.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/closing-probe.json deleted file mode 100644 index e04406c4c79..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/closing-probe.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nReply when ready.", - "why": "no closing probe" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-a.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-a.json deleted file mode 100644 index 9a48f709d64..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-a.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing, one question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "date range appears only in a recap line, not as a question" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Which date puts an invoice into a calendar month's export?**\nContext:\n- Finance closes its books on the calendar month, and an invoice has issue, due and paid dates (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "which date places an invoice in a month is a sub-decision, not the date range one export covers" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-b.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-b.json deleted file mode 100644 index 368a0108dac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-b.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "amount and currency format appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-c.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-c.json deleted file mode 100644 index d316c5e9396..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing after A and D were answered, one question on who may export; the recap line asks nothing" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "who may export, phrased as who may start and download the file" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "who may export appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-d.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-d.json deleted file mode 100644 index cdb7c3a2087..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/covers-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be prepared after the click and downloaded once it's ready, instead of being built during the click like the page's other downloads?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D phrased as preparing the file after the click and downloading it once it's ready" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D appears only inside the question that defers the mechanism to planning; D itself is never asked" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nSettled by evidence: a large export runs as a background job, because one month is about 40,000 rows and in-request downloads time out after 30 seconds (your notes).\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D settled in a preface line with live alternatives left, and never asked" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "a question on who may export that mentions the finished file is not D" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-question.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-question.json deleted file mode 100644 index 1449335f937..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-question.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the Later line names the rest without asking them" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the recap line is not a question" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "a whole round" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-recommendation.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-recommendation.json deleted file mode 100644 index 822b3d3d89e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/one-recommendation.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "pass": [ - {"answer": "Q4: **Which role may export?**\nContext: support cannot see payment details (user, round 2).\nMy recommendation: `finance` only.\nBasis: verified, user, round 2\nAlternatives:\n- (a) Both roles\n\nIs there something else that would change this recommendation?", "why": "one recommendation"}, - {"answer": "Q4: Which date range does one export cover?\nWithheld: whether finance ever reconciles part-months, which only finance can confirm.\n\nIs there something else that would change this?", "why": "one withheld question"} - ], - "fail": [ - {"answer": "", "why": "empty"}, - {"answer": "Q4: Which role? My recommendation: finance.\n\nQ5: Which range? My recommendation: one month.", "why": "two questions"} - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-a.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-a.json deleted file mode 100644 index 37f9b860713..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-a.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one full-layout question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Exports need a scope.\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-b.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-b.json deleted file mode 100644 index 5ee8637317c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-b.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts need a format.\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-c.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-c.json deleted file mode 100644 index 955c4a7135b..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Someone has to be allowed to export.\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-d.json b/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-d.json deleted file mode 100644 index 0290138ab91..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm-step-3/samples/quality-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- Building the file takes a while for big months.\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "only the deferral question, D itself not asked" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/closing-probe.md b/plugins/planning/evals/interview-pacing-single-arm/graders/closing-probe.md deleted file mode 100644 index 1fd5f0c18bf..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/closing-probe.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: 'something else that would change' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-a.md b/plugins/planning/evals/interview-pacing-single-arm/graders/covers-a.md deleted file mode 100644 index 765e72238ec..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-b.md b/plugins/planning/evals/interview-pacing-single-arm/graders/covers-b.md deleted file mode 100644 index 7925aa6602e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-c.md b/plugins/planning/evals/interview-pacing-single-arm/graders/covers-c.md deleted file mode 100644 index c5213709dcb..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-d.md b/plugins/planning/evals/interview-pacing-single-arm/graders/covers-d.md deleted file mode 100644 index f76266389ee..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/covers-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/one-question.md b/plugins/planning/evals/interview-pacing-single-arm/graders/one-question.md deleted file mode 100644 index fef3f505ea7..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/one-question.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/one-recommendation.md b/plugins/planning/evals/interview-pacing-single-arm/graders/one-recommendation.md deleted file mode 100644 index 638d3b48888..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/one-recommendation.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'My recommendation:|Withheld:' -flags: i -match: "count:1" -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-a.md b/plugins/planning/evals/interview-pacing-single-arm/graders/quality-a.md deleted file mode 100644 index 598ed376d33..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-a.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:export(?: file)? (?:cover|contain|hold)|date range|one calendar month|a whole calendar month|chosen month)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-b.md b/plugins/planning/evals/interview-pacing-single-arm/graders/quality-b.md deleted file mode 100644 index fc9c858a09c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-b.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)(?![^\n]*\bcolumns?:)[^\n]*(?:currenc|cents|decimal|amounts? (?:be|appear|written|shown|formatted))[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-c.md b/plugins/planning/evals/interview-pacing-single-arm/graders/quality-c.md deleted file mode 100644 index 92b82adfb7d..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-c.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:\broles?\b|who (?:may|can)|\bsupport\b)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-d.md b/plugins/planning/evals/interview-pacing-single-arm/graders/quality-d.md deleted file mode 100644 index 0ad035250ac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/graders/quality-d.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: regex -pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:it''?s|it is|the file is) ready|time ?out|stream|chunk|40,000|large export)[^\n]*(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nContext:(?:(?!\n[^\n]{0,6}(?:My recommendation:|Withheld:))(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S]))*?(?:\(user|your notes|round 1|calendar month|cents|\bUSD\b|\bEUR\b|\bGBP\b|spreadsheet|\bsupport\b|\bfinance\b|40,000|30 seconds|30-second))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}(?:My recommendation:(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nBasis:|Withheld:))(?=(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\nAlternatives:[^\S\n]*(?:\n[^\S\n]*)?(?:-|\(a\)))' -flags: i -target: last_message -arm: both ---- diff --git a/plugins/planning/evals/interview-pacing-single-arm/prompt.md b/plugins/planning/evals/interview-pacing-single-arm/prompt.md deleted file mode 100644 index 337403c6756..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/prompt.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -description: "Single arm, first message, of the round-versus-single pacing comparison (paired with interview-pacing-round-arm, same task and facts; only the pacing request differs). Source: the consumer's own writing, synthetic. An eval run cannot set the plugin's question_pacing option, so the case uses the in-session request, which the skill says overrides it. Why hard: the four open decisions are not listed, so the skill must find them, then ask only one, in the full layout, without previewing the rest. Graders are regexes; the deciding rule is in ../interview-pacing.md." -tags: [interview, pacing, pacing-comparison, hard, no-network] -runs: 6 -max_turns: 30 -timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply asks exactly one question, on one of date range, amount format and currencies, who may export, or large exports, in the full layout with the closing probe, lists no other question, then stops" ---- - -/planning:interview me Add a CSV export of invoices to the billing admin page. - -Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. - -What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. - -From now on, ask me one question at a time. Go ahead, then stop for my answer. diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/closing-probe.json b/plugins/planning/evals/interview-pacing-single-arm/samples/closing-probe.json deleted file mode 100644 index e04406c4c79..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/closing-probe.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "ends with the closing probe" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nReply when ready.", - "why": "no closing probe" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-a.json b/plugins/planning/evals/interview-pacing-single-arm/samples/covers-a.json deleted file mode 100644 index 9a48f709d64..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-a.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing, one question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "date range appears only in a recap line, not as a question" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Which date puts an invoice into a calendar month's export?**\nContext:\n- Finance closes its books on the calendar month, and an invoice has issue, due and paid dates (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "which date places an invoice in a month is a sub-decision, not the date range one export covers" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-b.json b/plugins/planning/evals/interview-pacing-single-arm/samples/covers-b.json deleted file mode 100644 index 368a0108dac..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-b.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "amount and currency format appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-c.json b/plugins/planning/evals/interview-pacing-single-arm/samples/covers-c.json deleted file mode 100644 index d316c5e9396..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "single pacing after A and D were answered, one question on who may export; the recap line asks nothing" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "who may export, phrased as who may start and download the file" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nDecided earlier:\nQ2: one calendar month across all accounts, amounts as decimals with a currency column, `finance` role only.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "who may export appears only in a recap line, not as a question" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-d.json b/plugins/planning/evals/interview-pacing-single-arm/samples/covers-d.json deleted file mode 100644 index cdb7c3a2087..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/covers-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be prepared after the click and downloaded once it's ready, instead of being built during the click like the page's other downloads?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D phrased as preparing the file after the click and downloading it once it's ready" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D appears only inside the question that defers the mechanism to planning; D itself is never asked" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\nSettled by evidence: a large export runs as a background job, because one month is about 40,000 rows and in-request downloads time out after 30 seconds (your notes).\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "D settled in a preface line with live alternatives left, and never asked" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q6: **Who may start an export and download the finished file?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) The other way round (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "a question on who may export that mentions the finished file is not D" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/one-question.json b/plugins/planning/evals/interview-pacing-single-arm/samples/one-question.json deleted file mode 100644 index 1449335f937..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/one-question.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the Later line names the rest without asking them" - }, - { - "answer": "Decided so far: Q4 (large exports): background build, mechanism at planning. Q5 (scope): one calendar month.\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one question; the recap line is not a question" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "a whole round" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/one-recommendation.json b/plugins/planning/evals/interview-pacing-single-arm/samples/one-recommendation.json deleted file mode 100644 index 822b3d3d89e..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/one-recommendation.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "pass": [ - {"answer": "Q4: **Which role may export?**\nContext: support cannot see payment details (user, round 2).\nMy recommendation: `finance` only.\nBasis: verified, user, round 2\nAlternatives:\n- (a) Both roles\n\nIs there something else that would change this recommendation?", "why": "one recommendation"}, - {"answer": "Q4: Which date range does one export cover?\nWithheld: whether finance ever reconciles part-months, which only finance can confirm.\n\nIs there something else that would change this?", "why": "one withheld question"} - ], - "fail": [ - {"answer": "", "why": "empty"}, - {"answer": "Q4: Which role? My recommendation: finance.\n\nQ5: Which range? My recommendation: one month.", "why": "two questions"} - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-a.json b/plugins/planning/evals/interview-pacing-single-arm/samples/quality-a.json deleted file mode 100644 index 37f9b860713..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-a.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - }, - { - "answer": "**Round 2** (one question per message, as you asked)\nLater: amount format, which roles can export, how a large export completes.\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\nIs there something else that would change these recommendations?", - "why": "one full-layout question on the date range" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on date range" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Exports need a scope.\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-b.json b/plugins/planning/evals/interview-pacing-single-arm/samples/quality-b.json deleted file mode 100644 index 5ee8637317c..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-b.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on amount and currency format" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts need a format.\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-c.json b/plugins/planning/evals/interview-pacing-single-arm/samples/quality-c.json deleted file mode 100644 index 955c4a7135b..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-c.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on who may export" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Someone has to be allowed to export.\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-d.json b/plugins/planning/evals/interview-pacing-single-arm/samples/quality-d.json deleted file mode 100644 index 0290138ab91..00000000000 --- a/plugins/planning/evals/interview-pacing-single-arm/samples/quality-d.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "pass": [ - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "every planted decision asked in full layout, D's mechanism question deferred to planning beside D's own question (the shape the old judge failed in runs 1, 2 and 4)" - } - ], - "fail": [ - { - "answer": "", - "why": "empty" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "no question on how a large export completes" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question carries a recommendation but no Basis line" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question lists no alternatives" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be built in the background and offered for download when it is ready, instead of inside the web request?**\nContext:\n- Building the file takes a while for big months.\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "the question's context cites no fact the user stated" - }, - { - "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q5: **Should how the background build works (job runner, file storage, ready notice) wait for planning?**\nContext:\n- The export will run in the background, and your notes name no job runner (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", - "why": "only the deferral question, D itself not asked" - } - ] -} diff --git a/plugins/planning/evals/interview-pacing.md b/plugins/planning/evals/interview-pacing.md index 560abbe86e2..2c0aa7e2237 100644 --- a/plugins/planning/evals/interview-pacing.md +++ b/plugins/planning/evals/interview-pacing.md @@ -199,3 +199,13 @@ pacings on this task; what differs is turns, about 12 single-pacing messages for Grader miss, reported and not re-graded (graders are frozen once a run starts): round run 3 asks D as "Can the export be asynchronous instead of an immediate download?", which `covers-d` does not match, so it counts as no D. Read by hand, Cr is 6/6 and the verdict is the same. + +## Amendment 3 (2026-11-21, cases retired) + +The six `interview-pacing-*` cases are removed from the suite. The experiment concluded with run 3 +(ROUNDS HOLD UP) and measured a round layout that has since changed: rounds now print at a +`question_detail` level, `standard` by default, which shows no `Basis:` line. This file freezes the +graders once a run starts, so the cases are retired, not edited. Question quality at the new layout +is checked by `interview-round-quality-standard`, which reuses the round-arm prompt and is not part +of this experiment. To rerun the experiment, restore the six case folders from the commit before +their removal. diff --git a/plugins/planning/evals/interview-person-deferred-document/prompt.md b/plugins/planning/evals/interview-person-deferred-document/prompt.md index acd6d42b734..26ebd0cecc3 100644 --- a/plugins/planning/evals/interview-person-deferred-document/prompt.md +++ b/plugins/planning/evals/interview-person-deferred-document/prompt.md @@ -1,6 +1,6 @@ --- description: "Contract case for the document hand-off on an assumption-sweep `person` item the user cannot answer (the retention window the team's DBA sets). Why hard: the Brief is locked and the natural handoff is /planning:plan alone, so the one question only the DBA can answer stays parked in the Brief with nobody asked to answer it, or the run fills in a window on her behalf." -tags: [interview, handoff, document, assumption-sweep] +tags: [interview, handoff, document, assumption-sweep, no-network] runs: 3 max_turns: 12 timeout_seconds: 600 diff --git a/plugins/planning/evals/interview-prd-seeds-brief/prompt.md b/plugins/planning/evals/interview-prd-seeds-brief/prompt.md index 9812daf9371..b45472496f6 100644 --- a/plugins/planning/evals/interview-prd-seeds-brief/prompt.md +++ b/plugins/planning/evals/interview-prd-seeds-brief/prompt.md @@ -1,6 +1,6 @@ --- description: "Unattended lock with a locked PRD.md in the topic's memory slice: the Brief's ### Problem carries the PRD's Problem (pickers walking back to the desk for a pick-list reprint), and the PRD's one assumption (handheld scanners have a browser) sits under ### Captured assumptions tagged (from PRD), never as a constraint-ledger row, and the Goal's Vision line carries the PRD's Vision (every floor task on the device in the picker's hand). Needs --allow-tools Write." -tags: [interview, brief, prd, assumptions, problem, scaffold] +tags: [interview, brief, prd, assumptions, problem, scaffold, no-network] runs: 3 max_turns: 30 timeout_seconds: 900 diff --git a/plugins/planning/evals/interview-prescribed-solution-rethink/case.yaml b/plugins/planning/evals/interview-prescribed-solution-rethink/case.yaml index ae55067c749..68dcab9784c 100644 --- a/plugins/planning/evals/interview-prescribed-solution-rethink/case.yaml +++ b/plugins/planning/evals/interview-prescribed-solution-rethink/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-prescribed-solution-rethink description: A request that prescribes a solution and states no problem gets its problem worked back out with an evidence rating and one rethink-or-accept question, and no Brief is written -tags: [interview, problem-framing, prescribed-solution, scaffold] +tags: [interview, problem-framing, prescribed-solution, scaffold, no-network] expected_outcome: The reply states the problem the retry wrapper would fix (for example, webhook deliveries lost on transient endpoint failures), rates its evidence observed, reported or inferred, and asks whether to rethink the approach against that problem or accept the retry wrapper; no Write or Edit touches PLAN.md. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-prototype-split-by-question/prompt.md b/plugins/planning/evals/interview-prototype-split-by-question/prompt.md index d4467e3d474..ed6ba8da632 100644 --- a/plugins/planning/evals/interview-prototype-split-by-question/prompt.md +++ b/plugins/planning/evals/interview-prototype-split-by-question/prompt.md @@ -1,9 +1,9 @@ --- description: "The interview's outcome is a throwaway prototype that has to settle three unrelated doubts. Why hard: the easy path scopes one prototype that tries to answer everything, or asks only scoping questions, never asking what the prototype must prove or splitting it so the pieces can be built at once." -tags: [interview, prototype, quality-bar] +tags: [interview, prototype, quality-bar, no-network] runs: 3 -max_turns: 12 -timeout_seconds: 600 +max_turns: 30 +timeout_seconds: 900 allowed_tools: [Read, Glob, Grep, Skill] expected_outcome: "The next round asks what the prototype must prove, with a recommendation listing the three doubts as separate questions, and recommends one small prototype per question that can be built at the same time, each in its own worktree" --- diff --git a/plugins/planning/evals/interview-quality-bar-asked-bookmark-sync/prompt.md b/plugins/planning/evals/interview-quality-bar-asked-bookmark-sync/prompt.md index 2656ed210bf..d1c263a08e3 100644 --- a/plugins/planning/evals/interview-quality-bar-asked-bookmark-sync/prompt.md +++ b/plugins/planning/evals/interview-quality-bar-asked-bookmark-sync/prompt.md @@ -2,8 +2,8 @@ description: "Hard case. Why hard: a vague build request with no stated quality bar; the interview's own question budget scales with settled artifacts and reversibility, not with an asked bar, so a round 1 without the bar question is the current default. Row 38 of the Pocock sync." tags: [interview, quality-bar, row38, no-network] runs: 3 -max_turns: 12 -timeout_seconds: 600 +max_turns: 30 +timeout_seconds: 900 allowed_tools: [Read, Glob, Grep, Skill] expected_outcome: "Round 1 asks which quality bar the work is for (a prototype or throwaway, an internal tool, or production) before deeper scoping questions" --- diff --git a/plugins/planning/evals/interview-quality-bar-asked-ticket-dashboard/prompt.md b/plugins/planning/evals/interview-quality-bar-asked-ticket-dashboard/prompt.md index d0fffff096b..9b64a9d4531 100644 --- a/plugins/planning/evals/interview-quality-bar-asked-ticket-dashboard/prompt.md +++ b/plugins/planning/evals/interview-quality-bar-asked-ticket-dashboard/prompt.md @@ -1,9 +1,9 @@ --- description: "Hard case. Why hard: a second vague build request, in a different domain, with no stated quality bar; the round can fill with scope and data-source questions and never ask how robust the result must be. Row 38 of the Pocock sync." -tags: [interview, quality-bar, row38] +tags: [interview, quality-bar, row38, no-network] runs: 3 -max_turns: 12 -timeout_seconds: 600 +max_turns: 30 +timeout_seconds: 900 allowed_tools: [Read, Glob, Grep, Skill] expected_outcome: "Round 1 asks which quality bar the work is for (a prototype or throwaway, an internal tool, or production) before deeper scoping questions" --- diff --git a/plugins/planning/evals/interview-quality-bar-stated-production/graders/asks-a-round.md b/plugins/planning/evals/interview-quality-bar-stated-production/graders/asks-a-round.md index 928a6464a45..7581913ab40 100644 --- a/plugins/planning/evals/interview-quality-bar-stated-production/graders/asks-a-round.md +++ b/plugins/planning/evals/interview-quality-bar-stated-production/graders/asks-a-round.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '(?:^|\n)\s*(?:#{1,6}\s*)?(?:\*\*)?(?:Q1\b|1[.)])[^?]{0,300}\?' +pattern: '(?:^|\n)\s*(?:#{1,6}\s*)?(?:❓\s*)?(?:\*\*)?(?:Q1\b|1[.)])[^?]{0,300}\?' flags: i arm: both --- diff --git a/plugins/planning/evals/interview-quality-bar-stated-production/prompt.md b/plugins/planning/evals/interview-quality-bar-stated-production/prompt.md index c35555850b7..7c086810e16 100644 --- a/plugins/planning/evals/interview-quality-bar-stated-production/prompt.md +++ b/plugins/planning/evals/interview-quality-bar-stated-production/prompt.md @@ -1,9 +1,9 @@ --- description: "Control. The prompt states the bar (production, relied on by paying customers), so round 1 must not ask it again and its depth should match production. Row 38 of the Pocock sync." -tags: [interview, quality-bar, row38] +tags: [interview, quality-bar, row38, no-network] runs: 3 -max_turns: 12 -timeout_seconds: 600 +max_turns: 30 +timeout_seconds: 900 allowed_tools: [Read, Glob, Grep, Skill] expected_outcome: "Round 1 asks scoping questions without asking the quality bar again, and includes at least one production concern (failure handling, data correctness, security, monitoring, or rollback)" --- diff --git a/plugins/planning/evals/interview-quality-bar-stated-production/samples/asks-a-round.json b/plugins/planning/evals/interview-quality-bar-stated-production/samples/asks-a-round.json index 992de87ae01..6af0b751edf 100644 --- a/plugins/planning/evals/interview-quality-bar-stated-production/samples/asks-a-round.json +++ b/plugins/planning/evals/interview-quality-bar-stated-production/samples/asks-a-round.json @@ -11,6 +11,10 @@ { "answer": "### Q1\nWhich accounting systems must the first release support?", "why": "heading form" + }, + { + "answer": "**Round 1.** A nightly invoice export for paying customers.\nScope check: 3 of 27 questions registered\n\n❓ Q1: **What goes wrong today when month-end close needs these invoices?**\n\nContext:\n- Customers rely on it for their month-end close (user, round 1).\n\nDiscovery: the problem this export solves.", + "why": "the emoji question marker the default layout prints before Q1" } ], "fail": [ diff --git a/plugins/planning/evals/interview-quality-bar-stated-prototype/graders/asks-a-round.md b/plugins/planning/evals/interview-quality-bar-stated-prototype/graders/asks-a-round.md index a6f831320c7..82d36ce8442 100644 --- a/plugins/planning/evals/interview-quality-bar-stated-prototype/graders/asks-a-round.md +++ b/plugins/planning/evals/interview-quality-bar-stated-prototype/graders/asks-a-round.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '(?:^|\n)[ \t]*(?:#{1,6}[ \t]*)?(?:\*\*)?(?:Q1\b|1[.)])[^?]{0,400}\?' +pattern: '(?:^|\n)[ \t]*(?:#{1,6}[ \t]*)?(?:❓\s*)?(?:\*\*)?(?:Q1\b|1[.)])[^?]{0,400}\?' flags: i arm: both --- diff --git a/plugins/planning/evals/interview-quality-bar-stated-prototype/prompt.md b/plugins/planning/evals/interview-quality-bar-stated-prototype/prompt.md index 8abb8e17f3b..59f2fa70422 100644 --- a/plugins/planning/evals/interview-quality-bar-stated-prototype/prompt.md +++ b/plugins/planning/evals/interview-quality-bar-stated-prototype/prompt.md @@ -1,9 +1,9 @@ --- description: "Control. The prompt states the bar (a throwaway prototype), so round 1 must not ask it again and its depth should match a prototype. Row 38 of the Pocock sync." -tags: [interview, quality-bar, row38] +tags: [interview, quality-bar, row38, no-network] runs: 3 -max_turns: 12 -timeout_seconds: 600 +max_turns: 30 +timeout_seconds: 900 allowed_tools: [Read, Glob, Grep, Skill] expected_outcome: "Round 1 asks scoping questions, does not ask the quality bar again, and keeps production concerns (auth, scaling, monitoring, CI) out of the round or explicitly deferred" --- diff --git a/plugins/planning/evals/interview-quality-bar-stated-prototype/samples/asks-a-round.json b/plugins/planning/evals/interview-quality-bar-stated-prototype/samples/asks-a-round.json index 859da6eaa76..8653b09e96d 100644 --- a/plugins/planning/evals/interview-quality-bar-stated-prototype/samples/asks-a-round.json +++ b/plugins/planning/evals/interview-quality-bar-stated-prototype/samples/asks-a-round.json @@ -19,6 +19,10 @@ { "answer": "**Q1. Storage**\nWhere should the demo keep its data between reloads?", "why": "question on the line after the Q1 label" + }, + { + "answer": "**Round 1.** A throwaway habit tracker for a weekend demo.\nScope check: 2 of 27 questions registered\n\n❓ Q1: **What should your friend take away from the demo?**\n\nContext:\n- The demo is this weekend and nobody else will use it (user, round 1).\n\nDiscovery: what the prototype must show.", + "why": "the emoji question marker the default layout prints before Q1" } ], "fail": [ diff --git a/plugins/planning/evals/interview-readback-dictated-answer/prompt.md b/plugins/planning/evals/interview-readback-dictated-answer/prompt.md index 4d9d7d877e8..fa1ff69f78c 100644 --- a/plugins/planning/evals/interview-readback-dictated-answer/prompt.md +++ b/plugins/planning/evals/interview-readback-dictated-answer/prompt.md @@ -1,6 +1,6 @@ --- description: "Eval-first case for the per-answer read-back. Why hard: the dictated reply reads 'fee chair flag' for 'feature flag' and departs from two recommendations, so recording the answers as typed, or echoing them verbatim, both look like progress." -tags: [interview, read-back, dictation] +tags: [interview, read-back, dictation, no-network] runs: 3 max_turns: 12 timeout_seconds: 600 diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/checkable-basis-and-flip.md b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/checkable-basis-and-flip.md index f2ffc632b85..77c74e83ef3 100644 --- a/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/checkable-basis-and-flip.md +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/checkable-basis-and-flip.md @@ -5,7 +5,8 @@ arm: both Look at every question in the reply that carries a recommended answer. -PASS only if every such recommendation has: +PASS only if every such recommendation has the following, stated in its printed reason or in a +`Basis:` line under it: 1. A checkable basis: a file path with a line, a URL, tool output or a check the reply says it ran this session (a command, a search, a survey of the working directory) named with what it showed diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/scope-check-line.md b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/prompt.md b/plugins/planning/evals/interview-recommendation-basis-and-flip/prompt.md index 225b0eb9d73..ccc36745dde 100644 --- a/plugins/planning/evals/interview-recommendation-basis-and-flip/prompt.md +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/prompt.md @@ -1,11 +1,15 @@ --- -description: "Eval-first case for checkable recommendations. Why hard: the prompt states enough facts that every recommendation can rest on something the user said, but 'Basis: judgment' with no reference and no condition that would change it is the easy default." -tags: [interview, recommendation, basis] +description: "Eval-first case for checkable recommendations. Why hard: the prompt states enough facts that every recommendation can rest on something the user said, but 'Basis: judgment' with no reference and no condition that would change it is the easy default. The prompt starts past the intake round, so the first decision round carries the recommendations and its Scope check: header line." +tags: [interview, recommendation, basis, no-network] runs: 3 -max_turns: 12 -timeout_seconds: 600 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "Every recommendation in round 1 names a checkable basis (a file and line, a URL, a check run this session named with what it showed, a session-transcript finding the user said or a tool returned (never the reply's own earlier claim), or a fact the user stated), and names the condition that would change it only where one exists" +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "Every recommendation in round 2 names a checkable basis in its printed reason (a file and line, a URL, a check run this session named with what it showed, a session-transcript finding the user said or a tool returned (never the reply's own earlier claim), or a fact the user stated), and names the condition that would change it only where one exists" --- -/planning:interview We need rate limiting on our public REST API. Facts: we run on Postgres 16 with a single Redis 7 instance already used for sessions; traffic peaks at about 2,000 requests per second; our published partner terms promise 1,000 requests per minute per API key. Ask me your first round. +/planning:interview We need rate limiting on our public REST API. Facts: we run on Postgres 16 with a single Redis 7 instance already used for sessions; traffic peaks at about 2,000 requests per second; our published partner terms promise 1,000 requests per minute per API key. + +Round 1 is answered: Q1, the problem is that a few partner integrations send bursts that slow the API for everyone; Q2, the users are partner developers calling the API with their keys, and our on-call engineers. + +Ask me round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/scope-check-line.json b/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-recommendation-basis-and-flip/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/case.yaml b/plugins/planning/evals/interview-reopen-contradicted-answer/case.yaml new file mode 100644 index 00000000000..d383549a7da --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: interview-reopen-contradicted-answer +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/graders/calls-reopen.md b/plugins/planning/evals/interview-reopen-contradicted-answer/graders/calls-reopen.md new file mode 100644 index 00000000000..f92d2cc3bbe --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/graders/calls-reopen.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Bash +input_match: 'round\.sh\W{0,4}\s+reopen\b[^\n]*\bQ3\b' +min: 1 +arm: with-only +--- diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/graders/ledger-reopened.md b/plugins/planning/evals/interview-reopen-contradicted-answer/graders/ledger-reopened.md new file mode 100644 index 00000000000..12d85304f3b --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/graders/ledger-reopened.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '^(?=[\s\S]*\n- Q3 \| open \|[^\n]*reopened:: round \d+)(?=[\s\S]*\n- Q5 \|[^\n]*recheck:: Q3\b)(?=[\s\S]*\n## Reopened answers\s*\n(?:[^\n]*\n)*?- Q3 \| round \d+ \| was: [^\n]*Redis)' +target: { source: file, path: .work/api-rate-limit/interview-checklist.md } +arm: both +--- diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/graders/reply-shows-reopen.md b/plugins/planning/evals/interview-reopen-contradicted-answer/graders/reply-shows-reopen.md new file mode 100644 index 00000000000..d5beb557a17 --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/graders/reply-shows-reopen.md @@ -0,0 +1,16 @@ +--- +type: llm +arm: both +focus: last_message +--- + +PASS only if all of these hold: + +1. The reply asks Q3 again under the id Q3 (not a new id), as a reopened question. +2. It shows Q3's earlier answer (the counters in the existing shared Redis) and the fact that + contradicts it (docs/platform/redis.md: the shared Redis holds web sessions only). +3. It names Q5 (the counting scheme) as resting on Q3 and needing a recheck once Q3 is answered + again. + +FAIL if the reply changes Q3's answer itself without asking, asks the storage question under a new +id, or reopens Q3 without naming Q5. diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/prompt.md b/plugins/planning/evals/interview-reopen-contradicted-answer/prompt.md new file mode 100644 index 00000000000..502bc8116c4 --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/prompt.md @@ -0,0 +1,15 @@ +--- +description: "A fact reported after round 2 contradicts answered Q3, and Q5 depends on Q3. Why hard: the easy moves are to quietly change Q3's answer, to ask a fresh question under a new id, or to reopen Q3 and forget Q5; the skill reopens Q3 under its own number with round.sh reopen, keeps the old answer, and flags Q5 for recheck." +tags: [interview, reopen, register, scaffold, no-network] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] +expected_outcome: "The run calls round.sh reopen for Q3; the reply shows Q3 reopened under its own number with its old answer (the shared Redis) and the contradicting fact from docs/platform/redis.md, and names Q5 for recheck; the ledger's Q3 row is open with reopened::, Q5 carries recheck:: Q3, and a Reopened answers section holds Q3's old answer" +--- + +/planning:interview Resume the api-rate-limit interview; the ledger is .work/api-rate-limit/interview-checklist.md and rounds 1 and 2 are answered. + +The lookup I asked for came back: docs/platform/redis.md says the shared Redis instance holds web sessions only and allows no other application keys. + +Carry on with round 3, then stop for my answers. diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/samples/calls-reopen.json b/plugins/planning/evals/interview-reopen-contradicted-answer/samples/calls-reopen.json new file mode 100644 index 00000000000..c1005b783ad --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/samples/calls-reopen.json @@ -0,0 +1,41 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Bash", + "input": { + "command": "bash \"/plugins/planning/surface/round.sh\" reopen --ledger .work/api-rate-limit/interview-checklist.md Q3 --round 3 --fact-file .work/api-rate-limit/fact-q3.txt" + } + } + ], + "why": "Q3 reopened through the tool" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Edit", + "input": { + "file_path": ".work/api-rate-limit/interview-checklist.md", + "old_string": "| answered | round 2 | Where", + "new_string": "| open | round 2 | Where" + } + } + ], + "why": "the row is hand-edited" + }, + { + "answer": [ + { + "tool": "Bash", + "input": { + "command": "bash round.sh reopen --ledger x.md Q5 --round 3 --fact-file f" + } + } + ], + "why": "the wrong row is reopened" + } + ] +} diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/samples/ledger-reopened.json b/plugins/planning/evals/interview-reopen-contradicted-answer/samples/ledger-reopened.json new file mode 100644 index 00000000000..da6ce3cfa06 --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/samples/ledger-reopened.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "# /planning:interview Checklist\n\n## Open-question register\n\n- Q1 | answered | round 1 | What problem? | bursts\n- Q3 | open | round 2 | Where are the rate-limit counters stored? | reopened:: round 3\n- Q4 | answered | round 2 | What limit? | 1,000 per minute\n- Q5 | answered | round 2 | How are requests counted? | a Redis counter per key per minute window; depends:: Q3; recheck:: Q3\n\n## Reopened answers\n\n- Q3 | round 3 | was: in the existing shared Redis 7 instance | fact: docs/platform/redis.md allows sessions only\n", + "why": "reopened, recheck flagged, old answer kept" + } + ], + "fail": [ + { + "answer": "# /planning:interview Checklist\n\n## Open-question register\n\n- Q1 | answered | round 1 | What problem? | bursts\n- Q3 | open | round 2 | Where are the rate-limit counters stored? | reopened:: round 3\n- Q4 | answered | round 2 | What limit? | 1,000 per minute\n- Q5 | answered | round 2 | How are requests counted? | a Redis counter per key per minute window; depends:: Q3\n\n## Reopened answers\n\n- Q3 | round 3 | was: in the existing shared Redis 7 instance | fact: docs/platform/redis.md allows sessions only\n", + "why": "Q5 not flagged" + }, + { + "answer": "# /planning:interview Checklist\n\n## Open-question register\n\n- Q1 | answered | round 1 | What problem? | bursts\n- Q3 | open | round 2 | Where are the rate-limit counters stored? | reopened:: round 3\n- Q4 | answered | round 2 | What limit? | 1,000 per minute\n- Q5 | answered | round 2 | How are requests counted? | a Redis counter per key per minute window; depends:: Q3; recheck:: Q3\n\n", + "why": "the old answer is dropped" + }, + { + "answer": "# /planning:interview Checklist\n\n## Open-question register\n\n- Q1 | answered | round 1 | What problem? | bursts\n- Q3 | answered | round 2 | Where are the rate-limit counters stored? | a dedicated Redis instance\n- Q4 | answered | round 2 | What limit? | 1,000 per minute\n- Q5 | answered | round 2 | How are requests counted? | a Redis counter per key per minute window; depends:: Q3; recheck:: Q3\n\n## Reopened answers\n\n- Q3 | round 3 | was: in the existing shared Redis 7 instance | fact: docs/platform/redis.md allows sessions only\n", + "why": "Q3's answer was changed in place" + } + ] +} diff --git a/plugins/planning/evals/interview-reopen-contradicted-answer/scaffold.sh b/plugins/planning/evals/interview-reopen-contradicted-answer/scaffold.sh new file mode 100755 index 00000000000..8c9b445e988 --- /dev/null +++ b/plugins/planning/evals/interview-reopen-contradicted-answer/scaffold.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail + +mkdir -p .work/api-rate-limit docs/platform +cat >docs/platform/redis.md <<'MD' +# Shared Redis + +The shared Redis 7 instance holds web sessions only. Application keys of any other kind are not +allowed on it; request a dedicated instance from the platform team. +MD +cat >.work/api-rate-limit/interview-checklist.md <<'MD' +# /planning:interview Checklist + +## Open-question register + +- Q1 | answered | round 1 | What problem does this solve? | a few partner integrations send bursts that slow the API for everyone +- Q2 | answered | round 1 | Who uses it? | partner developers calling the API with their keys +- Q3 | answered | round 2 | Where are the rate-limit counters stored? | in the existing shared Redis 7 instance +- Q4 | answered | round 2 | What limit applies? | 1,000 requests per minute per API key, as the partner terms promise +- Q5 | answered | round 2 | How are requests counted? | a Redis counter per key per minute window, expiring after the window; depends:: Q3 +MD +git init -q -b main +git config user.name eval +git config user.email eval@example.invalid +git add . +git commit -q -m "docs: platform redis policy" diff --git a/plugins/planning/evals/interview-reported-problem-instance-question/case.yaml b/plugins/planning/evals/interview-reported-problem-instance-question/case.yaml index c87f3862f56..db88dcbdafa 100644 --- a/plugins/planning/evals/interview-reported-problem-instance-question/case.yaml +++ b/plugins/planning/evals/interview-reported-problem-instance-question/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-reported-problem-instance-question description: A stated problem whose only evidence is what support reported gets one instance question (last concrete instance, what it cost, who hit it), never a why chain and never a question reopening the named solution -tags: [interview, problem-framing, reported-evidence, scaffold] +tags: [interview, problem-framing, reported-evidence, scaffold, no-network] expected_outcome: The reply asks the user, in one question, for the last concrete time the CSV export was slow, what it cost or how long it took, and which customer hit it; it asks no chain of why questions, does not ask whether to keep or replace streaming, and no Write or Edit touches PLAN.md. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/closing-probe.md b/plugins/planning/evals/interview-round-quality-standard/graders/closing-probe.md similarity index 100% rename from plugins/planning/evals/interview-pacing-round-arm/graders/closing-probe.md rename to plugins/planning/evals/interview-round-quality-standard/graders/closing-probe.md diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/covers-a.md b/plugins/planning/evals/interview-round-quality-standard/graders/covers-a.md similarity index 100% rename from plugins/planning/evals/interview-pacing-round-arm/graders/covers-a.md rename to plugins/planning/evals/interview-round-quality-standard/graders/covers-a.md diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/covers-b.md b/plugins/planning/evals/interview-round-quality-standard/graders/covers-b.md similarity index 100% rename from plugins/planning/evals/interview-pacing-round-arm/graders/covers-b.md rename to plugins/planning/evals/interview-round-quality-standard/graders/covers-b.md diff --git a/plugins/planning/evals/interview-pacing-round-arm/graders/covers-c.md b/plugins/planning/evals/interview-round-quality-standard/graders/covers-c.md similarity index 100% rename from plugins/planning/evals/interview-pacing-round-arm/graders/covers-c.md rename to plugins/planning/evals/interview-round-quality-standard/graders/covers-c.md diff --git a/plugins/planning/evals/interview-round-quality-standard/graders/covers-d.md b/plugins/planning/evals/interview-round-quality-standard/graders/covers-d.md new file mode 100644 index 00000000000..195ba3f932b --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/graders/covers-d.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?![^\n]*\bplanning\b)[^\n]*(?:background|(?:after|in|inside|outside|within|during) the (?:(?:web|same|click''s own) )*(?:request|click)|(?:when|once) (?:(?:it''?s|it is|the file is) )?ready|time ?out|stream|chunk|40,000|large export)' +flags: i +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-round-quality-standard/graders/question-quality.md b/plugins/planning/evals/interview-round-quality-standard/graders/question-quality.md new file mode 100644 index 00000000000..5a2b116679b --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/graders/question-quality.md @@ -0,0 +1,22 @@ +--- +type: llm +arm: both +focus: last_message +--- + +Find the round's questions on these four decisions: the date range one export covers, how amounts +are written across currencies, who may export, and how large exports are built. A separate +question that defers the large-export mechanism to planning does not replace the large-export +question. + +PASS only if each of the four has a question, and each of those four questions has all of: + +1. A `Context:` block with at least one line that cites a fact the user stated (the calendar-month + close, integer cents with a currency code, USD/EUR/GBP billing, the spreadsheet, the support and + finance roles, the 40,000 rows, the 30-second request timeout) and says where it came from. +2. Either a recommendation whose printed reason names the evidence it rests on and its source (for + example "(verified, your notes)"), or a `Withheld:` line naming the evidence that would settle it. +3. Alternatives, each saying what it would gain or cost. + +FAIL if any of the four decisions has no question, or any of those questions lacks one of the +three parts, or a recommendation's reason names no evidence (bare judgment). diff --git a/plugins/planning/evals/interview-round-quality-standard/graders/rule-between-questions.md b/plugins/planning/evals/interview-round-quality-standard/graders/rule-between-questions.md new file mode 100644 index 00000000000..aac529f3f83 --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/graders/rule-between-questions.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\S\n]*-{3}[^\S\n]*\n(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-round-quality-standard/graders/scope-check-line.md b/plugins/planning/evals/interview-round-quality-standard/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-round-quality-standard/prompt.md b/plugins/planning/evals/interview-round-quality-standard/prompt.md new file mode 100644 index 00000000000..9723a3a858a --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/prompt.md @@ -0,0 +1,17 @@ +--- +description: "Question quality at the default standard detail level, on the retired pacing experiment's round-arm task (round 2 of an invoice CSV export, four planted decisions: date range, amount format across currencies, who may export, large exports). Source: the consumer's own writing, synthetic. Not part of that frozen experiment. Why hard: the four decisions are not listed as a frontier; the skill must find each in the stated facts and ask all of them in one round, each with a Context line citing a stated fact, a recommendation whose printed reason names its evidence (or a Withheld: line), and alternatives, with a three-dash rule line between questions and a Scope check: line in the header." +tags: [interview, round-layout, question-quality, hard, no-network] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "Round 2, headed by a Scope check: line, asks a question on each of date range, amount format and currencies, who may export, and large exports, every question with a Context line citing a stated fact, a recommendation whose printed reason names its evidence or a Withheld: line, and alternatives, with a three-dash rule line between questions, ending with the closing probe" +--- + +/planning:interview me Add a CSV export of invoices to the billing admin page. + +Round 1 is answered and the terms are agreed: Q1, the quality bar is production; Q2, the export is for the finance team's month-end reconciliation; Q3, "invoice" means an issued invoice, drafts excluded. + +What I know about the system: finance closes its books on the calendar month. Invoices store amounts as integer cents with a currency code, and customers are billed in USD, EUR and GBP. Finance opens exported files in a spreadsheet. The admin page has two roles, `support` and `finance`; support staff can see invoice lists but not payment details. One calendar month of invoices for the largest customer account is about 40,000 rows, and the page's other downloads build the file inside the web request, which times out after 30 seconds. + +Ask round 2 now, the whole round in one message, then stop for my answers. diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/closing-probe.json b/plugins/planning/evals/interview-round-quality-standard/samples/closing-probe.json similarity index 100% rename from plugins/planning/evals/interview-pacing-round-arm/samples/closing-probe.json rename to plugins/planning/evals/interview-round-quality-standard/samples/closing-probe.json diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/covers-a.json b/plugins/planning/evals/interview-round-quality-standard/samples/covers-a.json similarity index 100% rename from plugins/planning/evals/interview-pacing-round-arm/samples/covers-a.json rename to plugins/planning/evals/interview-round-quality-standard/samples/covers-a.json diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/covers-b.json b/plugins/planning/evals/interview-round-quality-standard/samples/covers-b.json similarity index 100% rename from plugins/planning/evals/interview-pacing-round-arm/samples/covers-b.json rename to plugins/planning/evals/interview-round-quality-standard/samples/covers-b.json diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/covers-c.json b/plugins/planning/evals/interview-round-quality-standard/samples/covers-c.json similarity index 100% rename from plugins/planning/evals/interview-pacing-round-arm/samples/covers-c.json rename to plugins/planning/evals/interview-round-quality-standard/samples/covers-c.json diff --git a/plugins/planning/evals/interview-pacing-round-arm/samples/covers-d.json b/plugins/planning/evals/interview-round-quality-standard/samples/covers-d.json similarity index 95% rename from plugins/planning/evals/interview-pacing-round-arm/samples/covers-d.json rename to plugins/planning/evals/interview-round-quality-standard/samples/covers-d.json index cdb7c3a2087..cd903958ded 100644 --- a/plugins/planning/evals/interview-pacing-round-arm/samples/covers-d.json +++ b/plugins/planning/evals/interview-round-quality-standard/samples/covers-d.json @@ -7,6 +7,10 @@ { "answer": "**Round 2.** What we have so far: a production CSV export of issued invoices for month-end reconciliation.\nLater: column list, file naming, failure behavior.\n\n❓ Q4: **Should the file be prepared after the click and downloaded once it's ready, instead of being built during the click like the page's other downloads?**\nContext:\n- One month for the largest account is about 40,000 rows, and downloads built inside the web request stop after 30 seconds (your notes).\n\n➡️ My recommendation: **Yes, build it in the background**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Build it inside the request like the other downloads. Nothing new to run, but the largest months time out (your notes; origin: new (ours))\n\n❓ Q5: **Should the choice of job runner, file storage and ready notice wait for planning?**\nContext:\n- Q4 names a mechanism, and your notes name no job runner or file store (your notes).\n\n➡️ My recommendation: **Yes, leave the mechanism to planning**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Settle it now. One session decides everything, but before anyone has read the code (origin: new (ours))\n\n❓ Q6: **Should one export cover one calendar month across all customer accounts?**\nContext:\n- Finance closes its books on the calendar month (your notes).\n\n➡️ My recommendation: **Yes, one calendar month, all accounts**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Any date range. Flexible, but ranges that cross a month end stop matching the books (origin: new (ours))\n\n❓ Q7: **Should amounts be written as two-decimal numbers with a separate currency-code column?**\nContext:\n- Amounts are stored as integer cents in USD, EUR and GBP, and finance opens the file in a spreadsheet (your notes).\n\n➡️ My recommendation: **Yes, `1234.50` plus a currency column, no conversion**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Raw integer cents. Exact, but every formula divides by 100 (origin: new (ours))\n\n❓ Q8: **Should only the `finance` role be able to export?**\nContext:\n- Support staff can see invoice lists but not payment details (your notes).\n\n➡️ My recommendation: **Yes, `finance` only**\nOrigin: new (ours)\nBasis: verified, your notes\n\nAlternatives:\n- (a) Both roles, with payment columns removed for support (origin: new (ours))\n\n❓ Q9: **Which timezone sets the month boundary?**\nContext:\n- Finance closes on the calendar month, and nothing says in which timezone (your notes).\n\nWithheld: only the timezone finance's books close in can settle this.\n\nAlternatives:\n- (a) UTC\n- (b) The company's accounting timezone\n\nIs there something else that would change these recommendations?", "why": "D phrased as preparing the file after the click and downloading it once it's ready" + }, + { + "answer": "### Round 2\nScope check: 7 of 27 questions registered\n\n❓ Q4: **Should the export be requested, built outside the web request, and offered as a download once ready, instead of built while the person waits?**\n\nContext:\n- One month for the largest account is about 40,000 rows, and in-request downloads time out after 30 seconds (user, round 2).\n\n➡️ My recommendation: **Yes, build it outside the request.**\n", + "why": "Asks the large-export decision as 'built outside the web request ... once ready', the wording a correct F3 reply used." } ], "fail": [ diff --git a/plugins/planning/evals/interview-round-quality-standard/samples/rule-between-questions.json b/plugins/planning/evals/interview-round-quality-standard/samples/rule-between-questions.json new file mode 100644 index 00000000000..fe1ac8e278b --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/samples/rule-between-questions.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "❓ Q4: **Should one export cover one calendar month?**\n\nContext:\n- Finance closes on the calendar month (your notes).\n\n➡️ My recommendation: **Yes**. (verified, your notes)\n\nAlternatives:\n- (a) A free date range.\n\n---\n\n❓ Q5: **How are amounts written?**", + "why": "a --- rule between two questions" + } + ], + "fail": [ + { + "answer": "❓ Q4: **Should one export cover one calendar month?**\n\nAlternatives:\n- (a) A free date range.\n\n❓ Q5: **How are amounts written?**", + "why": "no rule between the questions" + }, + { + "answer": "---\n\n❓ Q4: **Should one export cover one calendar month?**\n\n❓ Q5: **How are amounts written?**\n\n---", + "why": "rules only before the first and after the last question" + } + ] +} diff --git a/plugins/planning/evals/interview-round-quality-standard/samples/scope-check-line.json b/plugins/planning/evals/interview-round-quality-standard/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-round-quality-standard/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-round-topic-groups/graders/scope-check-line.md b/plugins/planning/evals/interview-round-topic-groups/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-round-topic-groups/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-round-topic-groups/graders/topic-groups.md b/plugins/planning/evals/interview-round-topic-groups/graders/topic-groups.md index 8babde1e561..7977bbb5fc4 100644 --- a/plugins/planning/evals/interview-round-topic-groups/graders/topic-groups.md +++ b/plugins/planning/evals/interview-round-topic-groups/graders/topic-groups.md @@ -11,7 +11,10 @@ PASS only if all of these hold: (who is alerted, what an alert message includes). Each question appears under exactly one header. 2. Question ids run on from Q4 with no gap and no restart across the headers. -3. Each question keeps its own block with a recommendation, a `Basis:` line and its alternatives. +3. Each question keeps its own block with a recommendation whose reason names its basis, and its + alternatives. A question whose recommendation is withheld meets the recommendation part with a + `Withheld:` line that names the evidence that would settle it, and still keeps its + alternatives. The headers do not replace any of that. 4. The round ends with the closing probe, the invitation to name something that would change the recommendations, after the last question. No table summarizes the round's questions. diff --git a/plugins/planning/evals/interview-round-topic-groups/prompt.md b/plugins/planning/evals/interview-round-topic-groups/prompt.md index 211cbda7efc..0d49044a6d7 100644 --- a/plugins/planning/evals/interview-round-topic-groups/prompt.md +++ b/plugins/planning/evals/interview-round-topic-groups/prompt.md @@ -1,11 +1,11 @@ --- -description: "Round 2 of a webhook-sender interview whose frontier spans two topics, retry policy and failure alerting. Why hard: the round must keep every question in its full layout and add topic headers; the easy failures are one flat numbered list, or a closing summary table of the questions." +description: "Round 2 of a webhook-sender interview whose frontier spans two topics, retry policy and failure alerting. Why hard: the round must keep every question in its full layout and add topic headers; the easy failures are one flat numbered list, or a closing summary table of the questions. The round header carries a Scope check: line." tags: [interview, round-layout, topic-groups, no-network] runs: 3 max_turns: 30 timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "Round 2 asks the five decisions from Q4 on under at least two topic headers (retries and alerting), keeps each question in its full layout, and ends with the closing probe and no summary table" +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "Round 2 asks the five decisions from Q4 on under at least two topic headers (retries and alerting), keeps each question in its layout, carries a Scope check: line in the round header, and ends with the closing probe and no summary table" --- /planning:interview me Add retries and failure alerting to our outbound webhook sender. diff --git a/plugins/planning/evals/interview-round-topic-groups/samples/scope-check-line.json b/plugins/planning/evals/interview-round-topic-groups/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-round-topic-groups/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-round1-unknowns-offers-wayfind/prompt.md b/plugins/planning/evals/interview-round1-unknowns-offers-wayfind/prompt.md index bbe2a9fbabb..4234be3c8f0 100644 --- a/plugins/planning/evals/interview-round1-unknowns-offers-wayfind/prompt.md +++ b/plugins/planning/evals/interview-round1-unknowns-offers-wayfind/prompt.md @@ -1,9 +1,9 @@ --- description: "The opening request itself names unknowns in four separate areas. Why hard: no round has run yet, so the fog trigger cannot fire; the easy failures are to say nothing about wayfind until later rounds, or to treat the fog as a reason to switch to wayfind and skip the interview." -tags: [interview, routing, wayfind, round-1] +tags: [interview, routing, wayfind, round-1, no-network] runs: 3 -max_turns: 12 -timeout_seconds: 600 +max_turns: 30 +timeout_seconds: 900 allowed_tools: [Read, Glob, Grep, Skill] expected_outcome: "Round 1 includes a question recommending /planning:wayfind chart for the four unknown areas, worded so the user can accept or decline it, and the other round-1 questions go out in the same message; wayfind is not invoked" --- diff --git a/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/case.yaml b/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/case.yaml index 78cd9c7887d..2b50a74ff71 100644 --- a/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/case.yaml +++ b/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/case.yaml @@ -1,14 +1,14 @@ schema_version: "1.1" name: interview-scenario-walk-two-purchase-kinds -description: "One product sold two ways (a one-off bag and a monthly subscription of the same bag) leaves 'order' and 'refund' unsettled, and the user calls it confusing and asks for scenarios. Why hard: the easy reply defines the terms in prose or asks 'what is an order?', where the walk puts numbered, concrete scenarios inside one question, each with its own recommendation and the term or rule it forces." -tags: [interview, scenario-walk, domain-language, scaffold] -expected_outcome: "The round holds a numbered question whose Context carries S1, S2, ... scenarios, each concrete (a named person, an amount, a day or date), stating what it tests and ending with its own recommendation and the term or rule that answer forces, and the user is told to answer per scenario (for example 'S2: a')." +description: "One product sold two ways (a one-off bag and a monthly subscription of the same bag) leaves 'order' and 'refund' unsettled, and the user calls it confusing and asks for scenarios. Why hard: the easy reply defines the terms in prose or asks 'what is an order?', where the walk puts numbered, concrete scenarios inside one question, each with its own recommendation and the term or rule it forces. The prompt starts past the intake round, so the walk comes in the first decision round, headed by a Scope check: line." +tags: [interview, scenario-walk, domain-language, scaffold, no-network] +expected_outcome: "Round 2, headed by a Scope check: line, holds a numbered question whose Context carries S1, S2, ... scenarios, each concrete (a named person, an amount, a day or date), stating what it tests and ending with its own recommendation and the term or rule that answer forces, and the user is told to answer per scenario (for example 'S2: a')." context: scaffold_script: scaffold.sh execution: - prompt: "/planning:interview me We sell the same coffee bag two ways: a one-off purchase and a monthly subscription. I find it confusing what an 'order' is once a subscription renews, and what a refund applies to. Walk me through some scenarios to settle it. I'm here and will answer." - allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit] - max_turns: 20 + prompt: "Use the interview skill from the planning plugin, in me mode: We sell the same coffee bag two ways: a one-off purchase and a monthly subscription. I find it confusing what an 'order' is once a subscription renews, and what a refund applies to. Walk me through some scenarios to settle it. Round 1 is answered: Q1, the problem is that support cannot tell customers which purchase a refund covers; Q2, the users are our customers and the two support staff who issue refunds. I'm here and will answer; ask me round 2." + allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] + max_turns: 30 timeout_seconds: 900 graders: - name: skill-fired @@ -20,6 +20,11 @@ graders: type: regex pattern: '\bS1\**\s*:[\s\S]*\bS2\**\s*:' arm: both + - name: scope-check-line + type: regex + pattern: '(?:^|\n)[^\n]{0,6}Scope check:' + target: last_message + arm: both - name: scenario-walk-shape type: llm arm: both diff --git a/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/samples/scope-check-line.json b/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-scenario-walk-two-purchase-kinds/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-scope-cut-moscow/graders/scope-check-line.md b/plugins/planning/evals/interview-scope-cut-moscow/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-scope-cut-moscow/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-scope-cut-moscow/prompt.md b/plugins/planning/evals/interview-scope-cut-moscow/prompt.md index fe9e643dceb..a2ed9494d9c 100644 --- a/plugins/planning/evals/interview-scope-cut-moscow/prompt.md +++ b/plugins/planning/evals/interview-scope-cut-moscow/prompt.md @@ -1,11 +1,15 @@ --- -description: "Eval-first case for the scope-cut round. Why hard: eight capabilities for two engineers in one quarter is plainly too big, so recommending /planning:wayfind at once looks right, but nobody has yet asked which items the effort fails without." +description: "Eval-first case for the scope-cut round. Why hard: eight capabilities for two engineers in one quarter is plainly too big, so recommending /planning:wayfind at once looks right, but nobody has yet asked which items the effort fails without. The prompt starts past the intake round, so the scope cut comes with recommendations in the first decision round, headed by a Scope check: line." tags: [interview, scope, wayfind, no-network] runs: 3 max_turns: 30 timeout_seconds: 900 -allowed_tools: [Read, Glob, Grep, Skill] +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] expected_outcome: "Before recommending /planning:wayfind, the reply runs a scope cut: for each capability it asks or applies the Must test (if this is not delivered, is the effort cancelled or worked around?), recommends a bucket per item, keeps Must to about 60% of the effort or less, and routes the rest to /planning:wayfind or later" --- /planning:interview Build our internal developer portal this quarter with two engineers: a service catalog, team scorecards, onboarding guides, cloud cost dashboards, on-call schedules, an incident timeline, an API explorer, and a plugin system so other teams can extend it. + +Round 1 is answered: Q1, the problem is that engineers spend their first weeks finding which team owns which service and how to reach it; Q2, the users are our 120 product engineers and their team leads. + +Ask round 2 now, then stop for my answers. diff --git a/plugins/planning/evals/interview-scope-cut-moscow/samples/scope-check-line.json b/plugins/planning/evals/interview-scope-cut-moscow/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-scope-cut-moscow/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-single-pacing-one-question/graders/no-basis-line.md b/plugins/planning/evals/interview-single-pacing-one-question/graders/no-basis-line.md new file mode 100644 index 00000000000..0f5488f6658 --- /dev/null +++ b/plugins/planning/evals/interview-single-pacing-one-question/graders/no-basis-line.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '(?:^|\n)[^\S\n]{0,4}(?:[^\w\s]{1,2}[^\S\n]*)?(?:\*\*)?(?:Basis|Origin)(?:\*\*)?:' +match: not_contains +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-single-pacing-one-question/graders/one-question-full-layout.md b/plugins/planning/evals/interview-single-pacing-one-question/graders/one-question-full-layout.md index deac3eb8315..3f6b934fe5a 100644 --- a/plugins/planning/evals/interview-single-pacing-one-question/graders/one-question-full-layout.md +++ b/plugins/planning/evals/interview-single-pacing-one-question/graders/one-question-full-layout.md @@ -9,13 +9,17 @@ PASS only if all of these hold: 1. The reply asks exactly one question, and it is one of the five frontier decisions (attempt count, delay, what happens after the last attempt, who is alerted, what an alert message includes). -2. That question is in the full layout: a context block, a recommendation with its `Basis:` line, - and its alternatives. +2. That question is in the question layout: a context block, a recommendation whose reason names + its basis, and its alternatives. 3. The reply ends with a closing probe inviting something that would change the recommendation, and then stops for the answer. 4. The reply has no topic header and no index table, and it does not list, preview or number the other four decisions as questions to answer now. +The round header the skill prescribes is allowed and is not a topic header: the round's title line +(for example `## Round 2`), its `Scope check:` line, and a `Later:` line that names only topics +that are none of the five decisions (for example `Later: replay controls`). + FAIL if the reply asks two or more of the decisions, asks one and lists the others for the user to -answer in the same reply, adds an index table, or drops the recommendation, basis or alternatives -from the one question. +answer in the same reply (a `Later:` line naming any of the other four decisions included), adds +an index table, or drops the recommendation, basis or alternatives from the one question. diff --git a/plugins/planning/evals/interview-single-pacing-one-question/prompt.md b/plugins/planning/evals/interview-single-pacing-one-question/prompt.md index 43d166a5c61..41b4929aa7a 100644 --- a/plugins/planning/evals/interview-single-pacing-one-question/prompt.md +++ b/plugins/planning/evals/interview-single-pacing-one-question/prompt.md @@ -1,11 +1,11 @@ --- -description: "Single pacing, asked for in the session: the user wants one question at a time while round 2's frontier holds five independent decisions. An eval run cannot set the plugin's question_pacing option, so the case uses the in-session request, which the skill says overrides the option for the session. Why hard: the frontier is computed whole, so the easy failure is to ask all five, or to ask one and add the rest as a preview, a topic header or an index." +description: "Single pacing, asked for in the session: the user wants one question at a time while round 2's frontier holds five independent decisions. An eval run cannot set the plugin's question_pacing option, so the case uses the in-session request, which the skill says overrides the option for the session. Why hard: the frontier is computed whole, so the easy failure is to ask all five, or to ask one and add the rest as a preview, a topic header or an index. At the default standard detail level the question shows its recommendation with its reason and no separate Origin: or Basis: line." tags: [interview, round-layout, pacing, no-network] runs: 3 -max_turns: 12 -timeout_seconds: 600 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply asks exactly one of the five frontier decisions in the full question layout with the closing probe, with no topic header, no index table and no other question listed, then stops" +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "The reply asks exactly one of the five frontier decisions in the standard question layout (no Origin: or Basis: line) with the closing probe, with no topic header, no index table and no other question listed, then stops" --- /planning:interview me Add retries and failure alerting to our outbound webhook sender. diff --git a/plugins/planning/evals/interview-single-pacing-one-question/samples/no-basis-line.json b/plugins/planning/evals/interview-single-pacing-one-question/samples/no-basis-line.json new file mode 100644 index 00000000000..23e63555edf --- /dev/null +++ b/plugins/planning/evals/interview-single-pacing-one-question/samples/no-basis-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Round 2.\nScope check: 9 of 12 questions registered\n\n❓ Q4: **How many delivery attempts before giving up?**\n\nContext:\n- Today the sender makes one POST and retries nothing (your notes).\n\n➡️ My recommendation: **Five attempts**. Receivers deduplicate on event_id, so retries are safe (verified, user, round 1).\n\nAlternatives:\n- (a) Three attempts.\n\nIs there something else that would change this recommendation?", + "why": "the standard print: the basis rides in the reason, no Basis: or Origin: line" + }, + { + "answer": "Q4: **How many attempts?**\nMy recommendation: **Five**. The basis is your round-1 answer.\n", + "why": "the word basis inside a sentence is not a label line" + } + ], + "fail": [ + { + "answer": "❓ Q4: **How many delivery attempts?**\n\nContext:\n- One POST today.\n\n➡️ My recommendation: **Five attempts**.\nOrigin: new (ours)\nBasis: verified, user, round 1\n", + "why": "the old full layout prints Origin: and Basis: lines" + }, + { + "answer": "Q4: **How many attempts?**\nMy recommendation: **Five**.\n**Basis:** verified, your notes\n", + "why": "a bold Basis: label line" + } + ] +} diff --git a/plugins/planning/evals/interview-single-pacing-one-question/samples/one-question-full-layout.json b/plugins/planning/evals/interview-single-pacing-one-question/samples/one-question-full-layout.json index 0c35c40146a..aab700c7ee4 100644 --- a/plugins/planning/evals/interview-single-pacing-one-question/samples/one-question-full-layout.json +++ b/plugins/planning/evals/interview-single-pacing-one-question/samples/one-question-full-layout.json @@ -1,8 +1,12 @@ { "pass": [ { - "answer": "Round 2, one question at a time. Decided: production quality, at-least-once delivery, receivers deduplicate on `event_id`.\n\nQ4: **What happens to an event after its last failed attempt?**\nContext: the sender makes one POST today and drops the event when it fails (user, round 2). At-least-once delivery means a dropped event breaks the guarantee (user, round 1).\nMy recommendation: move it to a dead-letter table that an operator can replay.\nOrigin: new (ours)\nBasis: verified, user, round 1 (at-least-once delivery)\nAlternatives:\n- (a) Drop it and log. Simplest, but breaks at-least-once delivery (user, round 1; origin: new (ours))\n- (b) Keep retrying forever. Never loses an event, but a dead receiver grows the queue without bound (origin: new (ours))\n\nIs there something else that would change this recommendation?", - "why": "one question in full layout, closing probe, no header, index or preview" + "answer": "Round 2, one question at a time. Decided: production quality, at-least-once delivery, receivers deduplicate on `event_id`.\nScope check: 4 of 27 questions registered; round 1 added 3, settled 3\n\n❓ Q4: **What happens to an event after its last failed attempt?**\n\nContext:\n- The sender makes one POST today and drops the event when it fails (user, round 2).\n- At-least-once delivery means a dropped event breaks the guarantee (user, round 1).\n\n➡️ My recommendation: **Move it to a dead-letter table that an operator can replay**. A dropped event would break at-least-once delivery (verified, user, round 1: webhooks are delivered at least once).\n\nAlternatives:\n- (a) Drop it and log. Simplest, but breaks at-least-once delivery (user, round 1)\n- (b) Keep retrying forever. Never loses an event, but a dead receiver grows the queue without bound\n\nIs there something else that would change this recommendation?", + "why": "one question in the standard layout (no Origin: or Basis: line), closing probe, no topic header, index or preview" + }, + { + "answer": "## Round 2\n\nScope check: 4 of 27 questions registered\nLater: replay controls\n\n❓ Q4: **What happens to an event after its last failed attempt?**\n\nContext:\n- The sender makes one POST today and drops the event when it fails (user, round 2).\n- At-least-once delivery means a dropped event breaks the guarantee (user, round 1).\n\n➡️ My recommendation: **Move it to a dead-letter table that an operator can replay**. A dropped event would break at-least-once delivery (verified, user, round 1: webhooks are delivered at least once).\n\nAlternatives:\n- (a) Drop it and log. Simplest, but breaks at-least-once delivery (user, round 1)\n- (b) Keep retrying forever. Never loses an event, but a dead receiver grows the queue without bound\n\nIs there something else that would change this recommendation?", + "why": "the prescribed round header: title line, Scope check: and a Later: line naming a topic outside the five decisions" } ], "fail": [ @@ -13,6 +17,10 @@ { "answer": "Q4: What happens to an event after its last failed attempt?\nMy recommendation: a dead-letter table. Basis: verified, user, round 1. Alternatives: (a) drop and log.\n\nComing up next: Q5 attempt count (five), Q6 delay (backoff from 30 seconds), Q7 who is alerted (#webhooks), Q8 alert contents (endpoint, last status, events waiting). Feel free to answer those now too.", "why": "previews the other four decisions for the user to answer in the same reply" + }, + { + "answer": "## Round 2\n\nScope check: 4 of 27 questions registered\nLater: delay between attempts, who gets alerted, what the alert says\n\n❓ Q4: **What happens to an event after its last failed attempt?**\n\nContext:\n- The sender drops the event when its one POST fails (user, round 2).\n\n➡️ My recommendation: **Move it to a dead-letter table**. A dropped event breaks at-least-once delivery (verified, user, round 1).\n\nAlternatives:\n- (a) Drop it and log. Simplest, but breaks at-least-once delivery\n\nIs there something else that would change this recommendation?", + "why": "the Later: line previews three of the other frontier decisions" } ] } diff --git a/plugins/planning/evals/interview-skips-quick-behavior-change/case.yaml b/plugins/planning/evals/interview-skips-quick-behavior-change/case.yaml index a2a96d4791c..2659451633b 100644 --- a/plugins/planning/evals/interview-skips-quick-behavior-change/case.yaml +++ b/plugins/planning/evals/interview-skips-quick-behavior-change/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-skips-quick-behavior-change description: "A behavior change whose diff is quick to review and cheap to retry is not gated by an interview; the user is told to build it, review the diff and align there. On Sonnet this case also passed with the skip rule removed from the skill, so it guards against the skill forcing an interview on a quick change and does not show that the rule causes the skip." -tags: [interview, routing, quick-change, scaffold] +tags: [interview, routing, quick-change, scaffold, no-network] expected_outcome: The reply does not treat the behavior change alone as a reason to interview; it says the change can be built directly (or makes it) and aligned on once the diff is reviewed, asks no numbered question round, writes no interview ledger, and no Write or Edit touches PLAN.md. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-skips-quick-change-typo/case.yaml b/plugins/planning/evals/interview-skips-quick-change-typo/case.yaml index 154bd651bff..39d21e4e11e 100644 --- a/plugins/planning/evals/interview-skips-quick-change-typo/case.yaml +++ b/plugins/planning/evals/interview-skips-quick-change-typo/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-skips-quick-change-typo description: "A comment typo fix, quick to review and cheap to retry, is not gated by an interview; the user is told it needs no interview or Brief. On Sonnet this case also passed with the skip rule removed from the skill, so it guards against the skill forcing an interview on a quick change and does not show that the rule causes the skip." -tags: [interview, routing, quick-change, scaffold] +tags: [interview, routing, quick-change, scaffold, no-network] expected_outcome: The reply treats the typo fix as quick to review and cheap to retry and says it needs no interview or Brief (it may make the fix itself); it asks no numbered question round, writes no interview ledger, and no Write or Edit touches PLAN.md. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-stated-problem-no-rethink/case.yaml b/plugins/planning/evals/interview-stated-problem-no-rethink/case.yaml index b928baa79e6..a6d2525ed75 100644 --- a/plugins/planning/evals/interview-stated-problem-no-rethink/case.yaml +++ b/plugins/planning/evals/interview-stated-problem-no-rethink/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-stated-problem-no-rethink description: A request that states its problem and a solution gets the problem restated and confirmed, and no rethink-or-accept question -tags: [interview, problem-framing, stated-problem, scaffold] +tags: [interview, problem-framing, stated-problem, scaffold, no-network] expected_outcome: The reply asks no rethink-or-accept question and does not question the retry wrapper as the fix; it may restate the stated problem (webhook deliveries dropped when customer endpoints return 503 during their deploys) and ask the user to confirm it now or in a later round; no Write or Edit touches PLAN.md. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/one-offer-line.md b/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/one-offer-line.md deleted file mode 100644 index f7fd4bfe95f..00000000000 --- a/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/one-offer-line.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '^\W*surface: \S[^\n]*\(default\)\. Also available: [^\n]+$' -flags: m -match: "count:1" -target: last_message -arm: with-only ---- diff --git a/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/opening-lines.md b/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/opening-lines.md new file mode 100644 index 00000000000..2ed2852f2ef --- /dev/null +++ b/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/opening-lines.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^\W*Asking here\b[^\n]*\.[^\S\n]+[^\n]*\balso available\b[^\n]*$' +flags: mi +match: "count:1" +target: last_message +arm: with-only +--- diff --git a/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/proceeds-to-round.md b/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/proceeds-to-round.md index c080dfba79a..f8d328b6dc0 100644 --- a/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/proceeds-to-round.md +++ b/plugins/planning/evals/interview-surface-unset-offer-one-line/graders/proceeds-to-round.md @@ -6,13 +6,19 @@ focus: last_message PASS only if all of these hold: -1. One line names the default surface and the other available surfaces together (for example - `surface: terminal (default). Also available: page. Ask for one by name to switch.`), and no - other line repeats or expands that offer. +1. One line says where the interview asks and names the other available surfaces together (for + example `Asking here, one round at a time. A local page is also available; say which to + switch.`), and no other line repeats or expands that offer. The `Pacing:` line the skill + prescribes, which reports the pacing value and its layer (for example `Pacing: one full round at + a time (the default; no setting overrides it)`), is not a repeat of the offer, even where its + words match the offer's "one round at a time". The `Questions are saved to ` line the + resolver prints after the offer is not part of the offer. 2. The reply does not ask which surface to use and does not wait for a surface choice: the offer is a statement, and the reply goes on to the interview. -3. The reply asks round 1 of the interview about the webhook sender's retries and alerting in the - conversation: at least one numbered question with a recommendation and its basis. +3. The reply asks round 1 of this interview, on the webhook sender's retries and alerting, in the + conversation: at least one numbered question, open or with a recommendation. An intake round + of open discovery questions (the problem, the quality bar, who it is for, where the code lives) + counts: the skill asks those in round 1 and holds the retry and alerting decisions for round 2. FAIL if the offer is missing, is spread over several lines or a list, is phrased as a question the user must answer before the interview starts, or if the reply stops before asking round 1. diff --git a/plugins/planning/evals/interview-surface-unset-offer-one-line/prompt.md b/plugins/planning/evals/interview-surface-unset-offer-one-line/prompt.md index a283d51b732..7bb1d6c86bc 100644 --- a/plugins/planning/evals/interview-surface-unset-offer-one-line/prompt.md +++ b/plugins/planning/evals/interview-surface-unset-offer-one-line/prompt.md @@ -1,11 +1,11 @@ --- -description: "Surface unset, more than one surface available: an eval run cannot set the plugin's surface option, so the skill sees the literal user_config token, which the surface registry reads as unset. The host has bash, curl and Python 3, so the page is available beside the default. Why hard: the easy failures are to skip the resolver and ask inline with no offer, to turn the offer into a question and stop for a surface choice, or to restate the alternatives across several lines." +description: "Surface unset, more than one surface available: an eval run cannot set the plugin's surface option, so the skill sees the literal user_config token, which the surface registry reads as unset. The host has bash, curl and Python 3, so the page is available beside the default. Why hard: the easy failures are to skip the resolver and ask inline with no offer, to turn the offer into a question and stop for a surface choice, or to restate the alternatives across several lines. The opening prints one sentence per line in the round-8 shape: its first line says where the questions are asked and names the other available surfaces." tags: [interview, surface, registry, no-network] runs: 3 -max_turns: 14 -timeout_seconds: 600 -allowed_tools: [Read, Glob, Grep, Skill, Bash] -expected_outcome: "The run resolves the surface through surfaces/registry.mjs, prints its offer line once (the default and the alternatives on one line), and proceeds to round 1 without waiting for a surface choice" +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "The run resolves the surface through surfaces/registry.mjs, prints its offer once as the opening's first line (where it asks, and the other available surfaces), and proceeds to round 1 without waiting for a surface choice" --- /planning:interview me Add retries and failure alerting to our outbound webhook sender. Today it makes one POST per event and retries nothing. diff --git a/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/one-offer-line.json b/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/one-offer-line.json deleted file mode 100644 index 8556c59dbdd..00000000000 --- a/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/one-offer-line.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "surface: terminal (default). Also available: page. Ask for one by name to switch.\n\nRound 1.\n\nQ1: **How many delivery attempts before the sender gives up?**\nContext: the sender makes one POST per event today and retries nothing (user, this message).\nMy recommendation: five attempts.\nBasis: verified, user, this message (one POST, no retries)\n\nIs there something else that would change these recommendations?", - "why": "the offer is one line, then the round follows" - } - ], - "fail": [ - { - "answer": "Round 1.\n\nQ1: **How many delivery attempts before the sender gives up?**\nMy recommendation: five attempts.\nBasis: verified, user, this message", - "why": "no offer line" - }, - { - "answer": "surface: terminal (default). Also available: page. Ask for one by name to switch.\n\nRound 1 ...\n\nsurface: terminal (default). Also available: page. Ask for one by name to switch.", - "why": "the offer line is printed twice" - }, - { - "answer": "Surfaces:\n- terminal (default)\n- page\n\nRound 1 ...", - "why": "the offer is spread over a list, not one line" - } - ] -} diff --git a/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/opening-lines.json b/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/opening-lines.json new file mode 100644 index 00000000000..4655c3aba4a --- /dev/null +++ b/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/opening-lines.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Asking here, one round at a time. A local page is also available; say which to switch.\nQuestions are saved to .work/webhook-retries/interview-checklist.md.\n\n**Round 1.** Retries and alerting for the webhook sender.\n\n❓ Q1: **What goes wrong today when a delivery fails?**", + "why": "the round-8 opening: where it asks and the alternatives on the first line, then the ledger line" + } + ], + "fail": [ + { + "answer": "surface: terminal (default). Also available: page. Ask for one by name to switch.\n\n**Round 1.**\n\n❓ Q1: **What goes wrong today?**", + "why": "the old one-line offer shape" + }, + { + "answer": "Asking here, one round at a time.\nA local page is also available; say which to switch.\n\nQ1: **What goes wrong today?**", + "why": "the offer is split, so the alternatives are not named on the line that says where it asks" + }, + { + "answer": "Asking here, one round at a time. A local page is also available; say which to switch.\n\nQ1: **What goes wrong today?**\n\nAsking here, one round at a time. A local page is also available; say which to switch.", + "why": "the offer is repeated" + } + ] +} diff --git a/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/proceeds-to-round.json b/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/proceeds-to-round.json index 0043bf5ec5f..91d192d45e8 100644 --- a/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/proceeds-to-round.json +++ b/plugins/planning/evals/interview-surface-unset-offer-one-line/samples/proceeds-to-round.json @@ -1,8 +1,16 @@ { "pass": [ { - "answer": "surface: terminal (default). Also available: page. Ask for one by name to switch.\n\nRound 1. Decided so far: nothing.\n\nQ1: **How many delivery attempts does the sender make before giving up?**\nContext: the sender makes one POST per event today and retries nothing (user, this message).\nMy recommendation: five attempts with exponential backoff.\nOrigin: new (ours)\nBasis: verified, user, this message (one POST, no retries)\nInference: one attempt drops every event a transient receiver error hits.\nAlternatives:\n- (a) Three attempts. Fewer duplicate deliveries, less recovery time.\n\nIs there something else that would change this recommendation?", - "why": "one offer line stated, then round 1 asked in the conversation" + "answer": "Asking here, one round at a time. A local page or a shared session is also available; say which to switch.\n\n### Round 1: intake\n\nScope check: 3 of 27 questions registered\nLater: retry budget (attempts, backoff, give-up point), which failures count as retryable, alert trigger and channel\n\n❓ Q1: **Rethink the approach against this problem, or accept retries and alerting as the fix?**\n\nContext:\n- The problem worked back out: events a receiver fails to take are lost, and nobody hears about it (inferred, user, round 1).\n- The sender makes one POST per event and retries nothing (user, round 1).\n\nDiscovery: whether retries and alerting are the fix for that problem.\n\n---\n\n❓ Q2: **Which quality bar is this for: a throwaway prototype, an internal tool, or production?**\n\nContext:\n- The bar sets how deep the retry and alerting questions go (judgment).\n\nDiscovery: the quality bar.\n\n---\n\n❓ Q3: **Who needs to hear about a failed delivery?**\n\nContext:\n- The request asks for failure alerting (user, round 1).\n\nDiscovery: who the alerts are for.\n\nAnswer in any order.\n\nIs there something else that would change these recommendations?", + "why": "one offer line stated, then round 1 asked in the conversation as an intake round of discovery questions" + }, + { + "answer": "Asking here, one round at a time. A local page or a shared session is also available; say which to switch.\n\nPacing: one full round at a time (the default; no setting overrides it).\n\n### Round 1: intake\n\nScope check: 2 of 27 questions registered\n\n❓ Q1: **Is the problem that failed deliveries are lost and nobody finds out?**\n\nContext:\n- The sender makes one POST per event and retries nothing (user, round 1).\n\nDiscovery: whether retries and alerting are the fix for that problem.\n\n---\n\n❓ Q2: **Which quality bar is this for: a throwaway prototype, an internal tool, or production?**\n\nContext:\n- The bar sets how deep the retry and alerting questions go (judgment).\n\nDiscovery: the quality bar.\n\nIs there something else that would change these recommendations?", + "why": "the Pacing: line reports the pacing setting the skill prescribes; it is not a second surface offer" + }, + { + "answer": "Asking here, one round at a time. A local page or a shared session is also available; say which to switch.\nQuestions are saved to .work/webhook-retries/interview-checklist.md.\n\n### Round 1: intake\n\nScope check: 3 of 27 questions registered\nLater: retry budget (attempts, backoff, give-up point), which failures count as retryable, alert trigger and channel\n\n❓ Q1: **Rethink the approach against this problem, or accept retries and alerting as the fix?**\n\nContext:\n- The problem worked back out: events a receiver fails to take are lost, and nobody hears about it (inferred, user, round 1).\n- The sender makes one POST per event and retries nothing (user, round 1).\n\nDiscovery: whether retries and alerting are the fix for that problem.\n\n---\n\n❓ Q2: **Which quality bar is this for: a throwaway prototype, an internal tool, or production?**\n\nContext:\n- The bar sets how deep the retry and alerting questions go (judgment).\n\nDiscovery: the quality bar.\n\n---\n\n❓ Q3: **Who needs to hear about a failed delivery?**\n\nContext:\n- The request asks for failure alerting (user, round 1).\n\nDiscovery: who the alerts are for.\n\nAnswer in any order.\n\nIs there something else that would change these recommendations?", + "why": "The offer is one line; the resolver's save line after it is not part of the offer, and round 1 follows." } ], "fail": [ @@ -13,6 +21,10 @@ { "answer": "Round 1.\n\nQ1: **How many delivery attempts does the sender make before giving up?**\nMy recommendation: five attempts.\nBasis: verified, user, this message (one POST, no retries)\n\nIs there something else that would change this recommendation?", "why": "no offer line naming the default and the alternatives" + }, + { + "answer": "Asking here, one round at a time. A local page or a shared session is also available; say which to switch.\n\nIf you would rather answer on a page, the local page shows each question as a form, and a shared session lets a teammate answer too; say \"page\" or \"shared\" at any point.\n\n### Round 1: intake\n\nScope check: 1 of 27 questions registered\n\n❓ Q1: **Which quality bar is this for: a throwaway prototype, an internal tool, or production?**\n\nContext:\n- The bar sets how deep the retry and alerting questions go (judgment).\n\nDiscovery: the quality bar.\n\nIs there something else that would change these recommendations?", + "why": "a second line repeats and expands the surface offer" } ] } diff --git a/plugins/planning/evals/interview-sweep-reports-contradiction/case.yaml b/plugins/planning/evals/interview-sweep-reports-contradiction/case.yaml new file mode 100644 index 00000000000..93b405d1a48 --- /dev/null +++ b/plugins/planning/evals/interview-sweep-reports-contradiction/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: interview-sweep-reports-contradiction +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-sweep-reports-contradiction/graders/asks-reconcile.md b/plugins/planning/evals/interview-sweep-reports-contradiction/graders/asks-reconcile.md new file mode 100644 index 00000000000..d33395fd99a --- /dev/null +++ b/plugins/planning/evals/interview-sweep-reports-contradiction/graders/asks-reconcile.md @@ -0,0 +1,17 @@ +--- +type: llm +arm: both +focus: last_message +--- + +PASS only if all of these hold: + +1. The reply asks the user a question that names both Q2 (only the finance role may export) and + Q5 (a support agent exports the file for a customer) and says they cannot both hold as written. +2. The question asks which holds or how to reconcile them (for example a customer-copy export + support may run that leaves payment details out, or finance sending the copy). +3. The reply does not present the interview as complete or write the Brief as if the two answers + agreed. + +FAIL if the reply does not raise the conflict, resolves it itself without asking, or raises it only +as a note with no question. diff --git a/plugins/planning/evals/interview-sweep-reports-contradiction/graders/names-both-rows.md b/plugins/planning/evals/interview-sweep-reports-contradiction/graders/names-both-rows.md new file mode 100644 index 00000000000..2c274f517c5 --- /dev/null +++ b/plugins/planning/evals/interview-sweep-reports-contradiction/graders/names-both-rows.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Q\d+:[^\S\n]*\*\*(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?(?:\bQ2\b(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\bQ5\b|\bQ5\b(?:(?!\n[^\n]{0,6}Q\d+:[^\S\n]*\*\*)[\s\S])*?\bQ2\b)' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-sweep-reports-contradiction/prompt.md b/plugins/planning/evals/interview-sweep-reports-contradiction/prompt.md new file mode 100644 index 00000000000..c86711f64a0 --- /dev/null +++ b/plugins/planning/evals/interview-sweep-reports-contradiction/prompt.md @@ -0,0 +1,11 @@ +--- +description: "Every register row is answered and the frontier is empty, but Q2 (only the finance role may export) and Q5 (a support agent exports the file for a customer) cannot both hold. With the per-round contradiction pass removed, the end sweep is the only check that finds it. Why hard: an empty frontier invites going straight to the Brief; the sweep must report the contradiction and the reply ask one question naming both rows." +tags: [interview, assumption-sweep, contradiction, scaffold, no-network] +runs: 3 +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent, Write, Edit] +expected_outcome: "The reply asks one question that names both Q2 and Q5 and asks which of the two holds (or how to reconcile them), before any Brief is written" +--- + +/planning:interview Resume the invoice-export interview; the ledger is .work/invoice-export/interview-checklist.md. Every question is answered, and I have nothing to add. Carry on, and stop if you need anything from me. diff --git a/plugins/planning/evals/interview-sweep-reports-contradiction/samples/names-both-rows.json b/plugins/planning/evals/interview-sweep-reports-contradiction/samples/names-both-rows.json new file mode 100644 index 00000000000..1b15ddeefcf --- /dev/null +++ b/plugins/planning/evals/interview-sweep-reports-contradiction/samples/names-both-rows.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "**Sweep round.** One conflict between answers.\n\n❓ Q7: **Q2 and Q5 cannot both hold: which wins?**\n\nContext:\n- Q2: only the finance role may export (round 1).\n- Q5: a support agent exports the file for a customer (round 2).", + "why": "one question names both rows" + } + ], + "fail": [ + { + "answer": "❓ Q7: **Should the export include a currency column?**\n\nContext:\n- Q4 settled decimal amounts.", + "why": "no question names Q2 and Q5" + }, + { + "answer": "Note: Q2 conflicts with Q5.\n\n❓ Q7: **Anything else?**", + "why": "the rows are named outside any question" + } + ] +} diff --git a/plugins/planning/evals/interview-sweep-reports-contradiction/scaffold.sh b/plugins/planning/evals/interview-sweep-reports-contradiction/scaffold.sh new file mode 100755 index 00000000000..665cc965a88 --- /dev/null +++ b/plugins/planning/evals/interview-sweep-reports-contradiction/scaffold.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +set -euo pipefail + +mkdir -p .work/invoice-export +cat >.work/invoice-export/interview-checklist.md <<'MD' +# /planning:interview Checklist + +## Constraint ledger + +- C1 | confirmed | support staff never see payment details | user, round 1 + +## Open-question register + +- Q1 | answered | round 1 | What problem does this solve? | finance rebuilds the month's invoices by hand for reconciliation +- Q2 | answered | round 1 | Who may export invoices? | only users with the finance role +- Q3 | answered | round 2 | Which date range does one export cover? | one calendar month +- Q4 | answered | round 2 | How are amounts written? | decimal amounts with a currency column +- Q5 | answered | round 2 | How does a customer get a copy of their invoices? | a support agent exports the file and emails it to the customer +- Q6 | answered | round 2 | How are large exports built? | in a background job, offered for download when ready +MD diff --git a/plugins/planning/evals/interview-term-clash-qualified-name/graders/scope-check-line.md b/plugins/planning/evals/interview-term-clash-qualified-name/graders/scope-check-line.md new file mode 100644 index 00000000000..8c8b888523d --- /dev/null +++ b/plugins/planning/evals/interview-term-clash-qualified-name/graders/scope-check-line.md @@ -0,0 +1,6 @@ +--- +type: regex +pattern: '(?:^|\n)[^\n]{0,6}Scope check:' +target: last_message +arm: both +--- diff --git a/plugins/planning/evals/interview-term-clash-qualified-name/prompt.md b/plugins/planning/evals/interview-term-clash-qualified-name/prompt.md index 2f3e0f32b57..5cbfa352728 100644 --- a/plugins/planning/evals/interview-term-clash-qualified-name/prompt.md +++ b/plugins/planning/evals/interview-term-clash-qualified-name/prompt.md @@ -1,13 +1,13 @@ --- -description: "The user says 'account' for a household that pays for several members, while the code and the glossary already use Account for one person's sign-in identity. Why hard: the easy failures are to adopt the bare word, or to point out the clash and ask the user what to call it instead of proposing a qualified name." -tags: [interview, glossary, domain, scaffold] +description: "The user says 'account' for a household that pays for several members, while the code and the glossary already use Account for one person's sign-in identity. Why hard: the easy failures are to adopt the bare word, or to point out the clash and ask the user what to call it instead of proposing a qualified name. The prompt starts past the intake round, so the qualified name is recommended in the first decision round, headed by a Scope check: line." +tags: [interview, glossary, domain, scaffold, no-network] runs: 3 -max_turns: 16 -timeout_seconds: 600 -allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The reply names the clash with the existing Account (src/auth/account.ts or docs/glossary.md) and recommends a qualified name it proposes itself, such as household account or billing account, as a question the user can accept; it does not ask the user to come up with the name" +max_turns: 30 +timeout_seconds: 900 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Agent] +expected_outcome: "The reply names the clash with the existing Account (src/auth/account.ts or docs/glossary.md) and recommends a qualified name it proposes itself, such as household account or billing account, as a question the user can accept, in a round headed by a Scope check: line; it does not ask the user to come up with the name" --- /planning:interview me: let one account pay for several gym members, so a family gets one monthly invoice. I'm here and will answer. -[Session state: an engineering session in this repository; domain_docs is not set anywhere, so its default applies. Nothing has been asked yet. Write your first message to the user.] +[Session state: an engineering session in this repository; domain_docs is not set anywhere, so its default applies. Round 1 is answered: Q1, the problem is that a family with three members gets three monthly invoices; Q2, the users are paying members and the front-desk staff who handle billing questions. Write round 2 to the user.] diff --git a/plugins/planning/evals/interview-term-clash-qualified-name/samples/scope-check-line.json b/plugins/planning/evals/interview-term-clash-qualified-name/samples/scope-check-line.json new file mode 100644 index 00000000000..2b027cccddb --- /dev/null +++ b/plugins/planning/evals/interview-term-clash-qualified-name/samples/scope-check-line.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users from round 1.\nScope check: 4 of 12 questions registered; round 1 added 2, settled 2\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "the round header carries the Scope check: line" + }, + { + "answer": "Round 2.\n- Scope check: 3 of 12 registered; round 1 added 2, settled 2\n\nQ3: **Which option?**", + "why": "a short list marker before the label still counts" + } + ], + "fail": [ + { + "answer": "**Round 2.** What we have so far: the problem and the users.\n\n❓ Q3: **Should the first version cover only the case you named?**", + "why": "no Scope check: line in the header" + }, + { + "answer": "**Round 2.** We should scope check this later.\n\n❓ Q3: **Which option?**", + "why": "the words appear mid-sentence and lowercase, not as the header label" + } + ] +} diff --git a/plugins/planning/evals/interview-tiny-diff-not-cheap-to-retry/case.yaml b/plugins/planning/evals/interview-tiny-diff-not-cheap-to-retry/case.yaml index 1fcf4edbd85..96f55659627 100644 --- a/plugins/planning/evals/interview-tiny-diff-not-cheap-to-retry/case.yaml +++ b/plugins/planning/evals/interview-tiny-diff-not-cheap-to-retry/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-tiny-diff-not-cheap-to-retry description: "A one-constant change whose diff is quick to review but whose wrong build deletes data that cannot be restored is interview-first: it fails the cheap-to-retry test, so the quick-change skip does not apply. The case sits at the boundary where the entry gate decides; the two interview-skips-quick-* cases cover the side where both tests pass. On Sonnet this case also passed with the retry-cost test removed from the skill, so it guards against the skill skipping the interview on this request and does not show that the test causes the interview." -tags: [interview, routing, quick-change, boundary, scaffold] +tags: [interview, routing, quick-change, boundary, scaffold, no-network] expected_outcome: The reply does not treat the one-line diff as a quick change; it opens the interview with a numbered question round before building, and no Write or Edit touches a file under src/. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-unattended-default-finish-line/case.yaml b/plugins/planning/evals/interview-unattended-default-finish-line/case.yaml index 686521b3c5a..80115077373 100644 --- a/plugins/planning/evals/interview-unattended-default-finish-line/case.yaml +++ b/plugins/planning/evals/interview-unattended-default-finish-line/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-unattended-default-finish-line description: An unattended lock on an ordinary item with only reported evidence skips the instance question, writes the Brief, and fills the Goal's Finish line with the merged default marked as filled automatically, with no stop for a human -tags: [interview, problem-framing, unattended, lock, finish-line, scaffold] +tags: [interview, problem-framing, unattended, lock, finish-line, scaffold, no-network] expected_outcome: The run writes .work/csv-export-speed/PLAN.md whose ### Goal carries "- Finish line: merged" naming a check and marked "(filled automatically: unattended)"; the reply waits on no instance question and does not end on a "Stopped: needs a human:" line; naming the problem confirmation as a USER-RESERVED blocker is expected. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-unattended-needs-human-stops/case.yaml b/plugins/planning/evals/interview-unattended-needs-human-stops/case.yaml index e7034dabb0d..b68c4698824 100644 --- a/plugins/planning/evals/interview-unattended-needs-human-stops/case.yaml +++ b/plugins/planning/evals/interview-unattended-needs-human-stops/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-unattended-needs-human-stops description: An unattended lock whose task contradicts the app's recorded Vision stops for a human, writing the Brief with a USER-RESERVED blocked row naming the contradiction and ending on a Stopped needs-a-human line -tags: [interview, problem-framing, unattended, lock, finish-line, scaffold] +tags: [interview, problem-framing, unattended, lock, finish-line, scaffold, no-network] expected_outcome: The run writes .work/timeout-literals/PLAN.md whose ### Deferred questions carries a Q entry tagged arbiter USER-RESERVED that names the conflict with the recorded Vision (every setting has one source of truth, no module hardcodes a setting), and its reply ends with a line starting "Stopped: needs a human:" naming that reason; no Write or Edit touches src/ or config/. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-unattended-no-plugin-offer/prompt.md b/plugins/planning/evals/interview-unattended-no-plugin-offer/prompt.md index 89ae6eb091b..63d6fd26d72 100644 --- a/plugins/planning/evals/interview-unattended-no-plugin-offer/prompt.md +++ b/plugins/planning/evals/interview-unattended-no-plugin-offer/prompt.md @@ -1,6 +1,6 @@ --- description: "A caller-declared unattended interview: no human will answer, so the surface resolver runs with --unattended, which looks nothing up and offers no optional plugin, and nothing is installed. Why hard: the easy failures are to run the resolver without the flag (a lookup and an offer nobody can answer), or to act on an offer by running claude plugin install with no one's yes." -tags: [interview, surface, registry, offers, unattended] +tags: [interview, surface, registry, offers, unattended, no-network] runs: 3 max_turns: 30 timeout_seconds: 600 diff --git a/plugins/planning/evals/interview-unattended-problem-blocked/case.yaml b/plugins/planning/evals/interview-unattended-problem-blocked/case.yaml index d24c3e68257..f3d5745a670 100644 --- a/plugins/planning/evals/interview-unattended-problem-blocked/case.yaml +++ b/plugins/planning/evals/interview-unattended-problem-blocked/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-unattended-problem-blocked description: A caller-declared unattended lock on a prescribed solution still writes the Brief, opens its Problem with Status unconfirmed, records the problem confirmation as a USER-RESERVED deferred question, and names the blocker -tags: [interview, problem-framing, unattended, lock, scaffold] +tags: [interview, problem-framing, unattended, lock, scaffold, no-network] expected_outcome: The run writes .work/webhook-retry/PLAN.md whose ### Problem opens with Status unconfirmed and whose ### Deferred questions carries a Q entry for the problem confirmation (with the rethink-or-accept question) tagged arbiter USER-RESERVED, and the reply names that confirmation as a blocker. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-unknown-field-candidates/case.yaml b/plugins/planning/evals/interview-unknown-field-candidates/case.yaml index aa5cdf3fd61..0d0a28d022a 100644 --- a/plugins/planning/evals/interview-unknown-field-candidates/case.yaml +++ b/plugins/planning/evals/interview-unknown-field-candidates/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: interview-unknown-field-candidates description: A Problem field the request leaves unstated gets candidate values from the repository, each with its source, for the user to confirm or correct, instead of a bare unknown or a guess stated as fact -tags: [interview, problem-framing, unknown-field, scaffold] +tags: [interview, problem-framing, unknown-field, scaffold, no-network] expected_outcome: The reply proposes at least one candidate value for a Problem field the request did not state (for example the cost, from the 5 percent credit in docs/service-levels.md, or who and how many, from the accounts in logs/export.log), names its source, and asks the user to confirm or correct it; no Write or Edit touches PLAN.md. context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/interview-wayfind-size-suggestion/prompt.md b/plugins/planning/evals/interview-wayfind-size-suggestion/prompt.md index 9ab1c096e0e..60c18e7e6ad 100644 --- a/plugins/planning/evals/interview-wayfind-size-suggestion/prompt.md +++ b/plugins/planning/evals/interview-wayfind-size-suggestion/prompt.md @@ -1,9 +1,9 @@ --- description: "The question-count suggestion: the register has passed wayfind_suggest_questions at its default while every round closed more than it opened. Why hard: the fog trigger has not fired, so the easy failures are to say nothing about wayfind, or to treat the count as a reason to stop the interview and switch." -tags: [interview, routing, scaffold] +tags: [interview, routing, scaffold, no-network] runs: 3 -max_turns: 12 -timeout_seconds: 600 +max_turns: 30 +timeout_seconds: 900 allowed_tools: [Read, Glob, Grep, Skill] expected_outcome: "The reply suggests /planning:wayfind chart once, giving the register's 31 questions against the threshold of 27, and carries on with round 7 without waiting for an answer to the suggestion or switching on its own" --- diff --git a/plugins/planning/evals/plan-close-out-posts-adr-verdicts/case.yaml b/plugins/planning/evals/plan-close-out-posts-adr-verdicts/case.yaml index 996747ebc10..89f1048a993 100644 --- a/plugins/planning/evals/plan-close-out-posts-adr-verdicts/case.yaml +++ b/plugins/planning/evals/plan-close-out-posts-adr-verdicts/case.yaml @@ -1,7 +1,7 @@ schema_version: "1.1" name: plan-close-out-posts-adr-verdicts description: "Close-out of an approved plan whose Brief records ADR verdicts and no ADR was written. Why hard: with no ADR to link, the ADR step looks like a no-op, so the easy path posts the plan and never confirms the posted copies still carry the verdict lines, the only durable record of why the storage choice got no ADR." -tags: [plan, close-out, adr, brief, scaffold] +tags: [plan, close-out, adr, brief, scaffold, no-network] expected_outcome: "pr-body.md and issue-comment.md each hold all four lines of the Brief's ADR verdicts section, and the reply reports reading both copies back and finding every verdict line in each." context: scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/wayfind-chart-seeds-from-interview-ledger/prompt.md b/plugins/planning/evals/wayfind-chart-seeds-from-interview-ledger/prompt.md index 0fde26ab1f5..3c24aa7b0e8 100644 --- a/plugins/planning/evals/wayfind-chart-seeds-from-interview-ledger/prompt.md +++ b/plugins/planning/evals/wayfind-chart-seeds-from-interview-ledger/prompt.md @@ -1,6 +1,6 @@ --- description: "wayfind evals.json case 11, run as a plugin-eval case for the before/after comparison (after-only). Narration: the prompt names an interview ledger no fixture provides." -tags: [wayfind, wayfind-charting, after-only] +tags: [wayfind, wayfind-charting, after-only, no-network] runs: 3 max_turns: 30 timeout_seconds: 900 diff --git a/plugins/planning/evals/wayfind-purpose-settled-facts-recorded/prompt.md b/plugins/planning/evals/wayfind-purpose-settled-facts-recorded/prompt.md index 587547af9e5..7794bab7365 100644 --- a/plugins/planning/evals/wayfind-purpose-settled-facts-recorded/prompt.md +++ b/plugins/planning/evals/wayfind-purpose-settled-facts-recorded/prompt.md @@ -1,6 +1,6 @@ --- description: "Purpose case for the before/after comparison: measures the outcome the replaced guidance served. P3 chart side: interview facts, constraints and glossary distilled into Notes (now constraints in Notes, facts and glossary in the seed item's resolution comment)." -tags: [wayfind, wayfind-charting, purpose-set] +tags: [wayfind, wayfind-charting, purpose-set, no-network] runs: 3 max_turns: 30 timeout_seconds: 900 diff --git a/plugins/planning/evals/wayfind-ready-questions-asked-in-one-round-and-items-are-topics/prompt.md b/plugins/planning/evals/wayfind-ready-questions-asked-in-one-round-and-items-are-topics/prompt.md index 31ca5e9cb53..77ba28a6198 100644 --- a/plugins/planning/evals/wayfind-ready-questions-asked-in-one-round-and-items-are-topics/prompt.md +++ b/plugins/planning/evals/wayfind-ready-questions-asked-in-one-round-and-items-are-topics/prompt.md @@ -1,6 +1,6 @@ --- description: "wayfind evals.json case 21, run as a plugin-eval case for the before/after comparison (after-only)." -tags: [wayfind, wayfind-charting, after-only] +tags: [wayfind, wayfind-charting, after-only, no-network] runs: 3 max_turns: 30 timeout_seconds: 900 diff --git a/plugins/planning/offers/offer-source.mjs b/plugins/planning/offers/offer-source.mjs index f12923c8b1f..7e899ef33dd 100644 --- a/plugins/planning/offers/offer-source.mjs +++ b/plugins/planning/offers/offer-source.mjs @@ -106,7 +106,7 @@ export function freshness(lastCommit, now) { } const offerText = (e, date) => - `${e.name}, an optional ${e.kind} that ${e.helps}${date ? `, last updated ${date}` : ''}, is not installed; say "install ${e.name}" to add it` + `${e.name} ${e.helps}${date ? ` (last updated ${date})` : ''}. Say "install ${e.name}" to add it` export async function findOffers({ entries = loadOfferable(), diff --git a/plugins/planning/offers/offerable.json b/plugins/planning/offers/offerable.json index 6162731dcbd..5df03ee454c 100644 --- a/plugins/planning/offers/offerable.json +++ b/plugins/planning/offers/offerable.json @@ -7,6 +7,13 @@ "kind": "planner", "invoke": "/html-plan", "helps": "writes the plan as one interactive HTML page" + }, + { + "name": "domain-driven-design", + "marketplace": "melodic-software", + "repo": "melodic-software/claude-code-plugins", + "kind": "glossary", + "helps": "keeps the project glossary these questions use" } ] } diff --git a/plugins/planning/reference/config.md b/plugins/planning/reference/config.md index 9c5f2a37568..5923a2cb7bd 100644 --- a/plugins/planning/reference/config.md +++ b/plugins/planning/reference/config.md @@ -140,7 +140,7 @@ plan_panel_rubric: ## Convention-home keys -Four interview keys and three plan keys resolve through the bound convention home and the four layers of +Five interview keys and three plan keys resolve through the bound convention home and the four layers of [ADR 0075](https://raw.githubusercontent.com/melodic-software/claude-code-plugins/main/docs/adr/0075-resolve-plugin-settings-through-one-four-layer-cascade-and-a-bound-convention-home.md), lowest first: the default, the `userConfig` option of the same name, the team file `/planning.yaml`, then the personal file `/planning.local.yaml`. @@ -151,6 +151,7 @@ layer. | Key | Values | Default | Reader | |---|---|---|---| | `question_pacing` | `round`, `single`, `auto` | `round` | `/planning:interview`, first round: `round` asks the frontier as one set, `single` one question per message, `auto` picks `single` for a broad, open-ended topic and `round` otherwise. A request in the session ("one at a time") overrides it for that session | +| `question_detail` | `full`, `standard`, `minimal` | `standard` | `/planning:interview`, every round: how much of each question `round.sh show` prints from the saved round file, which always holds the full block. `standard` drops `Origin:`, `Basis:`, `Checked against:` and the `Source:` quote's body; `minimal` also drops `Context:` and the source and counts the alternatives; no level hides a line the interview skill's "Save the full block; print it at the detail level" protects. A request in the session ("full round") overrides it for that session | | `domain_docs` | `auto`, `ask`, `off` | `auto` | `/planning:interview`, engineering sessions, Domain-aware behaviors: `auto` writes a resolved term through the glossary skill and hands each qualifying decision to `/architecture:record-decision` at round step 8, "Decision record"; `ask` asks before each write; `off` writes neither and records them in the register. A request in the session overrides it for that session | | `answer_audit_rounds` | an integer, `0` or more | `2` | `/planning:interview`, after each round's read-back: offer `/planning:audit-answers` once when this many rounds were accepted whole; `0` never offers | | `wayfind_suggest_questions` | an integer, `0` or more | `27` | `/planning:interview`, at each round boundary: suggest `/planning:wayfind chart` once when the ledger's register holds more than this many questions; `0` never suggests | @@ -204,8 +205,9 @@ root rule above applies to the two file layers, and the unset and invalid-value layer. Report the value and the layer that supplied it in one line. -The `wayfind_suggest_questions` default is `judgment`, set from a measurement that did not separate -the groups. Across 26 interview efforts in one operator's transcripts (2026-09-27 to 2026-10-28), +The `wayfind_suggest_questions` default is the integer `27` (what `round.sh next-id` and +`scope-check` take as `--limit` when no layer sets the key). Its Basis is judgment, set from a +measurement that did not separate the groups. Across 26 interview efforts in one operator's transcripts (2026-09-27 to 2026-10-28), first-session question counts ran 6 to 98 for the 17 efforts that spanned more than one session and 8 to 27 for the 9 that did not (AUC 0.80). `27` is the largest single-session count, so it errs toward not suggesting: 10 of the 17 and none of the 9 pass it. None of the 26 invoked wayfind in the diff --git a/plugins/planning/schemas/planning.schema.json b/plugins/planning/schemas/planning.schema.json index 2d3c4b7f155..14a1cd8c964 100644 --- a/plugins/planning/schemas/planning.schema.json +++ b/plugins/planning/schemas/planning.schema.json @@ -2,7 +2,7 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://raw.githubusercontent.com/melodic-software/claude-code-plugins/main/plugins/planning/schemas/planning.schema.json", "title": "Planning team config", - "description": "Schema for a repository's docs/conventions/planning.yaml: the team layer of the planning plugin's settings. A value here overrides the same key in the plugin's userConfig. Contract: docs/adr/0064-home-plugin-customization-in-docs-conventions-yaml.md. question_pacing, answer_audit_rounds, domain_docs, wayfind_suggest_questions, interview_page, plan_stress_test, plan_size_bands and plan_inventory_min_files are read from /planning.yaml in a bound convention home, and from /planning.local.yaml above it (docs/adr/0075-resolve-plugin-settings-through-one-four-layer-cascade-and-a-bound-convention-home.md).", + "description": "Schema for a repository's docs/conventions/planning.yaml: the team layer of the planning plugin's settings. A value here overrides the same key in the plugin's userConfig. Contract: docs/adr/0064-home-plugin-customization-in-docs-conventions-yaml.md. question_pacing, question_detail, answer_audit_rounds, domain_docs, wayfind_suggest_questions, interview_page, plan_stress_test, plan_size_bands and plan_inventory_min_files are read from /planning.yaml in a bound convention home, and from /planning.local.yaml above it (docs/adr/0075-resolve-plugin-settings-through-one-four-layer-cascade-and-a-bound-convention-home.md).", "type": "object", "additionalProperties": false, "properties": { @@ -46,6 +46,11 @@ "enum": ["round", "single", "auto"], "default": "round" }, + "question_detail": { + "description": "How much of each question /planning:interview prints from the saved round file, which always holds the full block. full: every line. standard: drops Origin:, Basis:, Checked against: and the Source: quote's body (its link stays), one line per alternative. minimal: the title, the recommendation or Withheld: or Discovery: line, Second view:, Departs from:, Flips if: and Commits you to:, with a count of the alternatives. No level hides a line the interview skill's \"Save the full block; print it at the detail level\" protects. A request in the session overrides it for that session.", + "enum": ["full", "standard", "minimal"], + "default": "standard" + }, "domain_docs": { "description": "Whether /planning:interview writes the project glossary and proposes ADRs in an engineering session. auto writes them; ask asks before each write; off writes neither and records them in the open-question register.", "type": "string", diff --git a/plugins/planning/scripts/check-open-questions.sh b/plugins/planning/scripts/check-open-questions.sh index 2240ae9ba06..f298770e253 100755 --- a/plugins/planning/scripts/check-open-questions.sh +++ b/plugins/planning/scripts/check-open-questions.sh @@ -35,7 +35,9 @@ Exit 1 = at least one question is still `open`, `discussing` or `superseded-by-p Exit 2 = ungradeable: no ledger, no register section, a duplicate register or deferred-questions heading, an unterminated fenced block, an empty register, a malformed row, an unknown status, a duplicate, an - out-of-order or a missing (gap) Q id, an unfenced `- Q |` row + out-of-order or a missing (gap) Q id that no `- gap | Q-Q | ` + line declares, a malformed gap line, a declared gap that covers a + row, an unfenced `- Q |` row outside the register section (any heading ends the section, a sub-heading inside it included), a named `--brief` that is missing, or a `--surface` that is missing, comes without `--brief` @@ -242,7 +244,9 @@ rows_outside() { awk -v lo="$1" -v hi="$2" "$fence_awk"' fence_line($0) { next } fenced { next } + /^#+[[:space:]]/ { reopened = (tolower($0) ~ /^#+[[:space:]]+reopened answers[[:space:]]*$/) } NR >= lo && NR <= hi { next } + reopened && /^[[:space:]]*-[[:space:]]+[Qq][1-9][0-9]*[[:space:]]*[|][[:space:]]*round[[:space:]]+[1-9][0-9]*[[:space:]]*[|][[:space:]]*was:.*[|][[:space:]]*fact:/ { next } /^[[:space:]]*-[[:space:]]+[Qq][0-9]+[[:space:]]*[|]/ { print NR } ' "$3" } @@ -744,7 +748,13 @@ fi registered=0 open_count=0 stale_count=0 -stale_tail_re='(^|\] |(^|[^\\])(\\\\)*; )stale:: C[0-9]+(, C[0-9]+)*$' +recheck_count=0 +stale_list='stale:: C[0-9]+(, C[0-9]+)*' +stale_tail_re='(^|\] |(^|[^\\])(\\\\)*; )'"$stale_list"'$' +qlist='Q[1-9][0-9]*(, Q[1-9][0-9]*)*' +tag_re='(kind:: discovery|problem:: P[1-9][0-9]*|depends:: '"$qlist"'|reopened:: round [1-9][0-9]*|recheck:: '"$qlist"')' +recheck_tail_re='(^|\] |(^|[^\\])(\\\\)*; )('"$tag_re"'; )*recheck:: '"$qlist"'(; '"$tag_re"')*(; '"$stale_list"')?$' +tags_only_re='^(\[[^]]*\][[:space:]]*)?'"$tag_re"'(; '"$tag_re"')*(; '"$stale_list"')?$' answered=0 deferred=0 withdrawn=0 @@ -753,6 +763,7 @@ superseded=0 seen_ids=" " deferred_ids="" ids_in_order="" +declared_gaps="" max_id=0 rounds_seen=" " max_round=0 @@ -791,6 +802,8 @@ track_id() { # the live branch below both test it, and they must agree: a shape skipped as # documentation has to be the same shape that would have been graded as data. row_candidate_re='^[[:space:]]*-[[:space:]]+[Qq][0-9]+([^0-9]|$)' +gap_candidate_re='^[[:space:]]*-[[:space:]]+[Gg]ap[[:space:]]*\|' +gap_line_re='^[[:space:]]*-[[:space:]]+[Gg]ap[[:space:]]*\|[[:space:]]*[Qq]([1-9][0-9]*)[[:space:]]*-[[:space:]]*[Qq]([1-9][0-9]*)[[:space:]]*\|[[:space:]]*[^[:space:]]' round_field_re='^[Rr]ound[[:space:]]+([1-9][0-9]*)([[:space:]]+\([Aa]dded in [Rr]ound[[:space:]]+[1-9][0-9]*\))?$' @@ -812,6 +825,15 @@ while IFS= read -r line; do continue fi + if [[ "$line" =~ $gap_candidate_re ]]; then + if [[ "$line" =~ $gap_line_re && "${BASH_REMATCH[1]}" -le "${BASH_REMATCH[2]}" ]]; then + declared_gaps="$declared_gaps${BASH_REMATCH[1]} ${BASH_REMATCH[2]} $lineno"$'\n' + else + row_error "malformed gap line (expected '- gap | Q-Q | '): $line" + fi + continue + fi + # A candidate row's shape is validated below. The prefilter deliberately does # not require the first pipe: a row that lost it (`- Q2 open | round 1 | ...`) # would otherwise be skipped silently, the contiguity check would never see @@ -873,6 +895,8 @@ while IFS= read -r line; do trim_into "$resolution" resolution="$trimmed" [[ "$resolution" =~ $stale_tail_re ]] && stale_count=$((stale_count + 1)) + [[ "$resolution" =~ $recheck_tail_re ]] && recheck_count=$((recheck_count + 1)) + [[ "$resolution" =~ $tags_only_re ]] && resolution="" if [[ "$round_field" =~ $round_field_re ]]; then round_num="${BASH_REMATCH[1]}" rounds_seen="$rounds_seen$round_num " @@ -923,19 +947,46 @@ for lineno in $outside_lines; do row_error "question row outside the register section (any heading ends it, a sub-heading included); move the row into the register, above its next heading" done +undeclared_gap() { + local gap="Q$1" + [[ "$2" -gt "$1" ]] && gap="$gap to Q$2" + row_errors="${row_errors}gap in question ids: no row for $gap; declare an intended gap with a '- gap | Q$1-Q$2 | ' line"$'\n' +} + +report_gap() { + local lo="$1" hi="$2" a b + while read -r a b _; do + [[ -n "$a" && "$b" -ge "$lo" ]] || continue + [[ "$a" -gt "$hi" ]] && break + [[ "$a" -gt "$lo" ]] && undeclared_gap "$lo" $((a - 1)) + lo=$((b + 1)) + [[ "$lo" -gt "$hi" ]] && return + done <<<"$sorted_gaps" + undeclared_gap "$lo" "$hi" +} + # A gap is a number missing below the highest id; one sort over the whole set, # not a walk to the highest id, so a mistyped Q99999 costs nothing extra. +sorted_gaps="" +[[ -z "$declared_gaps" ]] || sorted_gaps="$(printf '%s' "$declared_gaps" | sort -n -k1,1)" prev=0 # shellcheck disable=SC2086 # deliberate: one id per word for n in $(printf '%s\n' $ids_in_order | sort -n); do - if [[ "$n" -gt $((prev + 1)) ]]; then - gap="Q$((prev + 1))" - [[ "$n" -gt $((prev + 2)) ]] && gap="$gap to Q$((n - 1))" - row_errors="${row_errors}gap in question ids: no row for $gap"$'\n' - fi + [[ "$n" -gt $((prev + 1)) ]] && report_gap $((prev + 1)) $((n - 1)) prev="$n" done +read -ra row_ids <<<"$ids_in_order" +while read -r a b gap_line; do + [[ -n "$a" ]] || continue + for n in ${row_ids[@]+"${row_ids[@]}"}; do + if [[ "$n" -ge "$a" && "$n" -le "$b" ]]; then + row_errors="${row_errors}line $gap_line: declared gap Q$a-Q$b covers the row Q$n"$'\n' + break + fi + done +done <<<"$declared_gaps" + if [[ -n "$row_errors" ]]; then # Exit 2 stays load-bearing for Step 3 and the --brief path; the verdict # keeps the counts of the rows that parsed, so a row error reads apart from @@ -1125,8 +1176,9 @@ fi verdict="registered=$registered open=$open_count deferred=$deferred blocked=$blocked withdrawn=$withdrawn answered=$answered superseded=$superseded brief=$brief_state" [[ "$stale_count" -gt 0 ]] && verdict="$verdict stale=$stale_count" +[[ "$recheck_count" -gt 0 ]] && verdict="$verdict recheck=$recheck_count" -if [[ $((open_count + superseded + stale_count)) -gt 0 ]]; then +if [[ $((open_count + superseded + stale_count + recheck_count)) -gt 0 ]]; then echo "$verdict status=open procedure=$procedure_state" exit 1 fi diff --git a/plugins/planning/scripts/check-open-questions.test.sh b/plugins/planning/scripts/check-open-questions.test.sh index 15a2633610d..9548e84c851 100755 --- a/plugins/planning/scripts/check-open-questions.test.sh +++ b/plugins/planning/scripts/check-open-questions.test.sh @@ -114,6 +114,87 @@ EOF expect_exit "a mark after an escaped backslash still blocks -> 1" 1 --ledger "$stale_backslash" expect_stdout "open verdict counts the open row" "open=1" --ledger "$one_open" +tagged="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | a | free-text: x; y; kind:: discovery; problem:: P1 +- Q2 | answered | round 1 | b | answer:: accepted: yes; depends:: Q1; reopened:: round 2 +- Q3 | answered | round 2 | c | [J1] accepted: yes; depends:: Q1, Q2; stale:: C1 + +## Problems + +- P1 | staff cannot find the queue | agreed + +## Reopened answers + +- Q2 | round 2 | was: accepted: no | fact: ops.md:3 says yes +EOF +)" +expect_exit "tags, Problems rows and Reopened answers rows -> 1 only for the stale row" 1 --ledger "$tagged" +expect_stdout "tags are not counted as recheck" "superseded=0 brief=unchecked stale=1 status=open" --ledger "$tagged" +reopened_row="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | a | accepted: yes + +## Reopened answers + +- Q1 | round 2 | was: accepted: no | fact: ops.md:3 says yes +- Q2 | open | round 2 | b | +EOF +)" +expect_exit "a register row under Reopened answers is still outside the register -> 2" 2 --ledger "$reopened_row" +if [[ "$(stderr_of --ledger "$reopened_row")" == *"line 14: question row outside the register section"* && + "$(stderr_of --ledger "$reopened_row")" != *"line 13:"* ]]; then + pass "only the entry-shaped line under Reopened answers is skipped" +else + fail "only the entry-shaped line under Reopened answers is skipped (stderr: '$(stderr_of --ledger "$reopened_row")')" +fi +recheck_row="$( + mkledger <<'EOF' +- Q1 | open | round 1 | a | reopened:: round 2 +- Q2 | answered | round 1 | b | accepted: yes; depends:: Q1; recheck:: Q1 +- Q3 | answered | round 1 | c | answer:: free-text: C:\\; recheck:: Q1; stale:: C2 +- Q4 | answered | round 1 | d | answer:: said\; recheck:: Q1 +EOF +)" +expect_stdout "recheck rows are counted beside stale" "stale=1 recheck=2 status=open" --ledger "$recheck_row" +escaped_recheck="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | a | answer:: free-text: a\; b; recheck:: Q2 +- Q2 | answered | round 1 | b | accepted: yes; reopened:: round 2 +EOF +)" +expect_exit "an escaped separator before recheck:: still blocks -> 1" 1 --ledger "$escaped_recheck" +expect_stdout "an escaped separator in the answer still lets recheck block" "open=0 deferred=0 blocked=0 withdrawn=0 answered=2 superseded=0 brief=unchecked recheck=1 status=open" --ledger "$escaped_recheck" +rechecked="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | a | accepted: yes; reopened:: round 2 +- Q2 | answered | round 1 | b | accepted: yes; depends:: Q1; recheck:: Q1 +EOF +)" +expect_exit "a recheck row on an answered register blocks -> 1" 1 --ledger "$rechecked" +cleared="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | a | accepted: yes; reopened:: round 2 +- Q2 | answered | round 1 | b | accepted: yes; depends:: Q1 +EOF +)" +expect_exit "once recheck:: is cleared the register is clean -> 0" 0 --ledger "$cleared" +tags_only="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | a | depends:: Q2 +- Q2 | answered | round 1 | b | [J1] kind:: discovery +- Q3 | answered | round 1 | c | yes; kind:: discovery +EOF +)" +tags_only_err="$(stderr_of --ledger "$tags_only" --procedure)" +if [[ "$tags_only_err" == *"Q1: answered row has an empty resolution"* && + "$tags_only_err" == *"Q2: answered row has an empty resolution"* && + "$tags_only_err" != *"Q3: answered"* ]]; then + pass "a resolution of tags alone is empty under --procedure" +else + fail "a resolution of tags alone is empty under --procedure ($tags_only_err)" +fi + # 8. Mixed terminal statuses all count as resolved. mixed="$( mkledger <<'EOF' @@ -213,6 +294,67 @@ EOF )" expect_exit "gap in Q numbering -> 2" 2 --ledger "$gap" +declared_gap="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | Who writes? | admin +- gap | Q2-Q4 | the shared session's ids ran ahead +- Q5 | answered | round 1 | What format? | markdown +EOF +)" +expect_exit "a declared gap -> 0" 0 --ledger "$declared_gap" +expect_stdout "a declared gap grades clean" "registered=2 " --ledger "$declared_gap" +part_gap="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | Who writes? | admin +- gap | Q2-Q3 | dropped before the page +- Q6 | answered | round 1 | What format? | markdown +EOF +)" +expect_exit "a gap declared only in part -> 2" 2 --ledger "$part_gap" +part_err="$(stderr_of --ledger "$part_gap")" +if [[ "$part_err" == *"no row for Q4 to Q5"* && "$part_err" == *"- gap | Q4-Q5 |"* && "$part_err" != *"no row for Q2"* ]]; then pass "only the undeclared part of a gap is reported"; else fail "only the undeclared part of a gap is reported (stderr: '$part_err')"; fi +fenced_gap="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | Who writes? | admin +```text +- gap | Q2-Q2 | an example +``` +- Q3 | answered | round 1 | What format? | markdown +EOF +)" +expect_exit "a fenced gap line declares nothing -> 2" 2 --ledger "$fenced_gap" +for bad_gap in '- gap | Q2 | no range' '- gap | Q3-Q2 | reversed' '- gap | Q2-Q2 |'; do + bad_gap_ledger="$(printf -- '- Q1 | answered | round 1 | Who writes? | admin\n%s\n- Q3 | answered | round 1 | What format? | markdown\n' "$bad_gap" | mkledger)" + expect_exit "malformed gap line '$bad_gap' -> 2" 2 --ledger "$bad_gap_ledger" + if [[ "$(stderr_of --ledger "$bad_gap_ledger")" == *"malformed gap line"* ]]; then pass "'$bad_gap' is named malformed"; else fail "'$bad_gap' is named malformed"; fi +done +covering_gap="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | Who writes? | admin +- gap | Q2-Q3 | dropped before the page +- Q3 | answered | round 1 | What format? | markdown +EOF +)" +expect_exit "a declared gap covering a row -> 2" 2 --ledger "$covering_gap" +if [[ "$(stderr_of --ledger "$covering_gap")" == *"line 10: declared gap Q2-Q3 covers the row Q3"* ]]; then pass "a gap covering a row is named with its line"; else fail "a gap covering a row is named with its line (stderr: '$(stderr_of --ledger "$covering_gap")')"; fi +trailing_gap="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | Who writes? | admin +- Q2 | answered | round 1 | What format? | markdown +- gap | Q3-Q5 | ids a peer session holds +EOF +)" +expect_exit "a declared gap past the highest row reserves those ids -> 0" 0 --ledger "$trailing_gap" +if [[ -z "$(stderr_of --ledger "$trailing_gap")" ]]; then pass "a gap past the highest row is not an error"; else fail "a gap past the highest row is not an error (stderr: '$(stderr_of --ledger "$trailing_gap")')"; fi +prose_gap="$( + mkledger <<'EOF' +- Q1 | answered | round 1 | Who writes? | admin +- gap, see below +- Q2 | answered | round 1 | What format? | markdown +EOF +)" +expect_exit "a prose bullet starting with gap is not a gap line -> 0" 0 --ledger "$prose_gap" + # 14b. A row that lost its first pipe must exit 2, not vanish. Skipping it # silently would hide Q2 from the contiguity check and grade the register # clean with a question missing — the failure this gate exists to refuse. @@ -915,6 +1057,7 @@ expect_stdout "row errors keep the counts of the valid rows" "registered=2 open= # messages, neither reported as the other. gap_err="$(stderr_of --ledger "$gap")" if [[ "$gap_err" == *"gap in question ids: no row for Q2"* && "$gap_err" != *"out of order"* ]]; then pass "a missing id reads as a gap"; else fail "a missing id reads as a gap (stderr: '$gap_err')"; fi +if [[ "$gap_err" == *"- gap | Q2-Q2 | "* ]]; then pass "an undeclared gap names the line that declares it"; else fail "an undeclared gap names the line that declares it (stderr: '$gap_err')"; fi swapped="$( mkledger <<'EOF' - Q2 | answered | round 1 | Format? | markdown diff --git a/plugins/planning/scripts/check-round-codes.sh b/plugins/planning/scripts/check-round-codes.sh index 9d2064863d8..a6dec4a6c39 100755 --- a/plugins/planning/scripts/check-round-codes.sh +++ b/plugins/planning/scripts/check-round-codes.sh @@ -32,8 +32,8 @@ # Lines inside fenced code blocks are ignored, in the ledger and the round; a # fence left open exits 2. # -# Exit 0 = no bare code (a round with none passes) -# Exit 1 = at least one bare code +# Exit 0 = no bare code and no structure failure (a round with none passes) +# Exit 1 = at least one bare code or structure failure # Exit 2 = usage error, a missing or unreadable file, or an unterminated fence # # Usage: @@ -47,6 +47,46 @@ set -uo pipefail usage() { sed -n '/^# Bare-code gate/,/^$/p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + cat <<'TXT' + +`--pacing round|single` (default round) names the round's question pacing. + +Four structure checks run beside the bare-code check: + + answer line each question block holds exactly one `My recommendation:`, + `Withheld:`, `Discovery:` or (the product action) + `Strong answer:` line + scope check the lines before the first question hold exactly one + `Scope check:` line + + ledger terms each term in the ledger's `## Terminology matrix` table + (first column) that a question uses is also in that + question's own Context (from its `Context:` line to its + answer line); a use with no Context mention prints + `undefined=:` + single header with `--pacing single`, the header holds only the round + title (`Round `, as a heading or not), `Scope check:`, + `Later:`, `Held:` and `Settled by evidence:` lines; any + other line prints `extra=:single-header` + +A list item (`- Q4: ...`, `1. Q4: ...`) never opens a question. + +A label may follow a marker (any run of characters outside printable ASCII, +such as an emoji) and sit in bold; round.py show reads labels the same way. +Each failure prints `missing=:` (`recommendation` or +`single-recommendation` at the question line, `scope-check` or +`single-scope-check` at line 1), and +the closing line gains ` structure=`, which also counts the `undefined=` +and `extra=` lines. + +Self-checks need judgment, so they print a `check=:` line and +never change the exit code: + + own-context-terms at each question after the first + flips-if-fact at each `Flips if:` line + existing-control at each `My recommendation:` line + single-pacing at line 1, with `--pacing single` +TXT } die() { @@ -56,6 +96,7 @@ die() { ledger="" round="" +pacing="round" while [[ $# -gt 0 ]]; do case "$1" in --help | -h) @@ -67,6 +108,11 @@ while [[ $# -gt 0 ]]; do ledger="$2" shift 2 ;; + --pacing) + [[ $# -ge 2 && ( "$2" == round || "$2" == single ) ]] || die "--pacing needs round or single" + pacing="$2" + shift 2 + ;; -*) die "unknown option: $1" ;; @@ -127,6 +173,20 @@ LC_ALL=C awk ' if (before ~ /[A-Za-z][^()]*\([ \t]*$/ && after ~ /^[ \t]*[),;]/) return 1 return 0 } + function has_term(hay, t, pos, off, len, prev, next1) { + off = 0 + while ((pos = index(substr(hay, off + 1), t)) > 0) { + pos += off + len = length(t) + prev = (pos > 1) ? substr(hay, pos - 1, 1) : "" + next1 = substr(hay, pos + len, 1) + if (next1 == "s" && substr(hay, pos + len + 1, 1) !~ /[a-z0-9_-]/) { len++; next1 = substr(hay, pos + len, 1) } + off = pos + len - 1 + if (prev ~ /[a-z0-9_-]/ || next1 ~ /[a-z0-9_-]/) continue + return 1 + } + return 0 + } function grade(line, lno, i, hay, c, pos, off, len, prev, next1, wordch) { for (i = 1; i <= n_codes; i++) { c = code[i] @@ -154,6 +214,7 @@ LC_ALL=C awk ' } # The id of the question a line opens, or "" when it opens none. function question_id(line, c) { + if (line ~ /^([ \t]*>)*[ \t]*([-*+]|[0-9]+[.)])[ \t]/) return "" if (!match(line, /^[^A-Za-z0-9]*Q[0-9]+[.):]/)) return "" c = substr(line, RSTART, RLENGTH) # Q17.S1 is a qualified scenario id, not a question line. @@ -168,7 +229,21 @@ LC_ALL=C awk ' if (fence($0) || in_fence) next } FILENAME == ledger_file { - if ($0 ~ /^[ \t]*- [QCFS][0-9]+[ \t]*\|/) { + if ($0 ~ /^#+[ \t]/) in_matrix = ($0 ~ /^##[ \t]+Terminology matrix[ \t]*$/) + if (in_matrix && $0 ~ /^[ \t]*\|/) { + t = $0 + sub(/^[ \t]*\|[ \t]*/, "", t) + sub(/[ \t]*\|.*/, "", t) + gsub(/[`*_]/, "", t) + t = tolower(t) + if (t != "" && t != "term" && t !~ /^[-: ]+$/ && !(t in seen_term)) { + seen_term[t] = 1 + n_terms++ + term[n_terms] = t + } + next + } + if ($0 ~ /^[ \t]*- [QCFSP][0-9]+[ \t]*\|/) { c = $0 sub(/^[ \t]*- /, "", c) sub(/[ \t]*\|.*/, "", c) @@ -193,21 +268,64 @@ LC_ALL=C awk ' add_code(question_id($0), 0) next } + function close_block() { + if (!block_line) return + if (block_answers != 1) { + missing++ + printf "missing=%d:%s\n", block_line, (block_answers ? "single-recommendation" : "recommendation") + } + for (i = 1; i <= n_terms; i++) { + if (tused[i] && !tctx[i]) { + missing++ + printf "undefined=%d:%s\n", block_line, term[i] + } + } + } + { + if (!block_line && $0 ~ /^[^ -~]*[ \t]*(\*\*|__)?Scope check(\*\*|__)?:/) has_scope++ + if (block_line && $0 ~ /^[^ -~]*[ \t]*(\*\*|__)?(My recommendation|Withheld|Discovery|Strong answer)(\*\*|__)?:/) block_answers++ + } { if ((c = question_id($0)) != "") { + close_block() + block_line = FNR + block_answers = 0 delete used # The Q that opens a question line is the handle of that question, # named by the question itself whatever separator follows it. used[c] = 1 + delete tused + delete tctx + in_ctx = 0 + if (++n_blocks > 1) printf "check=%d:own-context-terms\n", FNR + } + if (block_line) { + if ($0 ~ /^[^ -~]*[ \t]*(\*\*|__)?Context(\*\*|__)?:/) in_ctx = 1 + else if ($0 ~ /^[^ -~]*[ \t]*(\*\*|__)?(My recommendation|Withheld|Discovery|Strong answer)(\*\*|__)?:/) in_ctx = 0 + hay = tolower($0) + for (i = 1; i <= n_terms; i++) { + if (has_term(hay, term[i])) { + tused[i] = 1 + if (in_ctx) tctx[i] = 1 + } + } + if ($0 ~ /^[^ -~]*[ \t]*(\*\*|__)?Flips if(\*\*|__)?:/) printf "check=%d:flips-if-fact\n", FNR + if ($0 ~ /^[^ -~]*[ \t]*(\*\*|__)?My recommendation(\*\*|__)?:/) printf "check=%d:existing-control\n", FNR + } else if (pacing == "single" && $0 !~ /^[ \t]*$/ && $0 !~ /^[^ -~]*[ \t]*((#+[ \t]+)?(\*\*|__)?Round[ \t]+[0-9]|(\*\*|__)?(Scope check|Later|Held|Settled by evidence)(\*\*|__)?:|---+[ \t]*$)/) { + missing++ + printf "extra=%d:single-header\n", FNR } grade($0, FNR) } END { if (in_fence) exit 4 - printf "codes=%d bare=%d status=%s\n", n_codes, bare, (bare ? "fail" : "ok") - exit (bare ? 1 : 0) + close_block() + if (pacing == "single") print "check=1:single-pacing" + if (has_scope != 1) { missing++; print "missing=1:" (has_scope ? "single-scope-check" : "scope-check") } + printf "codes=%d bare=%d status=%s structure=%d\n", n_codes, bare, ((bare || missing) ? "fail" : "ok"), missing + exit ((bare || missing) ? 1 : 0) } -' ledger_file="$ledger" "$ledger" pass=1 "$round" pass=2 "$round" +' ledger_file="$ledger" pacing="$pacing" "$ledger" pass=1 "$round" pass=2 "$round" status=$? case $status in 0 | 1) exit "$status" ;; diff --git a/plugins/planning/scripts/check-round-codes.test.sh b/plugins/planning/scripts/check-round-codes.test.sh index f5cc84733b5..3736dc45329 100755 --- a/plugins/planning/scripts/check-round-codes.test.sh +++ b/plugins/planning/scripts/check-round-codes.test.sh @@ -69,6 +69,7 @@ run() { round named.md \ 'Round 2. The retry budget (question Q14, retry budget) is three tries.' \ + 'Scope check: 2 of 27 questions registered' \ '' \ '❓ Q15: **Should a failed retry page the on-call?**' \ 'Context: each lane (one parallel builder session) retries alone (`src/retry.ts:12`).' \ @@ -80,7 +81,8 @@ round named.md \ '- (a) Page on every failure. Fast response, but noisy nights.' \ '' \ 'Q16: Does question Q15, paging on failure, change the alert threshold?' \ - 'Context: the threshold rests on Q15, paging on failure.' + 'Context: the threshold rests on Q15, paging on failure.' \ + '➡️ My recommendation: no; the alert threshold stays as it is.' run "a round that names every code passes" 0 named.md 'bare=0 status=ok' round bare.md \ @@ -106,10 +108,12 @@ round own-ids.md \ run "a question id from the round itself is checked" 1 own-ids.md '^bare=2:Q20$' round fenced.md \ + 'Scope check: 2 of 27 questions registered' \ 'Q15: Which layout?' \ '```text' \ 'C1 C3 Q14 lane' \ - '```' + '```' \ + 'Discovery: which layouts the readers already know.' run "a fenced block is ignored" 0 fenced.md 'status=ok' run "a fenced ledger row is not a code" 0 fenced.md '^codes=6 ' @@ -118,8 +122,10 @@ round words.md \ 'Context: the C10 rule and Q140 are not ledger ids, and planes are not lanes (one builder session each).' run "a plural term, named later, is still bare at first use" 1 words.md '^bare=1:lane$' round words2.md \ + 'Scope check: 2 of 27 questions registered' \ 'Q15: Should two builder sessions share a cache?' \ - 'Context: the C10 rule and Q140 are not ledger ids; planes and a laneway are not the term.' + 'Context: the C10 rule and Q140 are not ledger ids; planes and a laneway are not the term.' \ + 'My recommendation: no; each session keeps its own cache.' run "a longer token or word is not a code" 0 words2.md 'status=ok' round open.md 'Q15: Which?' '```' @@ -137,7 +143,7 @@ round connective.md 'Q15: Which store?' 'Context: this follows Q14, so the budge run "a comma and a connective do not name a code" 1 connective.md '^bare=2:Q14$' round words3.md 'Q15: Does Q14 hold?' run "a code inside the question line is bare" 1 words3.md '^bare=1:Q14$' -round dot-heading.md 'Q15. Should a failed retry page the on-call?' +round dot-heading.md 'Scope check: 2 of 27 questions registered' 'Q15. Should a failed retry page the on-call?' 'My recommendation: no.' run "a question opened with Q. names its own id" 0 dot-heading.md 'bare=0 status=ok' round paren-heading.md '**Q15)** Should a failed retry page the on-call?' 'Q16) Does it follow Q15?' run "a question opened with Q) names its own id" 1 paren-heading.md '^bare=2:Q15$' @@ -164,13 +170,22 @@ if [[ "$got" -eq 1 ]] && grep -q '^bare=3:S1$' <<<"$out" && grep -q 'bare=1 stat else fail "a scenario opened with a bare S: that is also a sweep id fails ($out)" fi -run "with no sweep rows a scenario opened with a bare S: passes" 0 walk-overlap.md 'bare=0 status=ok' +round walk-overlap-round.md \ + 'Scope check: 2 of 27 questions registered' \ + 'Q17: What is a booking once it moves?' \ + 'Context:' \ + 'S1: Priya holds Tuesday 10:00 and pays 40.00; on Monday she moves it to Thursday.' \ + 'Checked against: S1' \ + 'My recommendation: the same booking, moved.' +run "with no sweep rows a scenario opened with a bare S: passes" 0 walk-overlap-round.md 'bare=0 status=ok' round walk.md \ + 'Scope check: 2 of 27 questions registered' \ 'Q17: What is a booking once it moves?' \ 'Context:' \ 'Q17.S1: Priya holds Tuesday 10:00 and pays 40.00; on Monday she moves it to Thursday.' \ '(a) Same booking. (b) A new booking. Recommend (a): forces "reschedule".' \ - '**Q17.S2:** Omar cancels on Wednesday. Recommend (b), Q17.S1 still stands: forces "cancel".' + '**Q17.S2:** Omar cancels on Wednesday. Recommend (b), Q17.S1 still stands: forces "cancel".' \ + 'My recommendation: a moved booking stays the same booking.' if out="$(bash "$SUT" --ledger "$TMP/sweep-ledger.md" "$TMP/walk.md" 2>&1)" && grep -q 'codes=3 bare=0 status=ok' <<<"$out"; then pass "a scenario walk with question-qualified ids passes beside sweep rows" @@ -199,6 +214,230 @@ else fail "a scenario id shown with nothing beside it is bare ($out)" fi +cat >"$TMP/problem-ledger.md" <<'MD' +## Problems + +- P1 | staff cannot find the queue | agreed + +## Open-question register + +- Q1 | open | round 1 | Which queue? | problem:: P1 +MD +round problem-bare.md \ + 'Q2: Does the alert fix it?' \ + 'Context: this serves P1 and Q1 (the queue question).' +out="$(bash "$SUT" --ledger "$TMP/problem-ledger.md" "$TMP/problem-bare.md" 2>&1)" +got=$? +if [[ "$got" -eq 1 ]] && grep -q '^bare=2:P1$' <<<"$out" && grep -q 'codes=3 bare=1 status=fail' <<<"$out"; then + pass "a bare problem id fails" +else + fail "a bare problem id fails ($out)" +fi +round problem-named.md \ + 'Scope check: 2 of 27 questions registered' \ + 'Q2: Does the alert fix it?' \ + 'Context: this serves P1, staff cannot find the queue, and Q1 (the queue question).' \ + 'Withheld: the alert trial has not reported yet.' +if out="$(bash "$SUT" --ledger "$TMP/problem-ledger.md" "$TMP/problem-named.md" 2>&1)" && + grep -q 'bare=0 status=ok' <<<"$out"; then + pass "a named problem id passes" +else + fail "a named problem id passes ($out)" +fi + +cat >"$TMP/fixture-ledger.md" <<'MD' +## Open-question register + +- Q4 | answered | round 1 | One write per job? | yes +- Q10 | open | round 3 | Audit a failed retry? | +- Q11 | open | round 3 | Who reads the entries? | kind:: discovery +- Q12 | open | round 3 | Which queue? | +- Q13 | open | round 3 | Which store? | +MD +fixture="$SCRIPT_DIR/../surface/tests/fixtures/round-file/round-full.md" +if out="$(bash "$SUT" --ledger "$TMP/fixture-ledger.md" "$fixture" 2>&1)" && + grep -q 'codes=5 bare=0 status=ok' <<<"$out"; then + pass "the shared round fixture names every code" +else + fail "the shared round fixture names every code ($out)" +fi +if grep -q 'structure=0' <<<"$out"; then + pass "the shared round fixture passes the structure checks" +else + fail "the shared round fixture passes the structure checks ($out)" +fi +awk '{ print } /^- No reader is named yet\.$/ { + print "- Basis: no reader survey exists yet." + print "- My recommendation: none until a reader is named." +}' "$fixture" >"$TMP/fixture-bullets.md" +out="$(bash "$SUT" --ledger "$TMP/fixture-ledger.md" "$TMP/fixture-bullets.md" 2>&1)" +got=$? +if [[ "$got" -eq 0 ]] && grep -q 'bare=0 status=ok structure=0' <<<"$out"; then + pass "a Context bullet that starts with a label word is not a label" +else + fail "a Context bullet that starts with a label word is not a label (exit $got: $out)" +fi + +round no-rec.md \ + 'Scope check: 2 of 27 questions registered' \ + '' \ + '❓ Q15: **Should a failed retry page the on-call?**' \ + 'Context: each retry runs alone.' +run "a decision question with no recommendation fails" 1 no-rec.md '^missing=3:recommendation$' +run "a missing recommendation fails the summary" 1 no-rec.md 'bare=0 status=fail structure=1' +round discovery.md \ + 'Scope check: 2 of 27 questions registered' \ + '' \ + '❓ Q15: **Who reads the retry entries?**' \ + 'Context: no reader is named yet.' \ + 'Discovery: who reads them settles whether they need a dashboard.' +run "a discovery question with a Discovery: line passes" 0 discovery.md 'bare=0 status=ok structure=0' +round no-scope.md \ + 'Round 2. The retry budget is settled.' \ + '' \ + '❓ Q15: **Should a failed retry page the on-call?**' \ + '➡️ **My recommendation:** no; a retry stays silent.' +run "a header with no Scope check: line fails" 1 no-scope.md '^missing=1:scope-check$' +run "a bold recommendation label after a marker counts" 1 no-scope.md 'structure=1$' +round two-recs.md \ + 'Scope check: 2 of 27 questions registered' \ + 'Q15: Should a failed retry page the on-call?' \ + 'My recommendation: no.' \ + 'Withheld: the paging trial has not reported.' +run "a question with two answer lines fails" 1 two-recs.md '^missing=2:single-recommendation$' +round q-bullet.md \ + 'Scope check: 2 of 27 questions registered' \ + '' \ + '❓ Q15: **Should a failed retry page the on-call?**' \ + 'Context:' \ + '- Q14: you set the retry budget at three tries.' \ + '1. Q14: the same answer, in a numbered list.' \ + '➡️ My recommendation: no; a retry stays silent.' +run "a list item that opens with Q: is not a question line" 0 q-bullet.md 'bare=0 status=ok structure=0' +round two-scopes.md \ + 'Scope check: 2 of 27 questions registered' \ + 'Scope check: 3 of 27 questions registered' \ + 'Q15: Should a failed retry page the on-call?' \ + 'My recommendation: no.' +run "a header with two Scope check: lines fails" 1 two-scopes.md '^missing=1:single-scope-check$' +round product.md \ + 'Scope check: 3 of 15 questions registered' \ + '' \ + 'Q3: Who has this problem today, by name?' \ + 'Strong answer: one named person, their role, and the last time it cost them.' \ + 'Basis: reference/product-principles.md, principle 4 (a real person with the problem)' +run "a product-action round with a Strong answer: line passes" 0 product.md 'bare=0 status=ok structure=0' + +runx() { + local label="$1" want="$2" ledger="$TMP/$3" file="$TMP/$4" pattern="$5" out got + shift 5 + out="$(bash "$SUT" --ledger "$ledger" "$@" "$file" 2>&1)" + got=$? + if [[ "$got" -ne "$want" ]] || { [[ -n "$pattern" ]] && ! grep -qE -- "$pattern" <<<"$out"; }; then + fail "$label (want exit $want and /$pattern/, got exit $got)" + printf '%s\n' "$out" | sed 's/^/ /' >&2 + return + fi + pass "$label" +} +nox() { + local label="$1" ledger="$TMP/$2" file="$TMP/$3" pattern="$4" out + shift 4 + out="$(bash "$SUT" --ledger "$ledger" "$@" "$file" 2>&1)" + if grep -qE -- "$pattern" <<<"$out"; then + fail "$label (/$pattern/ printed)" + printf '%s\n' "$out" | sed 's/^/ /' >&2 + return + fi + pass "$label" +} + +round single-preamble.md \ + "Of the five open decisions, I'm starting with what happens to an event after its last failed attempt." \ + '### Round 2' \ + '' \ + 'Scope check: 4 of 27 questions registered' \ + '' \ + '❓ Q4: **After the last failed attempt, should the sender keep the event?**' \ + 'Context:' \ + '- Webhooks are delivered at least once (user, round 1).' \ + '➡️ My recommendation: **Yes: keep it.** Dropping breaks at-least-once delivery (verified, user, round 1).' +runx "under single pacing a header line that previews the frontier fails" 1 ledger.md single-preamble.md '^extra=1:single-header$' --pacing single +runx "under round pacing the same header note passes" 0 ledger.md single-preamble.md 'bare=0 status=ok structure=0' +round single-header.md \ + '## Round 2, one question at a time' \ + 'Scope check: 4 of 27 questions registered' \ + 'Later: replay controls' \ + 'Held: Q6 (alert threshold), waiting on the principle map' \ + 'Settled by evidence: Q7 (log format), JSON lines, Basis: verified, src/log.ts:12' \ + '---' \ + '' \ + '❓ Q4: **After the last failed attempt, should the sender keep the event?**' \ + 'Context:' \ + '- Webhooks are delivered at least once (user, round 1).' \ + '➡️ My recommendation: **Yes: keep it.** Dropping breaks at-least-once delivery (verified, user, round 1).' +runx "under single pacing the prescribed header lines pass" 0 ledger.md single-header.md 'structure=0$' --pacing single +runx "single pacing prints its self-check at line 1" 0 ledger.md single-header.md '^check=1:single-pacing$' --pacing single +nox "round pacing prints no single-pacing self-check" ledger.md single-header.md 'single-pacing' +out="$(bash "$SUT" --ledger "$TMP/ledger.md" --pacing sometimes "$TMP/single-header.md" 2>&1)" +got=$? +if [[ "$got" -eq 2 ]]; then + pass "an unknown pacing value is a usage error" +else + fail "an unknown pacing value is a usage error (exit $got: $out)" +fi + +cat >"$TMP/term-ledger.md" <<'MD' +## Open-question register + +- Q1 | open | round 2 | Agree the two upstream terms? | +- Q2 | open | round 2 | Add a cluster step? | + +## Terminology matrix + +| Term | Definition | Source | Status | +|---|---|---|---| +| cluster step | the step that attaches a report to the open cluster with the same dedupe fingerprint | vendor/upstream-triage/README.md:8-10 | proposed | +| `dedupe fingerprint` | a SHA-1 of the normalized first error line | vendor/upstream-triage/README.md:7 | proposed | +MD +round term-elsewhere.md \ + 'Scope check: 2 of 27 questions registered' \ + 'Q1: **Do these two upstream terms mean what you mean by them?**' \ + 'Context:' \ + '- **cluster step:** the step that attaches a report to the open cluster with the same dedupe fingerprint (vendor/upstream-triage/README.md:8-10).' \ + 'My recommendation: agree to both as written.' \ + '---' \ + 'Q2: **Should triage gain a cluster step?**' \ + 'Context:' \ + '- Upstream keeps a fingerprint-to-issue map (vendor/upstream-triage/README.md:8-10).' \ + 'My recommendation: yes, add the cluster step.' +runx "a matrix term used where only another question's Context defines it fails" 1 term-ledger.md term-elsewhere.md '^undefined=7:cluster step$' +runx "the undefined term counts in the structure total" 1 term-ledger.md term-elsewhere.md 'structure=1$' +round term-own.md \ + 'Scope check: 2 of 27 questions registered' \ + 'Q1: **Do these two upstream terms mean what you mean by them?**' \ + 'Context:' \ + '- **cluster step:** the step that attaches a report to the open cluster with the same dedupe fingerprint (vendor/upstream-triage/README.md:8-10).' \ + 'My recommendation: agree to both as written.' \ + '---' \ + 'Q2: **Should triage gain cluster steps?**' \ + "Context: Upstream's cluster step attaches a report to the open cluster with the same dedupe fingerprint (vendor/upstream-triage/README.md:8-10)." \ + 'My recommendation: yes, add the cluster step.' +runx "a matrix term each question's own Context mentions passes" 0 term-ledger.md term-own.md 'bare=0 status=ok structure=0' +runx "a question after the first prints its own-context-terms self-check" 0 term-ledger.md term-own.md '^check=7:own-context-terms$' +nox "the first question prints no own-context-terms self-check" term-ledger.md term-own.md '^check=2:own-context-terms$' + +round judged.md \ + 'Scope check: 2 of 27 questions registered' \ + '❓ Q9: **Should the limiter fail open when its store is down?**' \ + 'Context:' \ + '- The limiter keeps its counts in the session Redis (src/limit.ts:4).' \ + '➡️ My recommendation: **Yes: fail open.** A short over-admit beats refusing all traffic (verified, src/limit.ts:4).' \ + 'Flips if: a burst during a limiter outage hurts more than briefly over-admitting does' +runx "a Flips if: line prints its flips-if-fact self-check" 0 term-ledger.md judged.md '^check=6:flips-if-fact$' +runx "a recommendation prints its existing-control self-check" 0 term-ledger.md judged.md '^check=5:existing-control$' +runx "self-checks leave a clean round passing" 0 term-ledger.md judged.md 'bare=0 status=ok structure=0$' + if [[ "$fails" -gt 0 ]]; then printf '%d failure(s)\n' "$fails" >&2 exit 1 diff --git a/plugins/planning/skills/audit-answers/SKILL.md b/plugins/planning/skills/audit-answers/SKILL.md index 5863ec8dc86..b7e90ce628c 100644 --- a/plugins/planning/skills/audit-answers/SKILL.md +++ b/plugins/planning/skills/audit-answers/SKILL.md @@ -84,6 +84,7 @@ Validation needs a complete answer set. If the interview is already fully answer - a Deferred question tagged **`USER-RESERVED`** stays **deferred**. It is a carry-forward item whose arbiter re-confirms at the `/planning:plan` approval gate *with plan-time context*, so it is not auto-accepted, not validated, and **not turned into an audit question here**; it passes through untouched, arbiter tag intact. - a decision the interview's **auto-guard** class covers, a genuine user choice with real tradeoffs and no codebase answer, is held out of the auto-accept and routed to the human as a real question in the confirm round (Step 4). - an open row asking a fact no self-answer source settled (the shared step's **Needs the user** outcome) is held out of the auto-accept with the auto-guard decisions and routed to the human the same way. The orchestrator's recommendation never becomes the fact, and no validator supplies one. +- an open row whose resolution carries **`kind:: discovery`** (a question about what only the user knows, asked with no recommendation) is held out of the auto-accept, since it has no recommended answer to accept, and routed to the human the same way. - a register row at **`superseded-by-plan`** (a plan change displaced the user's answer) is held out of the auto-accept, never validated into `answered`, and routed to the human as a real question showing both the proposed and the displaced answer. Only the user's reply to that row moves it. - a register row whose resolution carries `hedged:` anywhere (including the page's `hedged:` answer and the legacy `free-text: hedged:` export) is validated, but it never closes on a CONFIRMED verdict: it is routed to the human in the Step 4 confirm round whatever the validators return. Its `open` commitment rows are held out of the auto-accept like any other floor item. @@ -144,7 +145,7 @@ Merge the validators' verdicts. Independence means one dissent is signal: any CH - **CONFIRMED by all** → collapse to a one-line summary per answer. The human skims, does not re-decide. A `free-text:` row in this block keeps its flag. - **`hedged:` rows** → real questions whatever the verdict, restating what the headline commits to and which of its rows are still open. -- **CHALLENGED / RECLASSIFIED / auto-guard-held / unsettled-fact / `superseded-by-plan` rows** → become real numbered human questions, asked in the `/planning:interview` round format (its recommendation-per-question, single-verdict-marker, and dependency-surfacing rules apply). Each challenge's *why* rides along so the human decides informed. +- **CHALLENGED / RECLASSIFIED / auto-guard-held / unsettled-fact / open `kind:: discovery` / `superseded-by-plan` rows** → become real numbered human questions, asked in the `/planning:interview` round format (its recommendation-per-question, single-verdict-marker, and dependency-surfacing rules apply). Each challenge's *why* rides along so the human decides informed. - **USER-RESERVED deferred questions** → listed as carry-forward items (arbiter tag intact), NOT resolved here. They re-confirm at the `/planning:plan` approval gate with plan-time context, so asking them now would strip that context and rewrite the Brief prematurely. ### Step 5. Human confirmation diff --git a/plugins/planning/skills/audit-answers/evals/evals.json b/plugins/planning/skills/audit-answers/evals/evals.json index 2d3d2efa72c..95c1ff1cab5 100644 --- a/plugins/planning/skills/audit-answers/evals/evals.json +++ b/plugins/planning/skills/audit-answers/evals/evals.json @@ -5,10 +5,11 @@ "id": 1, "name": "accepts-then-validates-with-fresh-agents", "prompt": "/planning:audit-answers — I finished the interview but I'm overloaded; accept the recommendations and have agents check them instead of me answering every round.", - "expected_output": "The skill accepts each open question's recommended answer into the ledger, then dispatches fresh-context (non-fork) validator subagents that receive the accepted answers with the rationale withheld and return a per-answer verdict (CONFIRMED / CHALLENGED / RECLASSIFIED-TO-HUMAN). It does NOT invent answers via subagents.", + "expected_output": "The skill accepts each open question's recommended answer into the ledger, skipping any `kind:: discovery` row (it has no recommendation, so it goes to the human), then dispatches fresh-context (non-fork) validator subagents that receive the accepted answers with the rationale withheld and return a per-answer verdict (CONFIRMED / CHALLENGED / RECLASSIFIED-TO-HUMAN). It does NOT invent answers via subagents.", "files": [], "expectations": [ "Output accepts the recommended answers first, then validates them — it does not derive/invent answers with subagents", + "A `kind:: discovery` row is not auto-accepted: it is held out and routed to the human as a question", "Validators are described as fresh-context (non-fork) subagents that receive the decisions with the persuasive rationale withheld", "Each accepted answer gets a per-answer verdict: CONFIRMED, CHALLENGED (with a reason), or RECLASSIFIED-TO-HUMAN" ] diff --git a/plugins/planning/skills/interview/SKILL.md b/plugins/planning/skills/interview/SKILL.md index 734272ba1ba..240456a9aa0 100644 --- a/plugins/planning/skills/interview/SKILL.md +++ b/plugins/planning/skills/interview/SKILL.md @@ -48,7 +48,7 @@ The **pre-clarity** stage. Upstream of exploration, research, and `/planning:pla ## Emit checklist -For interview sessions with ≥2 open questions or explicit `me` mode, copy `${CLAUDE_PLUGIN_ROOT}/skills/interview/templates/checklist.md` into the topic's memory slice as `//interview-checklist.md` (default `.work/`; **one ledger per topic**. Not per session). Re-interview appends `## Resolved (, )` sections; do not create `interview-checklist-2.md`. Tick each step as completed. Steps 1, 3, 4 are mandatory; Steps 1.5 + 2 are mode-conditional. +For interview sessions with ≥2 open questions or explicit `me` mode, Read the template `${CLAUDE_PLUGIN_ROOT}/skills/interview/templates/checklist.md`, then Write it to `//interview-checklist.md` in the topic's memory slice (default `.work/`; **one ledger per topic**. Not per session), never with a shell copy. Re-interview appends `## Resolved (, )` sections; do not create `interview-checklist-2.md`. Tick each step as completed. Steps 1, 3, 4 are mandatory; Steps 1.5 + 2 are mode-conditional. The ledger's `## Open-question register` is emitted the moment **any** round is asked, whatever that threshold says. It has to exist before the first reply arrives. The acceptance-criteria coverage prompt does not count as a round here: it writes no register row and does not by itself make the register exist or bring the register gate into scope ([`context/coverage-prompt.md`](context/coverage-prompt.md)). Register mechanics, the drift check, the unattended ladder, and the gate that grades it: [`context/loop.md`](context/loop.md) "The open-question register". @@ -83,26 +83,30 @@ Unknown actions route to `auto`; surface the unrecognized request as a one-line **Default action leans to `me` (relentless prose rounds).** When invoked with no args (or by proactive auto-trigger), bias toward `me`-mode. Drive open decisions through frontier-rounds prose Q&A. Fall back to direct synthesis (`lock`-style) ONLY when context heavily informs against asking: intent already crystal-clear with no open decisions, OR the user signaled "just lock it / stop asking". Auto-detect's synthesize-directly path is for the genuinely-clear case, not the default posture. -**Question surface: inline prose by default.** Rounds render as numbered inline prose. Dictation-friendly, no per-question cap, and each question carries its recommendation, reasoning, and probe in one readable block. `AskUserQuestion` is an opt-in surface, enabled via the plugin's `use_ask_user_question` user config (`${user_config.use_ask_user_question}`, default off). When opted in, use it ONLY for a round of mutually independent questions that fits one `AskUserQuestion` call and whose questions are **simple selections or binary confirms** (the per-call question limit: ). A card carries options, not a recommendation's reasoning or a constraint-surfacing probe, so any question needing its basis argued stays prose. Fall back to prose when the frontier exceeds that limit, any question in the round depends on another, or a question needs more than a pick. The card cannot express a dependency or a rationale, and chunking a round across multiple cards fragments it. When in doubt, prose. +**Question surface: inline prose by default.** Rounds render as numbered inline prose. Dictation-friendly, no cap on questions per round except the shared lane's ([`context/lane-shared.md`](context/lane-shared.md) "Round cap"), and each question carries its recommendation, reasoning, and probe in one readable block. `AskUserQuestion` is an opt-in surface, enabled via the plugin's `use_ask_user_question` user config (`${user_config.use_ask_user_question}`, default off). When opted in, use it ONLY for a round of mutually independent questions that fits one `AskUserQuestion` call and whose questions are **simple selections or binary confirms** (the per-call question limit: ). A card carries options, not a recommendation's reasoning or a constraint-surfacing probe, so any question needing its basis argued stays prose. Fall back to prose when the frontier exceeds that limit, any question in the round depends on another, or a question needs more than a pick. The card cannot express a dependency or a rationale, and chunking a round across multiple cards fragments it. When in doubt, prose. -**Question lanes.** The surface registry picks the lane; this skill names none. Before the first round, run `node "${CLAUDE_PLUGIN_ROOT}/surfaces/registry.mjs" resolve --surface '${user_config.surface}' --data-dir "${CLAUDE_PLUGIN_DATA}"`, adding `--argument ''` when the user asks for a surface in this session or states which one is configured (the `share` action passes `share`), and `--unattended` when the caller declared the run unattended (Step 1.5). It prints one JSON line: `surface`, `decidedBy`, `available`, `alternatives`, `read`, `handle`, `line`, `planners`, `offers` and `notices`. When `line` is not null, print it verbatim, once, as the first line of the message that asks round 1, and proceed without waiting for a surface choice; print each notice in one line. The line may also name an installed optional planner and offer an optional plugin that is not installed; neither is ever required, and the interview goes on either way. When the user says yes to an offer, run that entry's `offers[].install` command, which installs at user scope, then tell the user to run `/reload-plugins` to load it. Read each `read` path under `${CLAUDE_PLUGIN_ROOT}` and nothing else of another lane, and follow `handle.rules`; a later request for another surface runs the resolver again with that argument. When the resolver cannot run (no node or no Bash), say so in one line and ask inline prose rounds as below. Every lane carries one question shape, [`context/question-shape.md`](context/question-shape.md); read it before writing a question anywhere but this conversation. `use_emoji_question_markers`: `${user_config.use_emoji_question_markers}`; `question_pacing`: `${user_config.question_pacing}`; `shared_session_open_browser`: `${user_config.shared_session_open_browser}`. A value still shown as its `${user_config.KEY}` token is unset and takes that key's declared default: `use_emoji_question_markers` `true`, `shared_session_open_browser` `true`. `question_pacing` takes the value its cascade resolves ("One question at a time" below), default `round`. Spokes arrive unexpanded, so put these resolved values in place of their `${user_config.KEY}` tokens. +**Question lanes.** The surface registry picks the lane; this skill names none. Before the first round, once `question_pacing` has resolved ("Question pacing" below), run `node "${CLAUDE_PLUGIN_ROOT}/surfaces/registry.mjs" resolve --surface '${user_config.surface}' --pacing '' --ledger '//interview-checklist.md' --data-dir "${CLAUDE_PLUGIN_DATA}"`, adding `--argument ''` when the user asks for a surface in this session or states which one is configured (the `share` action passes `share`), and `--unattended` when the caller declared the run unattended (Step 1.5). It prints one JSON line: `surface`, `decidedBy`, `available`, `alternatives`, `read`, `handle`, `line`, `planners`, `offers` and `notices`. + +**Print `line` exactly as given.** When `line` is not null, its text, character for character, is the opening of the message that asks round 1: no rewording, no added or dropped word, each of its lines on its own line, printed once. Then proceed without waiting for a surface choice; print each notice in one line. Its later lines may name an installed optional planner and offer an optional plugin that is not installed; neither is ever required, and the interview goes on either way. When the user says yes to an offer, run that entry's `offers[].install` command, which installs at user scope, then tell the user to run `/reload-plugins` to load it. Read each `read` path under `${CLAUDE_PLUGIN_ROOT}` and nothing else of another lane, and follow `handle.rules`; a later request for another surface runs the resolver again with that argument. When the resolver cannot run (no node or no Bash), say so in one line and ask inline prose rounds as below. Every lane carries one question shape, [`context/question-shape.md`](context/question-shape.md); read it before writing the first question in any lane, this conversation included. `use_emoji_question_markers`: `${user_config.use_emoji_question_markers}`; `question_pacing`: `${user_config.question_pacing}`; `shared_session_open_browser`: `${user_config.shared_session_open_browser}`. A value still shown as its `${user_config.KEY}` token is unset and takes that key's declared default: `use_emoji_question_markers` `true`, `shared_session_open_browser` `true`. `question_pacing` takes the value its cascade resolves ("Question pacing" below), default `round`; `question_detail` (`${user_config.question_detail}`) the value "Keep the layout whole" below resolves, default `standard`. Spokes arrive unexpanded, so put these resolved values in place of their `${user_config.KEY}` tokens. ## Stance: supportive, depth-first, opinionated The Q&A path of this skill is one engine wrapped in a stop condition and an output contract. Four working principles drive it: -1. **Frontier-rounds loop**. A question is ready once every decision it rests on has an answer, and the set of ready questions is the **frontier**. On a webhook-retry task, the retry budget is ready on day one, while the alert threshold rests on that budget and so sits out until the budget is answered. Each round numbers every ready question, orders them by how much later work each answer would reshape, largest first, sends them together (one per message under `single` pacing, "One question at a time" below), and waits; the replies decide what is ready next time. One ready question still makes a round +1. **Frontier-rounds loop**. A question is ready once every decision it rests on has an answer, and the set of ready questions is the **frontier**. On a webhook-retry task, the retry budget is ready on day one, while the alert threshold rests on that budget and so sits out until the budget is answered. Each round numbers every ready question, orders them as [`context/loop.md`](context/loop.md) "Frontier rounds" item 2 says (new questions by how much later work each answer would reshape, largest first), sends them together (one per message under `single` pacing, "Question pacing" below), and waits; the replies decide what is ready next time. One ready question still makes a round 2. **Survey-then-deep**. Before asking blind, do a fast breadth pass (repo files, recent commits, existing skills, relevant project rules) so questions land in real context 3. **Climb-to-anchor**. Find the nearest `CLAUDE.md`, `AGENTS.md`, domain-vocabulary file, or module README by walking UP from the relevant directory toward repo root; let those shape questions instead of asking what is already documented -4. **Immediate doc maintenance** *(engineering sessions only)*. When an answer resolves a domain - term, invoke `/domain-driven-design:ubiquitous-language curate` via the Skill tool (if that plugin is enabled; else - record the term and its definition in the resolution field of the open-question register row that resolved it) IMMEDIATELY between questions, not - batched at end. Route - decisions, gotchas, and conventions to their proper homes (ADR, project rules, side note) in the - same response. A general session writes no repo docs. It drives to a shared-understanding summary - only - -**Intake the starting point.** Early in the loop (or before it), establish where the user is. One intake question that discloses their starting point; questions and recommendations calibrate to that disclosure. Route by what it shows; recommend, never auto-switch: +4. **The project's terms from the start**. Once the interview starts, past the quick-change skip + in [`context/routing.md`](context/routing.md), invoke `/domain-driven-design:ubiquitous-language lookup` + via the Skill tool, with one line saying why ("Reading the project glossary so these questions + use its terms."); when that skill is not listed, read the glossary file as loop.md "Step 1. + Survey before you ask" says, and the opening lines offer its plugin's install once ("Question lanes"). + Agree each term in the question where it first appears and write it only after the user + confirms its restatement ("Agree a term where it first appears"). Route decisions, gotchas, and + conventions to their proper homes (ADR, project rules, side note) in the same response. A + general session writes no repo docs. It drives to a shared-understanding summary only + +**Intake the starting point.** Early in the loop (or before it), establish where the user is. One discovery question in the intake round ([`context/lookups.md`](context/lookups.md) "The intake round") that discloses their starting point; questions and recommendations calibrate to that disclosure. Route by what it shows; recommend, never auto-switch, in the one-line shape [`context/lookups.md`](context/lookups.md) "The intake round" gives a routing suggestion: - **Idea stage** (a half-formed idea, "I just had this thought", the user does not yet know what they want): recommend `/planning:brainstorm`, which explores approaches and returns here with a handoff. If the user stays, follow [`problem-framing.md`](../../context/problem-framing.md) "Idea-stage requests". - **User-facing product work** whose users or success measures are unsettled: recommend `/planning:prd` first. @@ -112,7 +116,7 @@ The Q&A path of this skill is one engine wrapped in a stop condition and an outp Every other skill the interview hands to or detours through: [`context/composition.md`](context/composition.md). -**Ask the quality bar in round 1.** When the request does not already state it, round 1 asks which bar the work is for: a throwaway prototype, an internal tool, or production. When the request states it, never re-ask it; treat it as settled. Interview depth scales with the bar. A prototype gets the scoping questions only, with production concerns (accounts, scaling, monitoring, CI, coverage targets) named as deferred rather than asked. Production gets the full depth, including failure handling, data correctness, security, monitoring, and rollback; an internal tool sits between the two. When the user cannot judge which bar applies ("I don't know", "you tell me"), recommend one from evidence: who runs the result, what data it touches, and whether it ships to users. Ask them to confirm it like any recommendation. Never fall back to the prototype bar unasked. +**Ask the quality bar in round 1.** When the request does not already state it, round 1 asks, as a discovery question, which bar the work is for, its Context naming the three bars: a throwaway prototype, an internal tool, or production. When the request states it, never re-ask it; treat it as settled. Interview depth scales with the bar. A prototype gets the scoping questions only, with production concerns (accounts, scaling, monitoring, CI, coverage targets) named as deferred rather than asked. Production gets the full depth, including failure handling, data correctness, security, monitoring, and rollback; an internal tool sits between the two. When the user cannot judge which bar applies ("I don't know", "you tell me"), recommend one from evidence in the next round: who runs the result, what data it touches, and whether it ships to users. Ask them to confirm it like any recommendation. Never fall back to the prototype bar unasked. **Open with the problem when the request has none.** A request phrased as a solution with no stated problem follows [`problem-framing.md`](../../context/problem-framing.md) "Prescribed solution": read that section before round 1, then state the problem worked back out with its evidence rating, and make round 1's first question its rethink-or-accept question, never an open question asking what problem the solution solves; any why question comes after it. When the request names neither a problem nor a solution (an idea), round 1's first question asks what problem it solves and why the user wants it solved; keep that answer in the user's words for the Goal's Intent line. @@ -122,11 +126,11 @@ Every other skill the interview hands to or detours through: [`context/compositi A new effort starts here, however big. When the interview shows the effort will outrun a single session AND is still too foggy to phrase as sharp questions, the user can't yet list the decisions, let alone lock them. Name `/planning:wayfind` to the user (it charts the fog as a map of decision items and works the frontier down decision by decision, graduating to a Brief once it clears); recommend, never auto-switch. The user can run it in this same session: its chart step reads this topic's ledger, so settled answers become decisions so far and open questions become decision items. -**Question budget scales with what's already settled.** Upstream artifacts, research findings, exploration output, a PRD, a design resolution, count as settled prerequisites: an interview invoked after them starts with a smaller tree and fewer rounds; never re-ask what an artifact already answers. There is no numeric question cap, but a frontier that keeps *ballooning* (each round opens more branches than it closes) is the wayfind signal above, not a license for a marathon session. Surface the routing recommendation instead of grinding on. +**Question budget scales with what's already settled.** Upstream artifacts, research findings, exploration output, a PRD, a design resolution, count as settled prerequisites: an interview invoked after them starts with a smaller tree and fewer rounds; never re-ask what an artifact already answers. There is no numeric question cap (the shared lane's round cap, [`context/lane-shared.md`](context/lane-shared.md) "Round cap", limits a round, not the interview), but a frontier that keeps *ballooning* (each round opens more branches than it closes) is the wayfind signal above, not a license for a marathon session. Surface the routing recommendation instead of grinding on. **Bulk application work is not a decision set. The interview settles it once, it never executes it.** When the task applies a settled shape across a corpus (every document in a set, every file matching a pattern, every page of a spec), the interview's output is TWO things and no more: the small set of genuinely contested decisions, driven to decisions the normal way, plus an **execution contract**. One line in the session's output artifact, routed by domain like every other output: the Brief's `### Acceptance criteria` in an engineering session, the shared-understanding summary in a general one. Naming the per-unit close-out loop (one source unit at a time: apply, verify, close) and what *closed* means for a unit. NEVER enumerate the per-unit applications as decision rows: an interview that becomes the execution container turns every application step into a decision needing its own adoption ceremony, and the corpus's size becomes the session's length. **Tripwire. Count the candidates against the corpus:** when the candidate question count scales with the number of source units rather than with the number of genuine forks, that is execution masquerading as decisions. Collapse the per-unit rows into the loop, keep only the forks, and say what you collapsed. Distinct from the ballooning frontier above and remedied differently: ballooning routes to `/planning:wayfind`, unit-scaling collapses into the execution contract and hands off to execution. -Tone is collaborative but opinionated. You are not interrogating; you are helping the user think out loud by PROPOSING answers grounded in codebase evidence. When the user gives a definitive answer, lock it. When they hesitate, slow down and offer two or three concrete shapes the answer could take. Every option set names exactly ONE recommended option marked **(RECOMMENDED)** with its basis (each fact with its source, then the one-line inference), except a withheld question (see "Recommended answers"), which is asked open with no recommended option. The surface follows the resolution in "Question lanes" above: rounds go where its handle sends them, and `AskUserQuestion` only where `handle.rules` allows cards, when the user opted in and the round qualifies. +Tone is collaborative but opinionated. You are not interrogating; you are helping the user think out loud by PROPOSING answers grounded in codebase evidence. When the user gives a definitive answer, lock it. When they hesitate, slow down and offer two or three concrete shapes the answer could take. Every option set names exactly ONE recommended option marked **(RECOMMENDED)** with its basis (each fact with its source, then the one-line inference), except a withheld or discovery question (see "Recommended answers"), each asked open with no recommended option. The surface follows the resolution in "Question lanes" above: rounds go where its handle sends them, and `AskUserQuestion` only where `handle.rules` allows cards, when the user opted in and the round qualifies. **Do not nod along.** Agreement is not the job. When an answer conflicts with the code, a constraint row, or an earlier answer, name the conflict and its source once, before recording it, and ask whether they hold the answer or change it, registered like any question; the row stays `open` until they say. Never praise an answer or call it a good idea. @@ -134,10 +138,11 @@ Tone is collaborative but opinionated. You are not interrogating; you are helpin Read [`context/relentless-mode.md`](context/relentless-mode.md) before the first round when the action resolves to `me`. It holds what relentless mode means and its canonical framing. The round mechanics below apply to every mode that asks questions. -**Ask each round inline as one numbered set.** Per-question shape within the round: +**Ask each round inline as one numbered set.** Per-question shape within the round, as the saved round file holds it (the full block; "Keep the layout whole" says what each detail level prints): ```text Q: **** + Context: - () - () @@ -157,51 +162,59 @@ Basis: verified, - Q: : , one bullet per part, numbered on from Q] Alternatives: -- (a)