diff --git a/skills/plan/SKILL.md b/skills/plan/SKILL.md index bffe02f194..99b656526d 100644 --- a/skills/plan/SKILL.md +++ b/skills/plan/SKILL.md @@ -60,7 +60,7 @@ The highest tier any vector demands wins: `session` (all vectors low — one sit An explicit user time constraint is a ceiling, not one vector among fifteen: work that must fit one sitting (a hackathon demo, a stated hour count, "before my flight") caps the scale at `session`, and a day-or-two deadline caps it at `hobby`, no matter how public the audience or how ambitious the idea. The user's clock outranks every ambition vector. Print the constraint with the scale: `Scale: session (hackathon, one sitting)`. -The scale binds the whole chain, not just this invocation, and fixes a chain-wide question budget: five total questions at `session`, eight at `hobby`, twelve at `project`, uncapped at `product` and `venture` (approval STOP gates excluded). Every handoff to a chained review or specialist names the scale, any time box, and the questions already spent; the receiving workflow applies its proportional-planning port to the inherited scale and the remaining budget without re-asking or re-classifying upward. +The scale binds the whole chain, not just this invocation. The chain-wide question budget defaults to ZERO at every scale — full autonomy out of the box; budget above zero comes only from the developer profile's autonomy dial (`round((1 − autonomy) × 6)` across the whole chain, absent profile = autonomy 1.0; approval STOP gates and privacy-consent gates excluded). Scope, ambition, and technical depth are never question material — the specialist decides, states the choice in one line, and offers the opt-down. Every handoff names the scale, any time box, and the questions already spent; the receiving workflow deducts without re-asking or re-classifying upward. ## Top-level modes diff --git a/skills/plan/references/legacy/autoplan.md b/skills/plan/references/legacy/autoplan.md index 5bdfd1b38a..ca26774fee 100644 --- a/skills/plan/references/legacy/autoplan.md +++ b/skills/plan/references/legacy/autoplan.md @@ -57,7 +57,7 @@ Say to the user via AskUserQuestion: > "No design doc found for this branch. `$plan --mode Discovery --module office-hours` produces a structured problem > statement, premise challenge, and explored alternatives — it gives this review much -> sharper input to work with. Takes about 10 minutes. The design doc is per-feature, +> sharper input to work with. Takes a few minutes. The design doc is per-feature, > not per-product — it captures the thinking behind this specific change." Options: @@ -122,7 +122,7 @@ surfaced at a final approval gate. These rules auto-answer every intermediate question: 1. **Choose completeness** — Ship the whole thing. Pick the approach that covers more edge cases. -2. **Boil lakes** — Fix everything in the blast radius (files modified by this plan + direct importers). Auto-approve expansions that are in blast radius AND < 1 day CC effort (< 5 files, no new infra). +2. **Boil lakes** — Fix everything in the blast radius (files modified by this plan + direct importers). Auto-approve expansions that are inside the plan's blast radius, add no new infrastructure, and change no authority boundary (no new external service, spend, or publish surface). File count is not a criterion — a current agent's throughput makes it meaningless as an effort proxy. 3. **Pragmatic** — If two options fix the same thing, pick the cleaner one. 5 seconds choosing, not 5 minutes. 4. **DRY** — Duplicates existing functionality? Reject. Reuse what exists. 5. **Explicit over clever** — 10-line obvious fix > 200-line abstraction. Pick what a new contributor reads in 30 seconds. @@ -942,6 +942,19 @@ Suggest next step: `/ship` when ready to create the PR. - **Artifacts are deliverables.** Test plan artifact, failure modes registry, error/rescue table, ASCII diagrams — these must exist on disk or in the plan file when the review completes. If they don't exist, the review is incomplete. - **Sequential order.** CEO → Eng → DX. Each phase builds on the last. + +## Capability priors (evidence dated 2026-08; re-verify before citing past 2027) + +Never justify narrowing scope, adding human-review stages, or steering the user to a "safer" variant by asserting that an LLM can't do something reliably. The measured 2026 reality, with sources, is the opposite of the 2023-24 folklore: + +- Pairwise semantic matching ("do these two announcements describe the same thing?") is at practical saturation: 92–99 F1 zero-shot on real-world benchmarks (OpenSanctions Pairs, 755K pairs: GPT-4o 98.95 F1), BEATING production rule engines by ~8 points — and the error direction is inverted: ~3% false merges vs 14% for the rules (arxiv.org/html/2603.11051; futuresearch.ai measured precision 1.000, "zero false merges," at ~$1.12/1k rows in production). +- LLM-as-judge on objective correctness reaches Cohen's kappa ≈ 0.87 (21-judge study, 541K judgments, arxiv.org/html/2606.19544); verbosity bias has collapsed an order of magnitude below 2023 levels. +- Grounded-summarization hallucination is low single digits for current models (Vectara leaderboard) — model-dependent, so name the model, don't indict the class. + +What still genuinely fails, and may honestly shape a design: subjective preference judging under adversarial phrasing (kappa ~0.5), mega-schema extraction (hundreds of fields per document), and multi-document event clustering with small open models. Those are different task shapes — do not let them stand in for pairwise matching or objective judging. + +When a load-bearing capability is uncertain for the user's ACTUAL data: keep the ambitious design as the default, and put a 30-minute spike in the plan that measures it — ~50 hand-labeled examples as the calibration set, 85–90% agreement as the deploy gate (the 2026 practitioner standard). "Measure it this afternoon" is the modern answer; "downgrade the product" is the stale one. + ## Upstream judgment port: PR #679 @@ -987,7 +1000,7 @@ The binding scale comes from the first available source: the printed `Scale:` he - `project`: run the specialist's default workflow, batching question rounds where its source authorizes smart skips; size the roadmap in weeks. - `product` and `venture`: the full specialist workflow and its complete question pressure apply; this rule removes nothing. -The scale also fixes a chain-wide question budget — a ceiling on individual questions (not rounds) counted across the entire invocation and everything it chains into, reviews included: five at `session` (a hackathon demo or a one-sitting toy gets five questions, total, ever), eight at `hobby`, twelve at `project`, uncapped at `product` and `venture`. Every handoff carries the scale, the time box, and the questions already spent; the receiving specialist deducts from the remaining budget, never restarts it. A specialist whose remaining budget is zero infers the answer from the prompt, the repository, and stated constraints, states the inference and its default in one line, and proceeds — it does not ask. Approval STOP gates (approve/revise the plan, authorize a mutation) are outside the budget; everything else, including "which option do you prefer" refinements, spends it. Spend the budget on the questions whose wrong answer is most expensive to reverse, earliest. +The chain-wide question budget defaults to ZERO, at every scale — full autonomy out of the box. The current generation of models infers the answers from the prompt, the repository, and stated constraints; state each inference and its default in one line the user can correct in passing, and proceed. Scope, ambition, and technical depth are never what a question is for: decide, state the choice with a one-line opt-down, and keep moving. Budget above zero comes from the person, not the product: read `cat "${GSTACK_HOME:-$HOME/.gstack}"/developer-profile.json 2>/dev/null`, take `declared.autonomy` (newest inferred as fallback; absent profile means autonomy 1.0), and allow `round((1 − autonomy) × 6)` individual questions across the entire chain — this invocation plus everything it hands off to, reviews included. Every handoff carries the scale, the time box, and the questions already spent; the receiving specialist deducts, never restarts. Approval STOP gates (approve/revise the plan, authorize a mutation — authority, never inference) and privacy-consent gates are outside the budget; a consent gate with no pending question to bundle into does not become its own turn — skip the gated action, note the skip in one line, continue. Whatever the budget, spend it on the fork whose wrong answer is most expensive to reverse, earliest — never on confirming what the model already knows. - Never run a questioning round merely to classify scale. Classify from the prompt and cheap repository evidence, defaulting unknown vectors low; a specialist's own later questions may raise the scale mid-session, and an upgrade restores the full workflow from that point. Review specialists spend question rounds on decisions, not ceremony — at every scale, and sharpest at `session`/`hobby`: diff --git a/skills/plan/references/legacy/office-hours.md b/skills/plan/references/legacy/office-hours.md index fe2fe20605..8357c6a5e5 100644 --- a/skills/plan/references/legacy/office-hours.md +++ b/skills/plan/references/legacy/office-hours.md @@ -104,9 +104,9 @@ $GSTACK_BIN/gstack-learnings-search --limit 10 2>/dev/null || true If learnings are found, incorporate them into your analysis. When a review finding matches a past learning, note it: "Prior learning applied: [key] (confidence N, from [date])" -5. **Ask: what's your goal with this?** This is a real question, not a formality. The answer determines everything about how the session runs. +5. **Determine the goal — infer it; do not ask.** The answer determines everything about how the session runs, and a modern host reads it straight off the prompt: "for fun", "side project", a playful idea with no business framing → Builder; customers, revenue, "we", a market → Startup; a sponsor or team → intrapreneurship. Ambiguous prompts default to Builder — the cheaper mode to be wrong in, since a Builder doc upgrades to a Startup diagnostic on one word. Always state the inference in one line and keep moving: "Reading this as a side project — say 'startup' and I'll switch to the hard questions." - Via AskUserQuestion, ask: + Ask only when the user's profile grants question budget AND the prompt genuinely supports two different session shapes; then use AskUserQuestion: > Before we dig in — what's your goal with this? > @@ -326,7 +326,9 @@ Both are outcome-framed. Only one has the 'whoa.' Builder mode's job is to surfa ### Questions (generative, not interrogative) -Ask these **ONE AT A TIME** via AskUserQuestion. The goal is to brainstorm and sharpen the idea, not interrogate. +**Default: zero questions.** Someone building for fun wants momentum, not an interview. A modern model answers the sharpening prompts itself from the idea, the landscape search, and taste; it asks only when the user's profile grants budget (see the chain-wide budget rule) AND a prompt below is a true taste fork — two defensible products, not one right answer. + +Work through these prompts YOURSELF and present your answers as assertions the user can correct in passing — not as questions: - **What's the coolest version of this?** What would make it genuinely delightful? - **Who would you show this to?** What would make them say "whoa"? @@ -334,9 +336,9 @@ Ask these **ONE AT A TIME** via AskUserQuestion. The goal is to brainstorm and s - **What existing thing is closest to this, and how is yours different?** - **What would you add if you had unlimited time?** What's the 10x version? -**Smart-skip:** If the user's initial prompt already answers a question, skip it. Only ask questions whose answers aren't yet clear. +State them in the design doc as "here's the version I'd build and why" — commit to the ambitious reading of the idea. Scope, feature depth, and how technical to go are never worth a question — decide, state, offer the one-line opt-down. Spend a question only on a true identity fork (two defensible DIFFERENT products, e.g. "personal tool" vs "public bot other people follow") or on a fact you cannot infer. Otherwise spend zero: a wrong assertion costs the user one sentence to correct; a question costs them the momentum they came here with. -**STOP** after each question. Wait for the response before asking the next. +**STOP** after a question if you ask one. Never queue a second before the first is answered — and remember the cap. **Escape hatch:** If the user says "just do it," expresses impatience, or provides a fully formed plan → fast-track to Phase 4 (Alternatives Generation). If user provides a fully formed plan, skip Phase 2 entirely but still run Phase 3 and Phase 4. @@ -356,7 +358,7 @@ grep -li "\|\|" "${GSTACK_HOME:-$HOME/.gstack}"/pr If matches found, read the matching design docs and surface them: - "FYI: Related design found — '{title}' by {user} on {date} (branch: {branch}). Key overlap: {1-line summary of relevant section}." -- Ask via AskUserQuestion: "Should we build on this prior design or start fresh?" +- Default to building on it and say so in one line ("Building on that doc — say 'fresh' to start over"). Ask via AskUserQuestion only when the prior doc and the new ask genuinely conflict; the question spends budget. This enables cross-team discovery — multiple users exploring the same project will see each other's design docs in `"${GSTACK_HOME:-$HOME/.gstack}"/projects/`. @@ -370,7 +372,7 @@ Read ETHOS.md for the full Search Before Building framework (three layers, eurek After understanding the problem through questioning, search for what the world thinks. This is NOT competitive research. This is understanding conventional wisdom so you can evaluate where it's wrong. -**Privacy gate:** Before searching, use AskUserQuestion: "I'd like to search for what the world thinks about this space to inform our discussion. This sends generalized category terms (not your specific idea) to a search provider. OK to proceed?" +**Privacy gate:** Before searching, get consent via AskUserQuestion: "I'd like to search for what the world thinks about this space to inform our discussion. This sends generalized category terms (not your specific idea) to a search provider. OK to proceed?" Consent is outside the question budget — bundle it into the same AskUserQuestion call as a pending substantive question instead of spending a turn on it alone. Options: A) Yes, search away B) Skip — keep this session private If B: skip this phase entirely and proceed to Phase 3. Use only in-distribution knowledge. @@ -633,7 +635,7 @@ already knows. A good test: would this insight save time in a future session? If ## Important Rules - **Never start implementation.** This skill produces design docs, not code. Not even scaffolding. -- **Questions ONE AT A TIME.** Never batch multiple questions into one AskUserQuestion. +- **Batch independent questions** into one AskUserQuestion round (the tool carries up to four); one-at-a-time is for the Startup diagnostic, where each answer shapes the next push. The zero-default budget applies either way — a batched round still spends one budget slot per question in it. - **The assignment is mandatory.** Every session ends with a concrete real-world action — something the user should do next, not just "go build it." - **If user provides a fully formed plan:** skip Phase 2 (questioning) but still run Phase 3 (Premise Challenge) and Phase 4 (Alternatives). Even "simple" plans benefit from premise checking and forced alternatives. - **Completion status:** @@ -641,6 +643,19 @@ already knows. A good test: would this insight save time in a future session? If - DONE_WITH_CONCERNS — design doc approved but with open questions listed - NEEDS_CONTEXT — user left questions unanswered, design incomplete + +## Capability priors (evidence dated 2026-08; re-verify before citing past 2027) + +Never justify narrowing scope, adding human-review stages, or steering the user to a "safer" variant by asserting that an LLM can't do something reliably. The measured 2026 reality, with sources, is the opposite of the 2023-24 folklore: + +- Pairwise semantic matching ("do these two announcements describe the same thing?") is at practical saturation: 92–99 F1 zero-shot on real-world benchmarks (OpenSanctions Pairs, 755K pairs: GPT-4o 98.95 F1), BEATING production rule engines by ~8 points — and the error direction is inverted: ~3% false merges vs 14% for the rules (arxiv.org/html/2603.11051; futuresearch.ai measured precision 1.000, "zero false merges," at ~$1.12/1k rows in production). +- LLM-as-judge on objective correctness reaches Cohen's kappa ≈ 0.87 (21-judge study, 541K judgments, arxiv.org/html/2606.19544); verbosity bias has collapsed an order of magnitude below 2023 levels. +- Grounded-summarization hallucination is low single digits for current models (Vectara leaderboard) — model-dependent, so name the model, don't indict the class. + +What still genuinely fails, and may honestly shape a design: subjective preference judging under adversarial phrasing (kappa ~0.5), mega-schema extraction (hundreds of fields per document), and multi-document event clustering with small open models. Those are different task shapes — do not let them stand in for pairwise matching or objective judging. + +When a load-bearing capability is uncertain for the user's ACTUAL data: keep the ambitious design as the default, and put a 30-minute spike in the plan that measures it — ~50 hand-labeled examples as the calibration set, 85–90% agreement as the deploy gate (the 2026 practitioner standard). "Measure it this afternoon" is the modern answer; "downgrade the product" is the stale one. + ## Upstream judgment port: PR #679 @@ -696,7 +711,11 @@ The binding scale comes from the first available source: the printed `Scale:` he - `project`: run the specialist's default workflow, batching question rounds where its source authorizes smart skips; size the roadmap in weeks. - `product` and `venture`: the full specialist workflow and its complete question pressure apply; this rule removes nothing. -The scale also fixes a chain-wide question budget — a ceiling on individual questions (not rounds) counted across the entire invocation and everything it chains into, reviews included: five at `session` (a hackathon demo or a one-sitting toy gets five questions, total, ever), eight at `hobby`, twelve at `project`, uncapped at `product` and `venture`. Every handoff carries the scale, the time box, and the questions already spent; the receiving specialist deducts from the remaining budget, never restarts it. A specialist whose remaining budget is zero infers the answer from the prompt, the repository, and stated constraints, states the inference and its default in one line, and proceeds — it does not ask. Approval STOP gates (approve/revise the plan, authorize a mutation) are outside the budget; everything else, including "which option do you prefer" refinements, spends it. Spend the budget on the questions whose wrong answer is most expensive to reverse, earliest. +**The default question budget is ZERO, at every scale.** Full autonomy out of the box: the current generation of models infers the answers from the prompt, the repository, and a landscape search, states each inference in one line where the user can correct it in passing, and proceeds to the plan. A question to the user is not the default path — it is the opt-in path. + +Only two things interrupt autonomously, and neither is a preference question: approval STOP gates (approve/revise the plan, authorize a mutation — that is authority, never inference) and privacy-consent gates. A consent gate with no pending question to bundle into does not become its own interview turn: skip the gated action, note the skip in one line ("landscape search skipped — needs consent; say 'search' to grant it"), and continue. Autonomy never overrides consent; it routes around it. + +**The budget above zero comes from the person, not the product.** Read the developer profile before the first would-be question: `cat "${GSTACK_HOME:-$HOME/.gstack}"/developer-profile.json 2>/dev/null`. Take `declared.autonomy`, falling back to the newest inferred value; absent profile means autonomy 1.0. Questions allowed across the entire chain — this invocation plus everything it hands off to, reviews included — is `round((1 − autonomy) × 6)`: autonomy 1.0 asks nothing, 0.5 gets three, 0 gets six. Every handoff carries the questions already spent; the receiving specialist deducts, never restarts. Whatever the budget, spend it on the fork whose wrong answer is most expensive to reverse, earliest — never on confirming what the model already knows. This is the consumption side of `/plan-tune`: users who want to be asked tune autonomy down; nobody has to configure anything to get the autonomous default. - Never run a questioning round merely to classify scale. Classify from the prompt and cheap repository evidence, defaulting unknown vectors low; a specialist's own later questions may raise the scale mid-session, and an upgrade restores the full workflow from that point. Review specialists spend question rounds on decisions, not ceremony — at every scale, and sharpest at `session`/`hobby`: diff --git a/skills/plan/references/legacy/plan-ceo-review.md b/skills/plan/references/legacy/plan-ceo-review.md index 5fafe453f0..2ad6f87e87 100644 --- a/skills/plan/references/legacy/plan-ceo-review.md +++ b/skills/plan/references/legacy/plan-ceo-review.md @@ -162,7 +162,7 @@ Say to the user via AskUserQuestion: > "No design doc found for this branch. `$plan --mode Discovery --module office-hours` produces a structured problem > statement, premise challenge, and explored alternatives — it gives this review much -> sharper input to work with. Takes about 10 minutes. The design doc is per-feature, +> sharper input to work with. Takes a few minutes. The design doc is per-feature, > not per-product — it captures the thinking behind this specific change." Options: @@ -390,7 +390,7 @@ Every expansion proposal you generate in SCOPE EXPANSION or SELECTIVE EXPANSION FLAT (avoid): "Add real-time notifications. Users would see workflow results faster — latency drops from ~30s polling to <500ms push. Effort: ~1 hour CC." -EXPANSIVE (aim for): "Imagine the moment a workflow finishes — the user sees the result instantly, no tab-switching, no polling, no 'did it actually work?' anxiety. Real-time feedback turns a tool they check into a tool that talks to them. Concrete shape: WebSocket channel + optimistic UI + desktop notification fallback. Effort: human ~2 days / CC ~1 hour. Makes the product feel 10x more alive." +EXPANSIVE (aim for): "Imagine the moment a workflow finishes — the user sees the result instantly, no tab-switching, no polling, no 'did it actually work?' anxiety. Real-time feedback turns a tool they check into a tool that talks to them. Concrete shape: WebSocket channel + optimistic UI + desktop notification fallback. Effort: one agent session plus your review time. Makes the product feel 10x more alive." Both are outcome-framed. Only one makes the user feel the cathedral. Lead with the felt experience, close with concrete effort and impact. @@ -536,10 +536,11 @@ Think ahead to implementation: What decisions will need to be made during implem HOUR 4-5 (integration): What will surprise them? HOUR 6+ (polish/tests): What will they wish they'd planned for? ``` -NOTE: These represent human-team implementation hours. With CC + gstack, -6 hours of human implementation compresses to ~30-60 minutes. The decisions -are identical — the implementation speed is 10-20x faster. Always present -both scales when discussing effort. +NOTE: These represent human-team implementation hours — keep them ONLY as a +decision-ordering device (what breaks first, what integrates last). Do not +present effort in human-hours or hardcode a speed multiplier: agent speed +changes monthly and any fixed ratio drifts stale. When discussing effort, +size it as agent sessions plus the user's review time. Surface these as questions for the user NOW, not as "figure it out later." @@ -557,7 +558,6 @@ Context-dependent defaults: * Feature enhancement or iteration on existing system → default SELECTIVE EXPANSION * Bug fix or hotfix → default HOLD SCOPE * Refactor → default HOLD SCOPE -* Plan touching >15 files → suggest REDUCTION unless user pushes back * User says "go big" / "ambitious" / "cathedral" → EXPANSION, no question * User says "hold scope but tempt me" / "show me options" / "cherry-pick" → SELECTIVE EXPANSION, no question @@ -613,6 +613,19 @@ the user will see a plan whose review report is missing or stale, and will the report. The report is a separate, structured, table-bearing section that must be the file's terminal heading. + +## Capability priors (evidence dated 2026-08; re-verify before citing past 2027) + +Never justify narrowing scope, adding human-review stages, or steering the user to a "safer" variant by asserting that an LLM can't do something reliably. The measured 2026 reality, with sources, is the opposite of the 2023-24 folklore: + +- Pairwise semantic matching ("do these two announcements describe the same thing?") is at practical saturation: 92–99 F1 zero-shot on real-world benchmarks (OpenSanctions Pairs, 755K pairs: GPT-4o 98.95 F1), BEATING production rule engines by ~8 points — and the error direction is inverted: ~3% false merges vs 14% for the rules (arxiv.org/html/2603.11051; futuresearch.ai measured precision 1.000, "zero false merges," at ~$1.12/1k rows in production). +- LLM-as-judge on objective correctness reaches Cohen's kappa ≈ 0.87 (21-judge study, 541K judgments, arxiv.org/html/2606.19544); verbosity bias has collapsed an order of magnitude below 2023 levels. +- Grounded-summarization hallucination is low single digits for current models (Vectara leaderboard) — model-dependent, so name the model, don't indict the class. + +What still genuinely fails, and may honestly shape a design: subjective preference judging under adversarial phrasing (kappa ~0.5), mega-schema extraction (hundreds of fields per document), and multi-document event clustering with small open models. Those are different task shapes — do not let them stand in for pairwise matching or objective judging. + +When a load-bearing capability is uncertain for the user's ACTUAL data: keep the ambitious design as the default, and put a 30-minute spike in the plan that measures it — ~50 hand-labeled examples as the calibration set, 85–90% agreement as the deploy gate (the 2026 practitioner standard). "Measure it this afternoon" is the modern answer; "downgrade the product" is the stale one. + ## Upstream judgment port: PR #679 @@ -648,7 +661,7 @@ The binding scale comes from the first available source: the printed `Scale:` he - `project`: run the specialist's default workflow, batching question rounds where its source authorizes smart skips; size the roadmap in weeks. - `product` and `venture`: the full specialist workflow and its complete question pressure apply; this rule removes nothing. -The scale also fixes a chain-wide question budget — a ceiling on individual questions (not rounds) counted across the entire invocation and everything it chains into, reviews included: five at `session` (a hackathon demo or a one-sitting toy gets five questions, total, ever), eight at `hobby`, twelve at `project`, uncapped at `product` and `venture`. Every handoff carries the scale, the time box, and the questions already spent; the receiving specialist deducts from the remaining budget, never restarts it. A specialist whose remaining budget is zero infers the answer from the prompt, the repository, and stated constraints, states the inference and its default in one line, and proceeds — it does not ask. Approval STOP gates (approve/revise the plan, authorize a mutation) are outside the budget; everything else, including "which option do you prefer" refinements, spends it. Spend the budget on the questions whose wrong answer is most expensive to reverse, earliest. +The chain-wide question budget defaults to ZERO, at every scale — full autonomy out of the box. The current generation of models infers the answers from the prompt, the repository, and stated constraints; state each inference and its default in one line the user can correct in passing, and proceed. Scope, ambition, and technical depth are never what a question is for: decide, state the choice with a one-line opt-down, and keep moving. Budget above zero comes from the person, not the product: read `cat "${GSTACK_HOME:-$HOME/.gstack}"/developer-profile.json 2>/dev/null`, take `declared.autonomy` (newest inferred as fallback; absent profile means autonomy 1.0), and allow `round((1 − autonomy) × 6)` individual questions across the entire chain — this invocation plus everything it hands off to, reviews included. Every handoff carries the scale, the time box, and the questions already spent; the receiving specialist deducts, never restarts. Approval STOP gates (approve/revise the plan, authorize a mutation — authority, never inference) and privacy-consent gates are outside the budget; a consent gate with no pending question to bundle into does not become its own turn — skip the gated action, note the skip in one line, continue. Whatever the budget, spend it on the fork whose wrong answer is most expensive to reverse, earliest — never on confirming what the model already knows. - Never run a questioning round merely to classify scale. Classify from the prompt and cheap repository evidence, defaulting unknown vectors low; a specialist's own later questions may raise the scale mid-session, and an upgrade restores the full workflow from that point. Review specialists spend question rounds on decisions, not ceremony — at every scale, and sharpest at `session`/`hobby`: diff --git a/skills/plan/references/legacy/plan-devex-review.md b/skills/plan/references/legacy/plan-devex-review.md index 5bdd46515b..249a72c2e6 100644 --- a/skills/plan/references/legacy/plan-devex-review.md +++ b/skills/plan/references/legacy/plan-devex-review.md @@ -692,7 +692,7 @@ The binding scale comes from the first available source: the printed `Scale:` he - `project`: run the specialist's default workflow, batching question rounds where its source authorizes smart skips; size the roadmap in weeks. - `product` and `venture`: the full specialist workflow and its complete question pressure apply; this rule removes nothing. -The scale also fixes a chain-wide question budget — a ceiling on individual questions (not rounds) counted across the entire invocation and everything it chains into, reviews included: five at `session` (a hackathon demo or a one-sitting toy gets five questions, total, ever), eight at `hobby`, twelve at `project`, uncapped at `product` and `venture`. Every handoff carries the scale, the time box, and the questions already spent; the receiving specialist deducts from the remaining budget, never restarts it. A specialist whose remaining budget is zero infers the answer from the prompt, the repository, and stated constraints, states the inference and its default in one line, and proceeds — it does not ask. Approval STOP gates (approve/revise the plan, authorize a mutation) are outside the budget; everything else, including "which option do you prefer" refinements, spends it. Spend the budget on the questions whose wrong answer is most expensive to reverse, earliest. +The chain-wide question budget defaults to ZERO, at every scale — full autonomy out of the box. The current generation of models infers the answers from the prompt, the repository, and stated constraints; state each inference and its default in one line the user can correct in passing, and proceed. Scope, ambition, and technical depth are never what a question is for: decide, state the choice with a one-line opt-down, and keep moving. Budget above zero comes from the person, not the product: read `cat "${GSTACK_HOME:-$HOME/.gstack}"/developer-profile.json 2>/dev/null`, take `declared.autonomy` (newest inferred as fallback; absent profile means autonomy 1.0), and allow `round((1 − autonomy) × 6)` individual questions across the entire chain — this invocation plus everything it hands off to, reviews included. Every handoff carries the scale, the time box, and the questions already spent; the receiving specialist deducts, never restarts. Approval STOP gates (approve/revise the plan, authorize a mutation — authority, never inference) and privacy-consent gates are outside the budget; a consent gate with no pending question to bundle into does not become its own turn — skip the gated action, note the skip in one line, continue. Whatever the budget, spend it on the fork whose wrong answer is most expensive to reverse, earliest — never on confirming what the model already knows. - Never run a questioning round merely to classify scale. Classify from the prompt and cheap repository evidence, defaulting unknown vectors low; a specialist's own later questions may raise the scale mid-session, and an upgrade restores the full workflow from that point. Review specialists spend question rounds on decisions, not ceremony — at every scale, and sharpest at `session`/`hobby`: diff --git a/skills/plan/references/legacy/plan-eng-review.md b/skills/plan/references/legacy/plan-eng-review.md index 1598b216b0..d7a8171ebf 100644 --- a/skills/plan/references/legacy/plan-eng-review.md +++ b/skills/plan/references/legacy/plan-eng-review.md @@ -125,7 +125,7 @@ Say to the user via AskUserQuestion: > "No design doc found for this branch. `$plan --mode Discovery --module office-hours` produces a structured problem > statement, premise challenge, and explored alternatives — it gives this review much -> sharper input to work with. Takes about 10 minutes. The design doc is per-feature, +> sharper input to work with. Takes a few minutes. The design doc is per-feature, > not per-product — it captures the thinking behind this specific change." Options: @@ -180,7 +180,7 @@ If none was produced (user may have cancelled), proceed with standard review. Before reviewing anything, answer these questions: 1. **What existing code already partially or fully solves each sub-problem?** Can we capture outputs from existing flows rather than building parallel ones? 2. **What is the minimum set of changes that achieves the stated goal?** Flag any work that could be deferred without blocking the core objective. Be ruthless about scope creep. -3. **Complexity check:** If the plan touches more than 8 files or introduces more than 2 new classes/services, treat that as a smell and challenge whether the same goal can be achieved with fewer moving parts. +3. **Complexity check:** File count is not complexity — a current agent one-shots a 20-file change, and challenging a plan for its diff size is a human-era reflex that shrinks products for no reason. The smell is UNJUSTIFIED STRUCTURE: a new class, service, queue, or abstraction layer whose job an existing component or a built-in already does. Challenge each new moving part to justify itself against the goal; never challenge the goal for having moving parts. 4. **Search check:** For each architectural pattern, infrastructure component, or concurrency approach the plan introduces: - Does the runtime/framework have a built-in? Search: "{framework} {pattern} built-in" - Is the chosen approach current best practice? Search: "{pattern} best practice {current year}" @@ -199,11 +199,11 @@ Before reviewing anything, answer these questions: - How will users download or install it (GitHub Releases, package manager, container registry)? If the plan defers distribution, flag it explicitly in the "NOT in scope" section — don't let it silently drop. -If the complexity check triggers (8+ files or 2+ new classes/services), STOP before any review-section work. Call AskUserQuestion: name what's overbuilt, propose a minimal version that achieves the core goal, ask whether to reduce or proceed as-is. The AskUserQuestion call is a tool_use, not prose — call the tool directly. +If the complexity check finds unjustified structure, STOP before any review-section work. Call AskUserQuestion: name the specific component that duplicates an existing one or a built-in, propose the version without it, ask whether to reduce or proceed as-is. (Scope and ambition are not what this gate is for — only structure that does a job something else already does.) The AskUserQuestion call is a tool_use, not prose — call the tool directly. **STOP.** Do NOT proceed to Section 1 (Architecture review), edit the plan file with a proposed scope reduction, or call ExitPlanMode until the user responds. Naming the 80% solution in chat prose and continuing — or loading the AskUserQuestion schema via ToolSearch and then never invoking it — is the failure mode this gate exists to prevent. -If the complexity check does not trigger, present your Step 0 findings and proceed directly to Section 1. +If the complexity check finds no unjustified structure, present your Step 0 findings and proceed directly to Section 1. Always work through the full interactive review: one section at a time (Architecture → Code Quality → Tests → Performance) with at most 8 top issues per section. @@ -247,6 +247,19 @@ the user will see a plan whose review report is missing or stale, and will the report. The report is a separate, structured, table-bearing section that must be the file's terminal heading. + +## Capability priors (evidence dated 2026-08; re-verify before citing past 2027) + +Never justify narrowing scope, adding human-review stages, or steering the user to a "safer" variant by asserting that an LLM can't do something reliably. The measured 2026 reality, with sources, is the opposite of the 2023-24 folklore: + +- Pairwise semantic matching ("do these two announcements describe the same thing?") is at practical saturation: 92–99 F1 zero-shot on real-world benchmarks (OpenSanctions Pairs, 755K pairs: GPT-4o 98.95 F1), BEATING production rule engines by ~8 points — and the error direction is inverted: ~3% false merges vs 14% for the rules (arxiv.org/html/2603.11051; futuresearch.ai measured precision 1.000, "zero false merges," at ~$1.12/1k rows in production). +- LLM-as-judge on objective correctness reaches Cohen's kappa ≈ 0.87 (21-judge study, 541K judgments, arxiv.org/html/2606.19544); verbosity bias has collapsed an order of magnitude below 2023 levels. +- Grounded-summarization hallucination is low single digits for current models (Vectara leaderboard) — model-dependent, so name the model, don't indict the class. + +What still genuinely fails, and may honestly shape a design: subjective preference judging under adversarial phrasing (kappa ~0.5), mega-schema extraction (hundreds of fields per document), and multi-document event clustering with small open models. Those are different task shapes — do not let them stand in for pairwise matching or objective judging. + +When a load-bearing capability is uncertain for the user's ACTUAL data: keep the ambitious design as the default, and put a 30-minute spike in the plan that measures it — ~50 hand-labeled examples as the calibration set, 85–90% agreement as the deploy gate (the 2026 practitioner standard). "Measure it this afternoon" is the modern answer; "downgrade the product" is the stale one. + ## Upstream judgment port: PR #679 @@ -302,7 +315,7 @@ The binding scale comes from the first available source: the printed `Scale:` he - `project`: run the specialist's default workflow, batching question rounds where its source authorizes smart skips; size the roadmap in weeks. - `product` and `venture`: the full specialist workflow and its complete question pressure apply; this rule removes nothing. -The scale also fixes a chain-wide question budget — a ceiling on individual questions (not rounds) counted across the entire invocation and everything it chains into, reviews included: five at `session` (a hackathon demo or a one-sitting toy gets five questions, total, ever), eight at `hobby`, twelve at `project`, uncapped at `product` and `venture`. Every handoff carries the scale, the time box, and the questions already spent; the receiving specialist deducts from the remaining budget, never restarts it. A specialist whose remaining budget is zero infers the answer from the prompt, the repository, and stated constraints, states the inference and its default in one line, and proceeds — it does not ask. Approval STOP gates (approve/revise the plan, authorize a mutation) are outside the budget; everything else, including "which option do you prefer" refinements, spends it. Spend the budget on the questions whose wrong answer is most expensive to reverse, earliest. +The chain-wide question budget defaults to ZERO, at every scale — full autonomy out of the box. The current generation of models infers the answers from the prompt, the repository, and stated constraints; state each inference and its default in one line the user can correct in passing, and proceed. Scope, ambition, and technical depth are never what a question is for: decide, state the choice with a one-line opt-down, and keep moving. Budget above zero comes from the person, not the product: read `cat "${GSTACK_HOME:-$HOME/.gstack}"/developer-profile.json 2>/dev/null`, take `declared.autonomy` (newest inferred as fallback; absent profile means autonomy 1.0), and allow `round((1 − autonomy) × 6)` individual questions across the entire chain — this invocation plus everything it hands off to, reviews included. Every handoff carries the scale, the time box, and the questions already spent; the receiving specialist deducts, never restarts. Approval STOP gates (approve/revise the plan, authorize a mutation — authority, never inference) and privacy-consent gates are outside the budget; a consent gate with no pending question to bundle into does not become its own turn — skip the gated action, note the skip in one line, continue. Whatever the budget, spend it on the fork whose wrong answer is most expensive to reverse, earliest — never on confirming what the model already knows. - Never run a questioning round merely to classify scale. Classify from the prompt and cheap repository evidence, defaulting unknown vectors low; a specialist's own later questions may raise the scale mid-session, and an upgrade restores the full workflow from that point. Review specialists spend question rounds on decisions, not ceremony — at every scale, and sharpest at `session`/`hobby`: diff --git a/skills/plan/references/legacy/plan-tune.md b/skills/plan/references/legacy/plan-tune.md index 37a4d19c9b..a0888b0d5b 100644 --- a/skills/plan/references/legacy/plan-tune.md +++ b/skills/plan/references/legacy/plan-tune.md @@ -20,7 +20,9 @@ memorize them. **v1 scope (observational):** typed question registry, per-question explicit preferences, question logging, dual-track profile (declared + inferred), -plain-English inspection. No skills adapt behavior based on the profile yet. +plain-English inspection. The profile is consumed: office-hours scales its +chain-wide question budget by the autonomy dimension (high autonomy halves +it, low adds two), so tuning here changes how much /plan asks. Canonical reference: `docs/designs/PLAN_TUNING_V0.md`. diff --git a/skills/plan/references/legacy/spec.md b/skills/plan/references/legacy/spec.md index 6acd376ec4..1ffee0abac 100644 --- a/skills/plan/references/legacy/spec.md +++ b/skills/plan/references/legacy/spec.md @@ -71,7 +71,7 @@ confirm: "Flags: dedupe=ON, gate=ON, audit=OFF, execute=auto (plan mode = ...)." 4. **Why now?** (blocking other work? costing money? correctness bug? compliance risk?) 5. **How will we know it's done?** (observable, measurable outcome — not vibes) -Do NOT proceed until all five are answered without hand-waving. +Answer all five YOURSELF first — from the prompt, the design doc, the code, and the repo history. Present your answers as stated assumptions inside the draft; the user corrects in passing. Ask only for an answer that is genuinely unknowable from those sources (an unstated business constraint, an external account) and fits the chain question budget. Never proceed with a five-way hand-wave — but interrogating the user for what the model can read is the old failure, not rigor. **Step 1b (--dedupe is ON by default):** Before Phase 4, run dedupe check. Extract 2-4 keywords from the user's request and the working title you have in mind, then: @@ -110,7 +110,7 @@ Ask until you can answer: 4. **What's the smallest version that delivers the value?** Always find the MVP cut. 5. **What are the failure modes and rollback options?** What breaks if shipped wrong? -Do NOT proceed until scope is locked. +Lock scope by DECIDING it: state the cut you chose and why, with the one-line opt-down. A question here spends chain budget and is only for a true authority fork (whose data, whose money, which account), never for sizing. ### Phase 3: Technical Interrogation (HARD requirement: read code first) @@ -766,9 +766,11 @@ Add to the standard template: 2. **Don't ask questions you can answer by reading code.** Read first, ask informed. 3. **Don't include code unless it removes ambiguity.** Schemas and API shapes yes. Random implementation snippets no. -4. **Don't leave design decisions for the implementer.** Decide them in conversation. +4. **Don't leave design decisions for the implementer.** Decide them YOURSELF in the spec and mark each one `DECIDED: — `; reserve conversation for authority calls (spend, publish, data ownership) the model must not make alone. 5. **Flag when something should be multiple issues.** Propose epic + children if scope - has natural seams. Individual issues should be completable in 1-3 days. + has natural seams. Size an issue by review-ability — one coherent change a human + can review in one sitting — not by human build-days; an agent's build time is + minutes-to-hours and makes day-counts meaningless. 6. **Match template to content.** Bug fixes don't need architecture diagrams. New subsystems don't need "Current vs Expected Behavior." Use what applies. 7. **Verify before asserting.** Read the file first. Cite what you found. @@ -807,6 +809,19 @@ Add to the standard template: auto-closes the source issue. Conditional — partial PRs do NOT auto-close (codex F4). Branch-name inference is NOT used (codex F3). + +## Capability priors (evidence dated 2026-08; re-verify before citing past 2027) + +Never justify narrowing scope, adding human-review stages, or steering the user to a "safer" variant by asserting that an LLM can't do something reliably. The measured 2026 reality, with sources, is the opposite of the 2023-24 folklore: + +- Pairwise semantic matching ("do these two announcements describe the same thing?") is at practical saturation: 92–99 F1 zero-shot on real-world benchmarks (OpenSanctions Pairs, 755K pairs: GPT-4o 98.95 F1), BEATING production rule engines by ~8 points — and the error direction is inverted: ~3% false merges vs 14% for the rules (arxiv.org/html/2603.11051; futuresearch.ai measured precision 1.000, "zero false merges," at ~$1.12/1k rows in production). +- LLM-as-judge on objective correctness reaches Cohen's kappa ≈ 0.87 (21-judge study, 541K judgments, arxiv.org/html/2606.19544); verbosity bias has collapsed an order of magnitude below 2023 levels. +- Grounded-summarization hallucination is low single digits for current models (Vectara leaderboard) — model-dependent, so name the model, don't indict the class. + +What still genuinely fails, and may honestly shape a design: subjective preference judging under adversarial phrasing (kappa ~0.5), mega-schema extraction (hundreds of fields per document), and multi-document event clustering with small open models. Those are different task shapes — do not let them stand in for pairwise matching or objective judging. + +When a load-bearing capability is uncertain for the user's ACTUAL data: keep the ambitious design as the default, and put a 30-minute spike in the plan that measures it — ~50 hand-labeled examples as the calibration set, 85–90% agreement as the deploy gate (the 2026 practitioner standard). "Measure it this afternoon" is the modern answer; "downgrade the product" is the stale one. + ## Upstream judgment port: PR #679 @@ -832,7 +847,7 @@ The binding scale comes from the first available source: the printed `Scale:` he - `project`: run the specialist's default workflow, batching question rounds where its source authorizes smart skips; size the roadmap in weeks. - `product` and `venture`: the full specialist workflow and its complete question pressure apply; this rule removes nothing. -The scale also fixes a chain-wide question budget — a ceiling on individual questions (not rounds) counted across the entire invocation and everything it chains into, reviews included: five at `session` (a hackathon demo or a one-sitting toy gets five questions, total, ever), eight at `hobby`, twelve at `project`, uncapped at `product` and `venture`. Every handoff carries the scale, the time box, and the questions already spent; the receiving specialist deducts from the remaining budget, never restarts it. A specialist whose remaining budget is zero infers the answer from the prompt, the repository, and stated constraints, states the inference and its default in one line, and proceeds — it does not ask. Approval STOP gates (approve/revise the plan, authorize a mutation) are outside the budget; everything else, including "which option do you prefer" refinements, spends it. Spend the budget on the questions whose wrong answer is most expensive to reverse, earliest. +The chain-wide question budget defaults to ZERO, at every scale — full autonomy out of the box. The current generation of models infers the answers from the prompt, the repository, and stated constraints; state each inference and its default in one line the user can correct in passing, and proceed. Scope, ambition, and technical depth are never what a question is for: decide, state the choice with a one-line opt-down, and keep moving. Budget above zero comes from the person, not the product: read `cat "${GSTACK_HOME:-$HOME/.gstack}"/developer-profile.json 2>/dev/null`, take `declared.autonomy` (newest inferred as fallback; absent profile means autonomy 1.0), and allow `round((1 − autonomy) × 6)` individual questions across the entire chain — this invocation plus everything it hands off to, reviews included. Every handoff carries the scale, the time box, and the questions already spent; the receiving specialist deducts, never restarts. Approval STOP gates (approve/revise the plan, authorize a mutation — authority, never inference) and privacy-consent gates are outside the budget; a consent gate with no pending question to bundle into does not become its own turn — skip the gated action, note the skip in one line, continue. Whatever the budget, spend it on the fork whose wrong answer is most expensive to reverse, earliest — never on confirming what the model already knows. - Never run a questioning round merely to classify scale. Classify from the prompt and cheap repository evidence, defaulting unknown vectors low; a specialist's own later questions may raise the scale mid-session, and an upgrade restores the full workflow from that point. Review specialists spend question rounds on decisions, not ceremony — at every scale, and sharpest at `session`/`hobby`: diff --git a/skills/review/references/legacy/review.md b/skills/review/references/legacy/review.md index fbc0109561..69411cc2df 100644 --- a/skills/review/references/legacy/review.md +++ b/skills/review/references/legacy/review.md @@ -808,7 +808,7 @@ The binding scale comes from the first available source: the printed `Scale:` he - `project`: run the specialist's default workflow, batching question rounds where its source authorizes smart skips; size the roadmap in weeks. - `product` and `venture`: the full specialist workflow and its complete question pressure apply; this rule removes nothing. -The scale also fixes a chain-wide question budget — a ceiling on individual questions (not rounds) counted across the entire invocation and everything it chains into, reviews included: five at `session` (a hackathon demo or a one-sitting toy gets five questions, total, ever), eight at `hobby`, twelve at `project`, uncapped at `product` and `venture`. Every handoff carries the scale, the time box, and the questions already spent; the receiving specialist deducts from the remaining budget, never restarts it. A specialist whose remaining budget is zero infers the answer from the prompt, the repository, and stated constraints, states the inference and its default in one line, and proceeds — it does not ask. Approval STOP gates (approve/revise the plan, authorize a mutation) are outside the budget; everything else, including "which option do you prefer" refinements, spends it. Spend the budget on the questions whose wrong answer is most expensive to reverse, earliest. +The chain-wide question budget defaults to ZERO, at every scale — full autonomy out of the box. The current generation of models infers the answers from the prompt, the repository, and stated constraints; state each inference and its default in one line the user can correct in passing, and proceed. Scope, ambition, and technical depth are never what a question is for: decide, state the choice with a one-line opt-down, and keep moving. Budget above zero comes from the person, not the product: read `cat "${GSTACK_HOME:-$HOME/.gstack}"/developer-profile.json 2>/dev/null`, take `declared.autonomy` (newest inferred as fallback; absent profile means autonomy 1.0), and allow `round((1 − autonomy) × 6)` individual questions across the entire chain — this invocation plus everything it hands off to, reviews included. Every handoff carries the scale, the time box, and the questions already spent; the receiving specialist deducts, never restarts. Approval STOP gates (approve/revise the plan, authorize a mutation — authority, never inference) and privacy-consent gates are outside the budget; a consent gate with no pending question to bundle into does not become its own turn — skip the gated action, note the skip in one line, continue. Whatever the budget, spend it on the fork whose wrong answer is most expensive to reverse, earliest — never on confirming what the model already knows. - Never run a questioning round merely to classify scale. Classify from the prompt and cheap repository evidence, defaulting unknown vectors low; a specialist's own later questions may raise the scale mid-session, and an upgrade restores the full workflow from that point. Review specialists spend question rounds on decisions, not ceremony — at every scale, and sharpest at `session`/`hobby`: