diff --git a/commands/deep-research.md b/commands/deep-research.md index c9b2796..e3b8155 100644 --- a/commands/deep-research.md +++ b/commands/deep-research.md @@ -1,5 +1,5 @@ --- -description: ACH-enhanced deep research — perspective discovery, competing hypotheses, directed disconfirmation, evidence matrix, sensitivity analysis. +description: ACH-enhanced deep research — delegates to deterministic YAML workflow. --- # Deep Research @@ -12,139 +12,13 @@ Use regular `/devkit:research` for quick lookups. Use this when: - You need confidence calibration, not just an answer - The user says "deep research", "validate", "make sure this is right" -## Step 0: Harness Detection +## Invoke -```bash -if command -v devkit >/dev/null 2>&1; then - echo "Go harness detected — delegating to devkit workflow deep-research." - devkit workflow deep-research "{input}" - exit 0 -fi ``` - -## Step 1: Clarify - -``` -The user wants to deep-research: {input} - -Use AskUserQuestion to clarify: -- What specifically are we trying to learn? -- What constraints matter? -- What would a wrong answer cost? -- Any sources they already know about? - -Restate the research question precisely. -``` - -## Step 2: Discover Perspectives - -Search for 2-3 overview articles on the topic. Fetch with Jina Reader (`WebFetch https://r.jina.ai/{url}`). Extract major schools of thought, key voices, known debates. Summarize immediately — do not carry raw content forward. - -## Step 3: Decompose into Sub-Questions - -Break the question into 5-8 sub-questions with explicit retrieval goals and perspective labels. At least 2 queries must seek **disconfirming** evidence. - -## Step 4: Search (Parallel Fan-Out) - -**[PARALLEL]** Launch searches concurrently using the `researcher` agent (max 3 per batch): - -``` -Task: Execute web search for sub-question. -Agent: researcher -Input: Query + Goal + Perspective -``` - -## Budget - -- **Token budget:** ~400k tokens. -- **Early exit:** Only if the question turns out trivial after perspective discovery. - -## Step 5: Summarize and Extract Claims - -Fetch top 5-8 URLs with Jina Reader. Extract atomic claims (subject-predicate-object). Summarize immediately. Aim for 3-8 claims per source. - -## Step 6: Generate Competing Hypotheses - -Generate 2-4 competing hypotheses. Include at least one contrarian hypothesis. - -## Step 7: Directed Disconfirmation - -For EACH hypothesis, search specifically for evidence that would DISPROVE it. This is the critical ACH step — you're trying to kill each hypothesis, not confirm it. - -## Step 8: Build Evidence Matrix - -``` -| Evidence | H1 | H2 | H3 | -|----------|----|----|-----| -| Claim [source] | CC | I | N | - -CC=Strongly Consistent, C=Consistent, N=Neutral, I=Inconsistent, II=Strongly Inconsistent -``` - -Score by FEWEST inconsistencies (not most consistencies). - -## Step 8.5: Adversarial Debate (optional — use when hypotheses are close or stakes are high) - -When two or more hypotheses survive with similar scores in the evidence matrix, run an adversarial refinement cycle to stress-test them. Use the completed matrix as input. - -1. **Advocate** — For each surviving hypothesis, write the strongest possible case. Cite specific evidence from the matrix. Assume this hypothesis is correct and explain away inconsistencies. - -2. **Critic** — For each advocacy, write a targeted attack. Find the weakest link in the argument. Identify what the advocate glossed over or explained away too easily. Name the single observation that would kill this hypothesis. - -3. **Synthesize** — Given the advocacy and critique for all hypotheses, ask: Is there a composite hypothesis that accounts for more evidence than any individual one? If so, the composite replaces its parent hypotheses — rescore it against the evidence matrix as a new candidate. - -4. **Judge** — Present the surviving candidates (original + any composite) with randomized labels (Candidate A, B, C — not in hypothesis order) as a heuristic to reduce anchoring bias. Note: in a single-agent context this is a nudge, not a true blind. Evaluate using the evidence matrix on: - - Fewest inconsistencies in the matrix (primary) - - Least reliance on coincidence - - Most falsifiable (tiebreaker — prefer hypotheses that can be tested) - Pick a winner. If no clear winner, note the deadlock and carry both forward with explicit uncertainty. - -Skip this step if: one hypothesis has 2+ fewer inconsistencies than the runner-up in the matrix, or the research question is informational rather than decision-driving. - -## Step 9: Sensitivity Check - -Identify linchpin evidence — what single fact, if wrong, would change the conclusion? - -## Step 10: Self-Critique - -Review for: genuine disconfirmation effort, missed perspectives, source over-weighting, fairness to opposing views. One more search round if gaps found. - -## Step 11: Synthesize - -``` -## Deep Research: {question} - -### Direct Answer -{answer with confidence: HIGH / MEDIUM / LOW} - -### Competing Hypotheses -{for each: statement, supporting evidence, disconfirming evidence, status} - -### Evidence Matrix Summary -{matrix or prose summary} - -### Key Findings -{CONFIRMED / CONTESTED / UNCORROBORATED claims with sources} - -### Sensitivity Analysis -{linchpin evidence, confidence fragility} - -### What Would Change This Conclusion -{specific evidence that would flip the answer} - -### Recommendation -{what to do, why, with calibrated confidence} +devkit workflow run deep-research "{input}" ``` -## Rules +If `devkit workflow` is not available, activate the `/devkit:deep-research` skill which contains a condensed fallback for manual execution. -- Perspectives first — ground queries in real viewpoints -- Disconfirm > confirm — try to KILL hypotheses -- Summarize immediately — no raw content carried forward -- Evidence matrix is mandatory -- Sensitivity check is mandatory -- Self-critique before output -- Cite everything -- Be honest about uncertainty -- Use adversarial debate when hypotheses score similarly in the evidence matrix — don't just pick the first plausible one -- Judge evaluations use randomized labels as an anchoring-bias heuristic (not a true blind in single-agent context) +The YAML workflow (`workflows/deep-research.yml`) enforces the full ACH sequence deterministically: +clarify → perspectives → decompose → parallel search → extract claims → hypotheses → disconfirm → evidence matrix → self-critique → synthesize. diff --git a/docs/superpowers/specs/2026-04-09-deterministic-workflow-conversion-design.md b/docs/superpowers/specs/2026-04-09-deterministic-workflow-conversion-design.md new file mode 100644 index 0000000..c9b062c --- /dev/null +++ b/docs/superpowers/specs/2026-04-09-deterministic-workflow-conversion-design.md @@ -0,0 +1,409 @@ +# Deterministic Workflow Conversion + +Convert all rigid commands from LLM-interpreted markdown to Go-engine-driven YAML workflows. Same steps, same results, same triggers — Claude can't skip steps. + +## Problem + +24 markdown command files define multi-step procedures. Claude interprets these as prompts and routinely: +- Skips verification steps ("I already know this") +- Fabricates baselines instead of running tools +- Jumps to workarounds when a step fails instead of retrying correctly +- Skips disconfirmation in research (confirms own hypothesis) +- Presents results without running the actual commands + +## Solution + +Replace markdown command logic with YAML workflows executed by the Go engine. The engine owns the sequence — `command` steps run shell commands deterministically, `gate` checks enforce quality after each loop iteration, and Claude only handles the thinking within each step. + +## Engine Addition + +One new primitive needed: + +### `expect` field on command steps + +```yaml +- id: repro + command: "{{input}}" + expect: failure # step fails if exit code is 0 +``` + +Values: `success` (default — non-zero exit is informational), `failure` (step fails if exit code is 0). Enables bugfix reproduction gates: repro must fail before fix, pass after. + +## Conversion Plan + +### PR 1: Research workflows + +**research.yml** +```yaml +steps: + - id: clarify + model: smart + prompt: | + Clarify the research question. Identify 3-5 sub-questions. + {{input}} + + - id: search + model: smart + prompt: | + Search for answers to each sub-question. Use web search, grep, + and file reads. Do NOT answer from memory. + + Sub-questions: {{clarify}} + + For each finding, cite the source. + loop: + max: 5 + until: SUFFICIENT_EVIDENCE + + - id: corroborate + model: smart + prompt: | + Cross-check findings against 2+ independent sources. + Flag anything with only one source. + + Findings: {{search}} + + - id: synthesize + model: smart + prompt: | + Synthesize findings into a clear answer. + Lead with the conclusion, then supporting evidence. + + Corroborated findings: {{corroborate}} +``` + +**deep-research.yml** +```yaml +steps: + - id: clarify + model: smart + prompt: | + Clarify the research question. Identify perspectives that + might disagree. {{input}} + + - id: search + model: smart + prompt: | + Search exhaustively. Use web search, grep, file reads. + Do NOT answer from memory. + + Question: {{clarify}} + loop: + max: 8 + until: SUFFICIENT_EVIDENCE + + - id: hypotheses + model: smart + prompt: | + Form 2-3 competing hypotheses from the evidence. + {{search}} + + - id: disconfirm + model: smart + prompt: | + For EACH hypothesis, actively search for evidence that + DISPROVES it. Do not confirm — try to break each one. + + Hypotheses: {{hypotheses}} + loop: + max: 5 + until: DISCONFIRMATION_COMPLETE + + - id: matrix + model: smart + prompt: | + Build an evidence matrix: hypotheses as columns, evidence + as rows. Mark consistent/inconsistent/neutral. + + Evidence: {{search}} + Disconfirmation: {{disconfirm}} + + - id: synthesize + model: smart + prompt: | + Synthesize. Which hypothesis survives disconfirmation best? + Rate confidence. Flag remaining uncertainties. + + Matrix: {{matrix}} +``` + +### PR 2: Self-improvement loops + +All follow the same pattern — `command` step for baseline, `gate` on the loop: + +**self-test.yml** (example — others are identical pattern) +```yaml +steps: + - id: baseline + command: "{{input}} 2>&1 || true" + + - id: improve + model: smart + prompt: | + Current test output: + {{baseline}} + + Generate or improve tests to increase coverage. + Focus on untested code paths and edge cases. + ONE test file at a time. + loop: + max: 10 + until: "exit code: 0" + gate: "{{input}}" + + - id: verify + command: "{{input}} 2>&1 || true" + + - id: summary + model: fast + prompt: | + Test improvement session complete. + Before: {{baseline}} + After: {{verify}} + Summarize what was added. +``` + +**self-perf.yml**, **self-migrate.yml**, **self-improve.yml** — same structure, different prompts within each step. + +### PR 3: Lifecycle gates + +**bugfix.yml** +```yaml +steps: + - id: repro + command: "{{input}} 2>&1 || true" + + - id: diagnose + model: smart + prompt: | + Bug reproduction output: + {{repro}} + + Diagnose the root cause. Read relevant source files. + Identify the exact location of the bug. + + - id: fix + model: smart + prompt: | + Root cause: {{diagnose}} + + Fix the bug. Minimal change only. + Don't refactor surrounding code. + + - id: verify + command: "{{input}} 2>&1 || true" + + - id: check + model: fast + prompt: | + Before fix: {{repro}} + After fix: {{verify}} + + Did the fix resolve the bug? Say FIXED or NOT_FIXED. + branch: + - when: NOT_FIXED + goto: diagnose + - when: FIXED + goto: summary + + - id: summary + model: fast + prompt: | + Bug fix complete. + Reproduction: {{repro}} + Diagnosis: {{diagnose}} + Verification: {{verify}} + Summarize what was wrong and what was changed. +``` + +**feature.yml** +```yaml +steps: + - id: explore + model: smart + prompt: | + Explore the codebase to understand relevant patterns, + conventions, and architecture. Identify 5-10 key files. + {{input}} + + - id: design + model: smart + prompt: | + Based on codebase exploration: + {{explore}} + + Propose 2-3 design approaches with trade-offs. + Recommend one. Include data flow and component boundaries. + + - id: plan + model: smart + prompt: | + Design: {{design}} + + Create a numbered implementation plan. + Order by dependency. Each step should be one logical change. + + - id: implement + model: smart + prompt: | + Plan: {{plan}} + + Implement the next unfinished step. + Small, focused changes. Follow existing patterns. + loop: + max: 15 + until: ALL_STEPS_COMPLETE + + - id: test + model: smart + prompt: | + Implementation complete. + + Write tests for the new feature. + Run them and fix any failures. + loop: + max: 5 + until: ALL_PASSING + + - id: summary + model: fast + prompt: | + Feature complete. + Design: {{design}} + Implementation: {{implement}} + Tests: {{test}} + Summarize what was built. +``` + +### PR 4: Shipping + utility + +**pr-ready.yml** +```yaml +steps: + - id: lint + command: "{{input}} 2>&1 || true" + + - id: lint-check + model: fast + prompt: | + Lint output: {{lint}} + Are there errors? Say CLEAN or HAS_ERRORS. + branch: + - when: HAS_ERRORS + goto: lint-fix + - when: CLEAN + goto: test + + - id: lint-fix + model: smart + prompt: | + Fix lint errors: {{lint}} + loop: + max: 5 + until: "exit code: 0" + gate: "{{input}}" + + - id: test + command: "{{test_command}} 2>&1 || true" + + - id: security + model: smart + prompt: | + Review changed files for security issues. + Check OWASP top 10 patterns. + + - id: changelog + model: fast + prompt: | + Generate changelog entry from git diff. + + - id: create-pr + model: smart + prompt: | + Create the PR with changelog and summary. +``` + +**audit.yml** — all `command` steps for tool execution: +```yaml +steps: + - id: detect + command: | + echo "go:$(test -f go.mod && echo yes || echo no)" + echo "node:$(test -f package.json && echo yes || echo no)" + echo "python:$(test -f requirements.txt -o -f pyproject.toml && echo yes || echo no)" + echo "rust:$(test -f Cargo.toml && echo yes || echo no)" + + - id: deps + model: smart + prompt: | + Detected ecosystems: {{detect}} + Run dependency audit commands for each detected ecosystem. + Report vulnerabilities, outdated packages, and license issues. + + - id: lint + model: smart + prompt: | + Run linters for detected ecosystems: {{detect}} + + - id: report + model: fast + prompt: | + Compile audit report. + Dependencies: {{deps}} + Lint: {{lint}} + Score overall health. +``` + +**tri-review.yml**, **tri-debug.yml**, **tri-security.yml**, **tri-dispatch.yml**, **tri-test-gen.yml** — add `command` step to capture diff/context deterministically before dispatch. + +### PR 5: Trim commands + thin wrappers + docs + +**Delete** these markdown command files (logic lives in YAML): +- autoloop.md, bugfix.md, deep-research.md, feature.md, refactor.md +- self-audit.md, self-improve.md, self-lint.md, self-perf.md, self-test.md, self-migrate.md +- tri-debug.md, tri-dispatch.md, tri-review.md, tri-security.md, tri-test-gen.md +- audit.md, decompose.md, pr-ready.md, repo-map.md + +**Keep as thin wrappers** (tab-completable, one-liner pointing to workflow): +- `tri-review.md` → "Run `devkit workflow tri-review`" +- `tri-debug.md` → "Run `devkit workflow tri-debug`" +- `tri-security.md` → "Run `devkit workflow tri-security`" +- `pr-ready.md` → "Run `devkit workflow pr-ready`" +- `pr-monitor.md` → stays as-is (no YAML equivalent yet) + +**Keep as-is** (not workflows): +- `status.md` — diagnostic +- `setup-rules.md` — one-time setup +- `workflow.md` — entry point + +**Context-activated** (move trigger logic to `skills/`): +- research, deep-research, bugfix, feature, refactor, self-test, self-lint, self-improve, audit, decompose + +**Docs updates:** +- README: Update to reflect ~8 slash commands +- creating-workflows skill: Document `expect` field +- ROADMAP: Add deterministic conversion as completed milestone + +## What Does NOT Change + +- 10 hooks (already deterministic shell scripts) +- 6 agents (used by workflows, not changed) +- Coding principle skills (clean-code, dry, yagni — judgment-based by design) +- Tool skills (gcli, creating-workflows) +- Companion plugins (superpowers, pr-review-toolkit, hookify, etc.) + +## Success Criteria + +- All 24 commands covered: converted to YAML, kept as thin wrapper, or kept as-is +- Zero duplicated logic between markdown and YAML +- All tests pass (`go test ./...`) +- All existing YAML workflows still parse (`TestParseRealWorkflows`) +- Tab-completion works for the ~8 kept commands +- Context-activation works for migrated workflows + +## Token Efficiency + +- `command` steps cost $0 (shell execution, no LLM) +- Baselines, linter runs, test runs, diff captures all move to `command` steps +- LLM only invoked for thinking steps (diagnosis, design, synthesis) +- Gate failures revert and retry — no tokens wasted on broken iterations diff --git a/skills/deep-research/SKILL.md b/skills/deep-research/SKILL.md index c355430..28bd45e 100644 --- a/skills/deep-research/SKILL.md +++ b/skills/deep-research/SKILL.md @@ -3,231 +3,42 @@ name: deep-research description: Deep research with Analysis of Competing Hypotheses — use when asked to do deep research, deeply investigate, validate claims, or when correctness is critical and the user wants rigorous analysis with disconfirmation testing. --- -# Deep Research Workflow +# Deep Research -ACH-enhanced research: clarify → discover perspectives → decompose → search → summarize → generate hypotheses → disconfirm → build evidence matrix → self-critique → synthesize. +ACH-enhanced deterministic research: clarify → perspectives → decompose → parallel search → extract claims → hypotheses → disconfirm → evidence matrix (with sensitivity check) → self-critique → synthesize. -This is the rigorous path. It costs more tokens but produces higher-confidence results by actively trying to disprove answers rather than just confirming them. +Costs more tokens (~400k budget) but produces higher-confidence results by actively trying to disprove answers. -## Step 1: Clarify +## Invoke -``` -The user wants to deep-research: {input} - -Before searching, clarify: -- What specifically are we trying to learn? -- What constraints matter (language, framework, scale)? -- What would a wrong answer cost? (helps calibrate rigor) -- Any sources they already know about? - -Restate the research question precisely. -``` - -Use `AskUserQuestion`. Don't proceed until the question is sharp. - -## Step 2: Discover Perspectives - -``` -Before generating search queries, survey the landscape. -Search for 2-3 overview/survey articles on the topic. -Fetch them with Jina Reader and extract: - -- What are the major schools of thought or approaches? -- Who are the key voices (companies, researchers, communities)? -- What are the known debates or controversies? -- What perspectives might be underrepresented? +Run the workflow via the devkit engine: -This grounds our search in real viewpoints, not LLM brainstorming. -Summarize immediately — do not carry raw content forward. ``` - -## Step 3: Decompose into Sub-Questions - -``` -Using the perspectives discovered in Step 2, break the research question -into 5-8 sub-questions, each with an explicit retrieval goal. - -Format: -- Query: - Goal: - Perspective: - -Rules: -- At least one query per major perspective/school of thought -- At least 2 queries explicitly seeking DISCONFIRMING evidence - (e.g., "problems with X", "X failures", "why X doesn't work", "X vs Y disadvantages") -- No two queries should return the same results -- Include at least one query targeting recent sources (last 12 months) +devkit workflow run deep-research "{input}" ``` -## Step 4: Search (Parallel Fan-Out) - -**[PARALLEL]** Launch sub-question searches concurrently using the `researcher` agent (max 3 agents per batch): - -``` -Task: Execute web search for a specific sub-question. -Agent: researcher -Input: Query + Goal + Perspective from decomposition step -Collect: titles, URLs, key snippets, publication date if available -``` +The YAML workflow (`workflows/deep-research.yml`) enforces the full ACH sequence deterministically. Claude handles thinking within each step; the engine owns the order. -Run in batches of 3. Collect all results before proceeding. - -## Budget - -- **Token budget:** ~400k tokens. Deep research is expensive but thorough. -- **Early exit:** Only if the question turns out to be trivial after Step 2. - -## Step 5: Summarize and Extract Claims - -``` -For the 5-8 most promising URLs, fetch clean content: - WebFetch https://r.jina.ai/{url} with header Accept: text/markdown - -For each source, extract ATOMIC CLAIMS — individual factual assertions: -- Claim: - Source: - Recency: - -CRITICAL: Summarize each page into atomic claims immediately. -Do NOT carry raw page content forward. -Aim for 3-8 claims per source. -``` - -## Step 6: Generate Competing Hypotheses - -``` -Based on the claims gathered, generate 2-4 COMPETING HYPOTHESES -that could answer the research question. - -Rules: -- Hypotheses must be mutually exclusive or meaningfully different -- Include at least one "contrarian" hypothesis that challenges the obvious answer -- Each hypothesis should be a clear, testable statement -- Don't include hypotheses with no supporting evidence at all - -Format: -- H1: -- H2: -- H3: -``` - -## Step 7: Directed Disconfirmation - -``` -For EACH hypothesis, search specifically for evidence that would DISPROVE it. +## Fallback (no engine) -This is the critical ACH step. You are not looking for confirmation. -You are trying to KILL each hypothesis. +If `devkit workflow` is not available, follow these steps manually: -For each hypothesis: -- Search: " wrong" or "problems with " or " better than " -- Fetch and summarize the most relevant disconfirming source -- Extract any new claims that contradict the hypothesis - -If you cannot find disconfirming evidence for a hypothesis after genuine effort, -note that — it's a signal of strength, not a gap to fill. -``` - -## Step 8: Build Evidence Matrix - -``` -Build a matrix: rows = evidence/claims, columns = hypotheses. - -For each cell, mark: -- CC (Strongly Consistent) — evidence directly supports this hypothesis -- C (Consistent) — evidence is compatible with this hypothesis -- N (Neutral) — evidence is irrelevant to this hypothesis -- I (Inconsistent) — evidence contradicts this hypothesis -- II (Strongly Inconsistent) — evidence directly disproves this hypothesis - -| Evidence | H1 | H2 | H3 | -|----------|----|----|-----| -| Claim 1 [source] | CC | I | N | -| Claim 2 [source] | N | CC | C | -| Claim 3 [source] | II | C | CC | -| ... | | | | - -Then score each hypothesis: -- Count inconsistencies (I + II). MORE inconsistencies = WEAKER hypothesis. -- The surviving hypothesis is the one with the FEWEST inconsistencies, - NOT the most consistencies. This is the key ACH insight. -``` - -## Step 9: Sensitivity Check - -``` -For the leading hypothesis, identify: -1. Which single piece of evidence, if wrong, would change the conclusion? -2. Are there any "linchpin" claims supported by only one source? -3. What new evidence would cause you to switch to a different hypothesis? - -This tells us how fragile or robust the conclusion is. -``` - -## Step 10: Self-Critique - -``` -Before writing the final synthesis, review your own work: - -1. Did I genuinely try to disprove each hypothesis, or did I softball the disconfirmation? -2. Are there perspectives I missed entirely? -3. Am I over-weighting recency or authority of any single source? -4. Would someone with the opposite view find my analysis fair? -5. Are there claims I'm treating as confirmed that are actually uncorroborated? - -If you find gaps, do ONE more targeted search round to fill them. -Otherwise, proceed to synthesis. -``` - -## Step 11: Synthesize - -``` -## Deep Research: {question} - -### Direct Answer -{clear answer with confidence level: HIGH / MEDIUM / LOW} - -### Competing Hypotheses Evaluated - -#### H1: {statement} — [REJECTED / SURVIVING / INCONCLUSIVE] -- Supporting evidence: {claims with sources} -- Disconfirming evidence: {claims with sources} -- Inconsistency count: X - -#### H2: {statement} — [REJECTED / SURVIVING / INCONCLUSIVE] -- Supporting evidence: {claims with sources} -- Disconfirming evidence: {claims with sources} -- Inconsistency count: X - -(repeat for each hypothesis) - -### Evidence Matrix Summary -{the matrix from Step 8, or a prose summary if >10 rows} - -### Key Findings -{top findings with corroboration status} -- CONFIRMED: {claim} — [source1], [source2] -- CONTESTED: {claim} — [source1] says X, [source2] says Y -- UNCORROBORATED: {claim} — [source] (single source) - -### Sensitivity Analysis -- Linchpin evidence: {what single fact, if wrong, changes the answer} -- Confidence fragility: {HIGH = robust across evidence, LOW = depends on 1-2 sources} - -### What Would Change This Conclusion -{specific evidence or events that would flip the answer} - -### Recommendation -{what to do and why, with explicit confidence calibration} -``` +1. **Clarify** — Use `AskUserQuestion` to sharpen the question; ask what a wrong answer would cost +2. **Discover perspectives** — Search for 2-3 overview articles; extract schools of thought, key voices, debates; summarize immediately +3. **Decompose** — 5-8 sub-questions with retrieval goals and perspective labels; at least 2 must seek disconfirming evidence +4. **Search** — Parallel fan-out using `researcher` agent (max 3 per batch); collect titles, URLs, snippets, dates +5. **Extract claims** — Fetch top 5-8 URLs via Jina Reader; extract atomic claims (3-8 per source); do NOT carry raw content forward +6. **Hypotheses** — Generate 2-4 competing hypotheses; include at least one contrarian; each must be testable +7. **Directed disconfirmation** — For EACH hypothesis, search for evidence that DISPROVES it; this is the critical ACH step +8. **Evidence matrix + sensitivity check** — Rows = claims, columns = hypotheses; mark CC/C/N/I/II; score by FEWEST inconsistencies (not most consistencies); identify linchpin evidence — what single fact, if wrong, changes the conclusion? +9. **Self-critique** — Did you genuinely try to disprove? Missed perspectives? Over-weighting a source? One more search round if gaps found (loop max 2) +10. **Synthesize** — Direct answer with confidence (HIGH/MEDIUM/LOW) → hypotheses evaluated → evidence matrix → key findings → sensitivity analysis → recommendation ## Rules -- Perspectives first — discover real viewpoints before generating queries -- Disconfirm, don't confirm — the goal is to DISPROVE hypotheses, not prove them -- Atomic claims — extract specific assertions, not vague summaries -- Summarize immediately — never carry raw fetched content into next steps +- Perspectives first — ground queries in real viewpoints, not LLM brainstorming +- Disconfirm > confirm — try to KILL hypotheses, not prove them +- Summarize immediately — never carry raw fetched content forward - Evidence matrix is mandatory — no skipping the structured comparison - Sensitivity check is mandatory — know how fragile your conclusion is - Self-critique before output — catch your own biases diff --git a/skills/research/SKILL.md b/skills/research/SKILL.md index 59f3c1e..b4434bc 100644 --- a/skills/research/SKILL.md +++ b/skills/research/SKILL.md @@ -3,142 +3,38 @@ name: research description: Research workflow — use when asked to research a topic, investigate options, compare approaches, or find the best solution to a technical question. NOT for "deep research" or "validate" requests — those go to deep-research. For complex or high-stakes questions where correctness is critical, use deep-research instead. --- -# Research Workflow +# Research -Complete research lifecycle: clarify → decompose → search → summarize → corroborate → synthesize. +Deterministic research workflow: clarify → decompose → parallel search → summarize → follow-up → synthesize. -## Step 1: Clarify +## Invoke -``` -The user wants to research: {input} - -Before searching, clarify: -- What specifically are we trying to learn? -- What constraints matter (language, framework, scale)? -- Any sources they already know about? - -Restate the research question precisely. -``` - -Use `AskUserQuestion` to ask these questions explicitly. Don't proceed until the question is clear. - -## Step 2: Decompose into Sub-Questions - -``` -Break the research question into 3-5 sub-questions, each with an explicit retrieval goal. - -Format: -- Query: - Goal: - -Rules: -- Each query must target a DIFFERENT angle (definition, evidence, criticism, alternatives, recency) -- No two queries should return the same results -- Include at least one query seeking disconfirming evidence or criticism -``` - -## Step 3: Search (Parallel Fan-Out) - -**[PARALLEL]** Launch all sub-question searches concurrently using the `researcher` agent (max 3 agents): +Run the workflow via the devkit engine: ``` -Task: Execute web search for a specific sub-question. -Agent: researcher -Input: Query + Goal from decomposition step -Collect: titles, URLs, key snippets +devkit workflow run research "{input}" ``` -All searches run in parallel. Collect results before proceeding. - -## Budget +The YAML workflow (`workflows/research.yml`) enforces the step sequence deterministically. Claude handles thinking within each step; the engine owns the order. -- **Token budget:** ~200k tokens. -- **Early exit:** If the first search pass clearly answers the question, skip follow-up. - -## Step 4: Summarize Sources - -``` -For the 3-5 most promising URLs, fetch clean content: - WebFetch https://r.jina.ai/{url} with header Accept: text/markdown +## Fallback (no engine) -CRITICAL: Immediately summarize each page into 3-5 key claims with source attribution. -Do NOT carry raw page content forward — summarize first, then discard the raw text. -This prevents context overflow on large pages. - -If Jina fails for a URL, fall back to raw WebFetch on the original URL. -``` - -## Step 5: Corroborate - -``` -For each key claim from Step 4: -- Count how many independent sources support it -- Flag any claims supported by only 1 source as "uncorroborated" -- Flag any claims where sources contradict each other +If `devkit workflow` is not available, follow these steps manually. Token budget: ~200k. Early exit if first search pass clearly answers the question. -Mark claims as: -- CONFIRMED (2+ independent sources agree) -- UNCORROBORATED (only 1 source) -- CONTESTED (sources disagree) -``` - -## Escalation Check - -After corroboration, evaluate whether this question needs deep research. **Ask the user** (via `AskUserQuestion`) to upgrade to `/devkit:deep-research` if ANY of these are true: - -1. **3+ CONTESTED claims** — sources actively contradict each other on key points -2. **High-stakes domain** — the question involves security, architecture decisions, compliance, data integrity, or financial impact -3. **User expressed uncertainty** — they said "I'm not sure if", "is this actually true", "I've heard conflicting things", or similar -4. **Most claims are UNCORROBORATED** — more than half the key claims have only one source - -Phrasing: "I'm finding [conflicting sources / low confidence / high-stakes implications] on this. Want me to switch to deep research with competing hypothesis analysis? It costs more tokens but produces higher-confidence results." - -If the user says no, continue with the standard synthesis. Do NOT auto-escalate — always ask first. - -## Step 6: Follow-Up - -``` -Review claims marked UNCORROBORATED or CONTESTED. -For each, run one targeted search to either confirm or resolve the conflict. -Use Jina Reader for any new sources, summarize immediately. - -If all key claims are confirmed or the question is answered, say "RESEARCH_COMPLETE". -``` - -Loop up to 2 times. - -## Step 7: Synthesize - -``` -## Research: {question} - -### Direct Answer -{clear answer to the research question} - -### Key Findings -{findings with source URLs and corroboration status} -- CONFIRMED: {claim} — [source1], [source2] -- UNCORROBORATED: {claim} — [source] (single source only) -- CONTESTED: {claim} — [source1] says X, [source2] says Y - -### Tradeoffs -{comparison between approaches} - -### Open Questions -{what couldn't be resolved, any CONTESTED claims without resolution} - -### Recommendation -{what to do and why, noting confidence level} -``` +1. **Clarify** — Use `AskUserQuestion` to sharpen the question before searching +2. **Decompose** — Break into 3-5 sub-questions with explicit retrieval goals; include at least one disconfirming query +3. **Search** — Launch searches in parallel using the `researcher` agent (max 3); collect titles, URLs, snippets +4. **Summarize** — Fetch top URLs via Jina Reader (`WebFetch https://r.jina.ai/{url}`); extract 3-5 claims per source immediately; do NOT carry raw content forward +5. **Corroborate** — Mark each claim CONFIRMED (2+ sources) / UNCORROBORATED (1 source) / CONTESTED (sources disagree) +6. **Escalation check** — Ask the user to upgrade to `/devkit:deep-research` if ANY: 3+ CONTESTED claims, high-stakes domain, user expressed uncertainty, or most claims UNCORROBORATED. Never auto-escalate. +7. **Follow-up** — For UNCORROBORATED/CONTESTED claims, run one targeted search to resolve (loop max 2) +8. **Synthesize** — Direct answer → key findings with corroboration status → tradeoffs → open questions → recommendation with confidence level ## Rules - Clarify before searching — don't waste searches on a vague question -- Decompose into sub-questions with explicit goals — no vague "broad search" +- Decompose with explicit goals — no vague "broad search" - Summarize immediately — never carry raw fetched content forward -- Track corroboration — every claim should note how many sources support it -- Surface contradictions — if sources disagree, say so explicitly -- Follow up on gaps — loop if key claims are uncorroborated +- Track corroboration — every claim notes how many sources support it - Cite sources — every finding links to where it came from -- Recommend — don't just dump information, give a clear recommendation with confidence level -- Escalate when warranted — if sources conflict, stakes are high, or confidence is low, ask the user to upgrade to deep-research. Never auto-escalate. +- Escalate when warranted — ask the user, never auto-escalate