From cbfd50eb35c1c91c126f5523575c555fe4bd5b26 Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Sun, 5 Apr 2026 11:12:44 -0400 Subject: [PATCH 1/3] Add ACH-enhanced deep research and upgrade research workflow MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two-tier research system: - research (upgraded): query decomposition with explicit goals, iterative summarization to prevent context overflow, source corroboration tracking (CONFIRMED/UNCORROBORATED/CONTESTED), follow-up on gaps - deep-research (new): Analysis of Competing Hypotheses pipeline — perspective discovery, directed disconfirmation, evidence matrix scoring by fewest inconsistencies, sensitivity analysis, self-critique pass Based on research into OpenAI Deep Research, Perplexity, Stanford STORM, gpt-researcher, and Anthropic's multi-agent research system. --- README.md | 12 +- ROADMAP.md | 8 +- commands/deep-research.md | 130 +++++++++++++++++++ skills/deep-research/SKILL.md | 236 ++++++++++++++++++++++++++++++++++ skills/research/SKILL.md | 110 +++++++++------- workflows/deep-research.yml | 234 +++++++++++++++++++++++++++++++++ workflows/research.yml | 93 ++++++++------ 7 files changed, 727 insertions(+), 96 deletions(-) create mode 100644 commands/deep-research.md create mode 100644 skills/deep-research/SKILL.md create mode 100644 workflows/deep-research.yml diff --git a/README.md b/README.md index c8a7607..73b80ac 100644 --- a/README.md +++ b/README.md @@ -119,6 +119,7 @@ Verify with `/context-mode:ctx-doctor` (plugin install) or check MCP tools are a | `/devkit:refactor` | Full refactor lifecycle — analyze, plan, restructure, verify, compare | | `/devkit:decompose` | Goal decomposition — break into task DAG, assign agents, execute in dependency order | | `/devkit:audit` | Full project health audit — deps, vulnerabilities, licenses, lint, security | +| `/devkit:deep-research` | ACH-enhanced deep research — competing hypotheses, directed disconfirmation, evidence matrix, sensitivity analysis | | `/devkit:status` | Health check — installed CLIs, available agents, ready commands | ### Self-Improvement Loops (Claude-only) @@ -174,7 +175,8 @@ These replace slash commands. Ask naturally and the skill fires: | `devkit:doc-gen` | "document this module", "generate API docs", "write docs for" | | `devkit:changelog` | "generate a changelog", "release notes", "what changed since" | | `devkit:onboard` | "explain this codebase", "help me understand the architecture", "onboard" | -| `devkit:research` | "research X", "deep dive on", "compare approaches for" | +| `devkit:research` | "research X", "compare approaches for", "investigate options" | +| `devkit:deep-research` | "deep research", "validate this", "make sure this is right", "rigorous analysis" | | `devkit:scrape` | "scrape this URL", "fetch content from", "extract from this page" | ### Coding Principles @@ -189,6 +191,7 @@ Loaded as reference material when relevant: | `devkit:yagni` | Build only what's needed, no speculative features or premature abstractions | | `devkit:dont-reinvent` | Use existing libraries, tools, and stdlib before building custom solutions | | `devkit:stuck` | Detect agent looping/failing, structured recovery — backtrack, simplify, escalate | +| `devkit:deep-research` | ACH-enhanced research — competing hypotheses, disconfirmation, evidence matrix | | `devkit:scratchpad` | Iteration memory protocol — prevents Groundhog Day loops by recording what was tried | ### Tools @@ -309,7 +312,7 @@ devkit/ │ └── plugin.json # Plugin metadata (name, version, author) ├── ROADMAP.md # Implemented features and future plans ├── PREFERENCES.md # Agent behavior guidelines -├── commands/ # 20 slash commands (tab-completable) +├── commands/ # 21 slash commands (tab-completable) │ ├── tri-*.md # Multi-agent dispatch (5) │ ├── self-*.md # Self-improvement loops (5) │ ├── pr-ready.md # PR preparation pipeline @@ -322,7 +325,7 @@ devkit/ │ ├── audit.md # Project health audit │ ├── repo-map.md # AST-based symbol index │ └── status.md # Health check -├── skills/ # 15 context-activated skills +├── skills/ # 16 context-activated skills │ ├── executing/SKILL.md # Principle: methodical execution │ ├── clean-code/SKILL.md # Principle: readability │ ├── dry/SKILL.md # Principle: don't repeat yourself @@ -336,6 +339,7 @@ devkit/ │ ├── test-gen/SKILL.md # Auto: "write tests for X" │ ├── onboard/SKILL.md # Auto: "explain this codebase" │ ├── research/SKILL.md # Auto: "research X" +│ ├── deep-research/SKILL.md # Auto: "deep research", ACH pipeline │ └── scrape/SKILL.md # Auto: "scrape this URL" ├── agents/ # 6 agents │ ├── reviewer.md # Opus, worktree isolation @@ -356,7 +360,7 @@ devkit/ │ ├── lang-review.sh # Language-aware code quality (Go/TS/Rust/Python/Shell) │ ├── subagent-stop.sh # Subagent work verification │ └── stop-gate.sh # Consolidated quality gate (cross-domain + vet/lint) -├── workflows/ # 12 YAML workflow definitions +├── workflows/ # 13 YAML workflow definitions ├── presets/ # Reserved for future use ├── .github/workflows/ # CI/CD │ ├── ci.yml # Build + test + vet on push/PR diff --git a/ROADMAP.md b/ROADMAP.md index c2d0036..8809976 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -2,8 +2,8 @@ ## Implemented -- **20 slash commands** — Lifecycle workflows, self-improvement loops, multi-agent dispatch, project health audit, post-PR monitoring, AST repo mapping -- **15 context-activated skills** — 6 auto-trigger workflows (test-gen, doc-gen, changelog, onboard, research, scrape) + 6 coding principles (executing, clean-code, DRY, YAGNI, dont-reinvent, stuck) + 2 tools (gcli, creating-workflows) + 1 iteration memory (scratchpad) +- **21 slash commands** — Lifecycle workflows, self-improvement loops, multi-agent dispatch, project health audit, post-PR monitoring, AST repo mapping +- **16 context-activated skills** — 7 auto-trigger workflows (test-gen, doc-gen, changelog, onboard, research, deep-research, scrape) + 6 coding principles (executing, clean-code, DRY, YAGNI, dont-reinvent, stuck) + 2 tools (gcli, creating-workflows) + 1 iteration memory (scratchpad) - **6 agents** — Scoped tool access, worktree isolation, model assignment - **10 hooks** — Safety (destructive command blocking, edit-time security patterns, PR gate), observability (audit trail, slop detection, post-validation, subagent verification, language-aware code review), optimization (RTK token compression) - **Graceful degradation** — tri:* commands work with 1-3 agents depending on installed CLIs @@ -12,7 +12,7 @@ - **Early-exit conditions** — Self-improvement loops stop when goal is met, not just at max iterations - **Token budget guidance** — Per-command budget recommendations with model downgrade patterns - **RTK token optimization** — Optional PreToolUse hook compresses Bash output via RTK (60-90% savings) -- **12 YAML workflows** — Portable workflow definitions (feature, bugfix, refactor, research, self-*, tri-*) +- **13 YAML workflows** — Portable workflow definitions (feature, bugfix, refactor, research, deep-research, self-*, tri-*) - **Separate marketplace** — Multi-plugin marketplace at `5uck1ess/marketplace` - **Companion ecosystem** — Evaluated official marketplace, documented holistic setup with 7 complementary plugins - **Hypothesis-driven perf** — Evidence gathering, ranked hypotheses, one-at-a-time testing replaces blind benchmark loops @@ -34,6 +34,6 @@ Items below were on the roadmap but determined to be unnecessary — either alre | Stop hook redesign | Still fires every turn, but exits early with `approve` when no files are changed — near-instant on clean trees, so the performance concern is moot. Revisit only if it causes measurable latency. | | Cost event hooks | Budget enforcement already exists in the Go engine via `overBudget()` + `addCost()` callbacks with hard limits | | Execution registry | Step tracking already handled by SQLite via `lib.DB` with status, cost, and timing per step | -| Preset library | The 12 YAML workflows and 15 skills already serve this purpose | +| Preset library | The 13 YAML workflows and 16 skills already serve this purpose | | Framework-specific review checklists | `lang-review.sh` covers language-level patterns; framework-specific rules are better added per-project via hookify | | Conditional hook firing | Hooks already self-filter internally (extension checks, changed-file checks); a generic condition system adds complexity for no current need | diff --git a/commands/deep-research.md b/commands/deep-research.md new file mode 100644 index 0000000..0ca813f --- /dev/null +++ b/commands/deep-research.md @@ -0,0 +1,130 @@ +--- +description: ACH-enhanced deep research — perspective discovery, competing hypotheses, directed disconfirmation, evidence matrix, sensitivity analysis. +--- + +# Deep Research + +Rigorous research for complex or high-stakes questions where correctness matters. Uses Analysis of Competing Hypotheses (ACH) to actively disprove answers rather than just confirming them. + +Use regular `/devkit:research` for quick lookups. Use this when: +- The answer has real consequences (architecture decisions, tool selection, security) +- Multiple conflicting sources exist +- You need confidence calibration, not just an answer +- The user says "deep research", "validate", "make sure this is right" + +## Step 0: Harness Detection + +```bash +if command -v devkit >/dev/null 2>&1; then + echo "Go harness detected — delegating to devkit workflow deep-research." + devkit workflow deep-research "{input}" + exit 0 +fi +``` + +## Step 1: Clarify + +``` +The user wants to deep-research: {input} + +Use AskUserQuestion to clarify: +- What specifically are we trying to learn? +- What constraints matter? +- What would a wrong answer cost? +- Any sources they already know about? + +Restate the research question precisely. +``` + +## Step 2: Discover Perspectives + +Search for 2-3 overview articles on the topic. Fetch with Jina Reader (`WebFetch https://r.jina.ai/{url}`). Extract major schools of thought, key voices, known debates. Summarize immediately — do not carry raw content forward. + +## Step 3: Decompose into Sub-Questions + +Break the question into 5-8 sub-questions with explicit retrieval goals and perspective labels. At least 2 queries must seek **disconfirming** evidence. + +## Step 4: Search (Parallel Fan-Out) + +**[PARALLEL]** Launch searches concurrently using the `researcher` agent (max 3 per batch): + +``` +Task: Execute web search for sub-question. +Agent: researcher +Input: Query + Goal + Perspective +``` + +## Budget + +- **Token budget:** ~400k tokens. +- **Early exit:** Only if the question turns out trivial after perspective discovery. + +## Step 5: Summarize and Extract Claims + +Fetch top 5-8 URLs with Jina Reader. Extract atomic claims (subject-predicate-object). Summarize immediately. Aim for 3-8 claims per source. + +## Step 6: Generate Competing Hypotheses + +Generate 2-4 mutually exclusive hypotheses. Include at least one contrarian hypothesis. + +## Step 7: Directed Disconfirmation + +For EACH hypothesis, search specifically for evidence that would DISPROVE it. This is the critical ACH step — you're trying to kill each hypothesis, not confirm it. + +## Step 8: Build Evidence Matrix + +``` +| Evidence | H1 | H2 | H3 | +|----------|----|----|-----| +| Claim [source] | CC | I | N | + +CC=Strongly Consistent, C=Consistent, N=Neutral, I=Inconsistent, II=Strongly Inconsistent +``` + +Score by FEWEST inconsistencies (not most consistencies). + +## Step 9: Sensitivity Check + +Identify linchpin evidence — what single fact, if wrong, would change the conclusion? + +## Step 10: Self-Critique + +Review for: genuine disconfirmation effort, missed perspectives, source over-weighting, fairness to opposing views. One more search round if gaps found. + +## Step 11: Synthesize + +``` +## Deep Research: {question} + +### Direct Answer +{answer with confidence: HIGH / MEDIUM / LOW} + +### Competing Hypotheses +{for each: statement, supporting evidence, disconfirming evidence, status} + +### Evidence Matrix Summary +{matrix or prose summary} + +### Key Findings +{CONFIRMED / CONTESTED / UNCORROBORATED claims with sources} + +### Sensitivity Analysis +{linchpin evidence, confidence fragility} + +### What Would Change This Conclusion +{specific evidence that would flip the answer} + +### Recommendation +{what to do, why, with calibrated confidence} +``` + +## Rules + +- Perspectives first — ground queries in real viewpoints +- Disconfirm > confirm — try to KILL hypotheses +- Summarize immediately — no raw content carried forward +- Evidence matrix is mandatory +- Sensitivity check is mandatory +- Self-critique before output +- Cite everything +- Be honest about uncertainty diff --git a/skills/deep-research/SKILL.md b/skills/deep-research/SKILL.md new file mode 100644 index 0000000..c355430 --- /dev/null +++ b/skills/deep-research/SKILL.md @@ -0,0 +1,236 @@ +--- +name: deep-research +description: Deep research with Analysis of Competing Hypotheses — use when asked to do deep research, deeply investigate, validate claims, or when correctness is critical and the user wants rigorous analysis with disconfirmation testing. +--- + +# Deep Research Workflow + +ACH-enhanced research: clarify → discover perspectives → decompose → search → summarize → generate hypotheses → disconfirm → build evidence matrix → self-critique → synthesize. + +This is the rigorous path. It costs more tokens but produces higher-confidence results by actively trying to disprove answers rather than just confirming them. + +## Step 1: Clarify + +``` +The user wants to deep-research: {input} + +Before searching, clarify: +- What specifically are we trying to learn? +- What constraints matter (language, framework, scale)? +- What would a wrong answer cost? (helps calibrate rigor) +- Any sources they already know about? + +Restate the research question precisely. +``` + +Use `AskUserQuestion`. Don't proceed until the question is sharp. + +## Step 2: Discover Perspectives + +``` +Before generating search queries, survey the landscape. +Search for 2-3 overview/survey articles on the topic. +Fetch them with Jina Reader and extract: + +- What are the major schools of thought or approaches? +- Who are the key voices (companies, researchers, communities)? +- What are the known debates or controversies? +- What perspectives might be underrepresented? + +This grounds our search in real viewpoints, not LLM brainstorming. +Summarize immediately — do not carry raw content forward. +``` + +## Step 3: Decompose into Sub-Questions + +``` +Using the perspectives discovered in Step 2, break the research question +into 5-8 sub-questions, each with an explicit retrieval goal. + +Format: +- Query: + Goal: + Perspective: + +Rules: +- At least one query per major perspective/school of thought +- At least 2 queries explicitly seeking DISCONFIRMING evidence + (e.g., "problems with X", "X failures", "why X doesn't work", "X vs Y disadvantages") +- No two queries should return the same results +- Include at least one query targeting recent sources (last 12 months) +``` + +## Step 4: Search (Parallel Fan-Out) + +**[PARALLEL]** Launch sub-question searches concurrently using the `researcher` agent (max 3 agents per batch): + +``` +Task: Execute web search for a specific sub-question. +Agent: researcher +Input: Query + Goal + Perspective from decomposition step +Collect: titles, URLs, key snippets, publication date if available +``` + +Run in batches of 3. Collect all results before proceeding. + +## Budget + +- **Token budget:** ~400k tokens. Deep research is expensive but thorough. +- **Early exit:** Only if the question turns out to be trivial after Step 2. + +## Step 5: Summarize and Extract Claims + +``` +For the 5-8 most promising URLs, fetch clean content: + WebFetch https://r.jina.ai/{url} with header Accept: text/markdown + +For each source, extract ATOMIC CLAIMS — individual factual assertions: +- Claim: + Source: + Recency: + +CRITICAL: Summarize each page into atomic claims immediately. +Do NOT carry raw page content forward. +Aim for 3-8 claims per source. +``` + +## Step 6: Generate Competing Hypotheses + +``` +Based on the claims gathered, generate 2-4 COMPETING HYPOTHESES +that could answer the research question. + +Rules: +- Hypotheses must be mutually exclusive or meaningfully different +- Include at least one "contrarian" hypothesis that challenges the obvious answer +- Each hypothesis should be a clear, testable statement +- Don't include hypotheses with no supporting evidence at all + +Format: +- H1: +- H2: +- H3: +``` + +## Step 7: Directed Disconfirmation + +``` +For EACH hypothesis, search specifically for evidence that would DISPROVE it. + +This is the critical ACH step. You are not looking for confirmation. +You are trying to KILL each hypothesis. + +For each hypothesis: +- Search: " wrong" or "problems with " or " better than " +- Fetch and summarize the most relevant disconfirming source +- Extract any new claims that contradict the hypothesis + +If you cannot find disconfirming evidence for a hypothesis after genuine effort, +note that — it's a signal of strength, not a gap to fill. +``` + +## Step 8: Build Evidence Matrix + +``` +Build a matrix: rows = evidence/claims, columns = hypotheses. + +For each cell, mark: +- CC (Strongly Consistent) — evidence directly supports this hypothesis +- C (Consistent) — evidence is compatible with this hypothesis +- N (Neutral) — evidence is irrelevant to this hypothesis +- I (Inconsistent) — evidence contradicts this hypothesis +- II (Strongly Inconsistent) — evidence directly disproves this hypothesis + +| Evidence | H1 | H2 | H3 | +|----------|----|----|-----| +| Claim 1 [source] | CC | I | N | +| Claim 2 [source] | N | CC | C | +| Claim 3 [source] | II | C | CC | +| ... | | | | + +Then score each hypothesis: +- Count inconsistencies (I + II). MORE inconsistencies = WEAKER hypothesis. +- The surviving hypothesis is the one with the FEWEST inconsistencies, + NOT the most consistencies. This is the key ACH insight. +``` + +## Step 9: Sensitivity Check + +``` +For the leading hypothesis, identify: +1. Which single piece of evidence, if wrong, would change the conclusion? +2. Are there any "linchpin" claims supported by only one source? +3. What new evidence would cause you to switch to a different hypothesis? + +This tells us how fragile or robust the conclusion is. +``` + +## Step 10: Self-Critique + +``` +Before writing the final synthesis, review your own work: + +1. Did I genuinely try to disprove each hypothesis, or did I softball the disconfirmation? +2. Are there perspectives I missed entirely? +3. Am I over-weighting recency or authority of any single source? +4. Would someone with the opposite view find my analysis fair? +5. Are there claims I'm treating as confirmed that are actually uncorroborated? + +If you find gaps, do ONE more targeted search round to fill them. +Otherwise, proceed to synthesis. +``` + +## Step 11: Synthesize + +``` +## Deep Research: {question} + +### Direct Answer +{clear answer with confidence level: HIGH / MEDIUM / LOW} + +### Competing Hypotheses Evaluated + +#### H1: {statement} — [REJECTED / SURVIVING / INCONCLUSIVE] +- Supporting evidence: {claims with sources} +- Disconfirming evidence: {claims with sources} +- Inconsistency count: X + +#### H2: {statement} — [REJECTED / SURVIVING / INCONCLUSIVE] +- Supporting evidence: {claims with sources} +- Disconfirming evidence: {claims with sources} +- Inconsistency count: X + +(repeat for each hypothesis) + +### Evidence Matrix Summary +{the matrix from Step 8, or a prose summary if >10 rows} + +### Key Findings +{top findings with corroboration status} +- CONFIRMED: {claim} — [source1], [source2] +- CONTESTED: {claim} — [source1] says X, [source2] says Y +- UNCORROBORATED: {claim} — [source] (single source) + +### Sensitivity Analysis +- Linchpin evidence: {what single fact, if wrong, changes the answer} +- Confidence fragility: {HIGH = robust across evidence, LOW = depends on 1-2 sources} + +### What Would Change This Conclusion +{specific evidence or events that would flip the answer} + +### Recommendation +{what to do and why, with explicit confidence calibration} +``` + +## Rules + +- Perspectives first — discover real viewpoints before generating queries +- Disconfirm, don't confirm — the goal is to DISPROVE hypotheses, not prove them +- Atomic claims — extract specific assertions, not vague summaries +- Summarize immediately — never carry raw fetched content into next steps +- Evidence matrix is mandatory — no skipping the structured comparison +- Sensitivity check is mandatory — know how fragile your conclusion is +- Self-critique before output — catch your own biases +- Cite everything — every claim links to its source +- Confidence calibration — HIGH/MEDIUM/LOW based on evidence robustness, not gut feel +- Be honest about uncertainty — "I don't know" with good reasoning beats a confident wrong answer diff --git a/skills/research/SKILL.md b/skills/research/SKILL.md index 091ce30..c6fe0b1 100644 --- a/skills/research/SKILL.md +++ b/skills/research/SKILL.md @@ -1,11 +1,11 @@ --- name: research -description: Deep research workflow — use when asked to research a topic, do a deep dive, investigate options, compare approaches, or find the best solution to a technical question. +description: Research workflow — use when asked to research a topic, do a deep dive, investigate options, compare approaches, or find the best solution to a technical question. For complex or high-stakes questions where correctness is critical, use deep-research instead. --- # Research Workflow -Complete research lifecycle: clarify → search (broad + specific + alternatives) → analyze → follow-up → synthesize. +Complete research lifecycle: clarify → decompose → search → summarize → corroborate → synthesize. ## Step 1: Clarify @@ -22,74 +22,79 @@ Restate the research question precisely. Use `AskUserQuestion` to ask these questions explicitly. Don't proceed until the question is clear. -## Step 2: Search (Three Perspectives) +## Step 2: Decompose into Sub-Questions -Run three search strategies using the `researcher` agent. - -**[PARALLEL]** Launch all three searches concurrently (max 3 agents): - -### Broad Search - -``` -Task: Web search with broad queries to find general sources. -Agent: researcher -Input: Research question + broad search terms -Collect: titles, URLs, key snippets ``` +Break the research question into 3-5 sub-questions, each with an explicit retrieval goal. -### Specific Search +Format: +- Query: + Goal: -``` -Task: Web search with specific technical queries targeting documentation, benchmarks, or comparisons. -Agent: researcher -Input: Research question + specific/technical search terms (different from broad) +Rules: +- Each query must target a DIFFERENT angle (definition, evidence, criticism, alternatives, recency) +- No two queries should return the same results +- Include at least one query seeking disconfirming evidence or criticism ``` -### Alternative Viewpoints +## Step 3: Search (Parallel Fan-Out) + +**[PARALLEL]** Launch all sub-question searches concurrently using the `researcher` agent (max 3 agents): ``` -Task: Web search for contrarian or alternative viewpoints. +Task: Execute web search for a specific sub-question. Agent: researcher -Input: "problems with", "alternatives to", "vs" queries related to the topic +Input: Query + Goal from decomposition step +Collect: titles, URLs, key snippets ``` -All three run in parallel. Collect results before proceeding to analysis. +All searches run in parallel. Collect results before proceeding. ## Budget -- **Token budget:** ~300k tokens. Web search and reading can be expensive. -- **Early exit:** If the first search pass answers the question clearly, skip follow-up loops. +- **Token budget:** ~200k tokens. +- **Early exit:** If the first search pass clearly answers the question, skip follow-up. -## Step 3: Analyze +## Step 4: Summarize Sources ``` -Read the 3-5 most promising URLs in depth. - -For each URL, fetch clean Markdown using Jina Reader: +For the 3-5 most promising URLs, fetch clean content: WebFetch https://r.jina.ai/{url} with header Accept: text/markdown -This strips boilerplate (nav, ads, footers) and returns clean article content. +CRITICAL: Immediately summarize each page into 3-5 key claims with source attribution. +Do NOT carry raw page content forward — summarize first, then discard the raw text. +This prevents context overflow on large pages. + If Jina fails for a URL, fall back to raw WebFetch on the original URL. +``` -Extract key findings, compare approaches, note contradictions. +## Step 5: Corroborate -Sources from: -- Broad search results -- Specific search results -- Alternative viewpoints +``` +For each key claim from Step 4: +- Count how many independent sources support it +- Flag any claims supported by only 1 source as "uncorroborated" +- Flag any claims where sources contradict each other + +Mark claims as: +- CONFIRMED (2+ independent sources agree) +- UNCORROBORATED (only 1 source) +- CONTESTED (sources disagree) ``` -## Step 4: Follow-Up +## Step 6: Follow-Up ``` -Are there gaps in the research? If the initial sources were weak -or missing key perspectives, search and read more to fill gaps. -Use Jina Reader (WebFetch https://r.jina.ai/{url}) for any new sources. +Review claims marked UNCORROBORATED or CONTESTED. +For each, run one targeted search to either confirm or resolve the conflict. +Use Jina Reader for any new sources, summarize immediately. + +If all key claims are confirmed or the question is answered, say "RESEARCH_COMPLETE". ``` -Loop up to 3 times until research is thorough enough. +Loop up to 2 times. -## Step 5: Synthesize +## Step 7: Synthesize ``` ## Research: {question} @@ -98,24 +103,29 @@ Loop up to 3 times until research is thorough enough. {clear answer to the research question} ### Key Findings -{findings with source URLs} +{findings with source URLs and corroboration status} +- CONFIRMED: {claim} — [source1], [source2] +- UNCORROBORATED: {claim} — [source] (single source only) +- CONTESTED: {claim} — [source1] says X, [source2] says Y ### Tradeoffs {comparison between approaches} ### Open Questions -{what couldn't be resolved} +{what couldn't be resolved, any CONTESTED claims without resolution} ### Recommendation -{what to do and why} +{what to do and why, noting confidence level} ``` ## Rules - Clarify before searching — don't waste searches on a vague question -- Multiple search strategies — broad, specific, and contrarian -- Read actual sources — don't summarize from snippets alone -- Note contradictions — if sources disagree, say so -- Follow up on gaps — loop if the initial pass missed something -- Cite sources — every finding should link to where it came from -- Recommend — don't just dump information, give a clear recommendation +- Decompose into sub-questions with explicit goals — no vague "broad search" +- Summarize immediately — never carry raw fetched content forward +- Track corroboration — every claim should note how many sources support it +- Surface contradictions — if sources disagree, say so explicitly +- Follow up on gaps — loop if key claims are uncorroborated +- Cite sources — every finding links to where it came from +- Recommend — don't just dump information, give a clear recommendation with confidence level +- For complex/high-stakes questions, suggest the user run `/devkit:deep-research` instead diff --git a/workflows/deep-research.yml b/workflows/deep-research.yml new file mode 100644 index 0000000..09d7224 --- /dev/null +++ b/workflows/deep-research.yml @@ -0,0 +1,234 @@ +name: Deep Research +description: ACH-enhanced deep research — perspective discovery, competing hypotheses, directed disconfirmation, evidence matrix + +budget: + limit: 400000 + downgrade: fast + +steps: + - id: clarify + model: smart + prompt: | + The user wants to deep-research: {{input}} + + Use ask_user to clarify: + - What specifically are we trying to learn? + - What constraints matter (language, framework, scale)? + - What would a wrong answer cost? + - Any sources they already know about? + + Restate the research question precisely. + + - id: perspectives + model: smart + prompt: | + Research question: {{clarify}} + + Before generating search queries, survey the landscape. + Search for 2-3 overview/survey articles on this topic. + Fetch with Jina Reader: WebFetch https://r.jina.ai/{url} + + Extract and summarize immediately: + - Major schools of thought or approaches + - Key voices (companies, researchers, communities) + - Known debates or controversies + - Underrepresented perspectives + + Do NOT carry raw content forward — summarize only. + + - id: decompose + model: smart + prompt: | + Research question: {{clarify}} + Perspectives discovered: {{perspectives}} + + Break the question into 5-8 sub-questions with explicit retrieval goals. + + Format: + - Query: + Goal: + Perspective: + + Rules: + - At least one query per major perspective + - At least 2 queries seeking DISCONFIRMING evidence + (e.g., "problems with X", "X failures", "why X doesn't work") + - Include at least one query targeting recent sources (last 12 months) + - No two queries should return the same results + + - id: search-1 + model: general + prompt: | + Execute web search for sub-questions 1-2 from this decomposition: + {{decompose}} + + Collect titles, URLs, key snippets, publication dates. + + - id: search-2 + model: general + prompt: | + Execute web search for sub-questions 3-4 from this decomposition: + {{decompose}} + + Collect titles, URLs, key snippets, publication dates. + + - id: search-3 + model: general + prompt: | + Execute web search for sub-questions 5+ from this decomposition: + {{decompose}} + + Prioritize the disconfirming-evidence queries. + Collect titles, URLs, key snippets, publication dates. + + - id: dispatch-searches + parallel: [search-1, search-2, search-3] + + - id: extract-claims + model: smart + prompt: | + Research question: {{clarify}} + + Search results: + {{search-1}} + {{search-2}} + {{search-3}} + + For the 5-8 most promising URLs, fetch clean content: + WebFetch https://r.jina.ai/{url} with header Accept: text/markdown + + Extract ATOMIC CLAIMS — individual factual assertions: + - Claim: + Source: + Recency: + + CRITICAL: Summarize each page into atomic claims immediately. + Do NOT carry raw page content forward. + Aim for 3-8 claims per source. + + - id: hypotheses + model: smart + prompt: | + Research question: {{clarify}} + Extracted claims: {{extract-claims}} + + Generate 2-4 COMPETING HYPOTHESES that answer the research question. + + Rules: + - Hypotheses must be mutually exclusive or meaningfully different + - Include at least one "contrarian" hypothesis challenging the obvious answer + - Each must be a clear, testable statement + - Don't include hypotheses with zero supporting evidence + + Format: + - H1: + - H2: + - H3: + + - id: disconfirm + model: smart + prompt: | + Research question: {{clarify}} + Hypotheses: {{hypotheses}} + Existing claims: {{extract-claims}} + + DIRECTED DISCONFIRMATION — the critical ACH step. + + For EACH hypothesis, search specifically for evidence that would DISPROVE it. + You are NOT looking for confirmation. You are trying to KILL each hypothesis. + + For each hypothesis: + 1. Search: " wrong", "problems with ", " better" + 2. Fetch and summarize the most relevant disconfirming source via Jina Reader + 3. Extract new claims that contradict the hypothesis + + If you cannot find disconfirming evidence after genuine effort, note that — + it's a signal of strength, not a gap. + + - id: evidence-matrix + model: smart + prompt: | + Research question: {{clarify}} + Hypotheses: {{hypotheses}} + Supporting claims: {{extract-claims}} + Disconfirming claims: {{disconfirm}} + + Build an EVIDENCE MATRIX. + Rows = all claims (supporting + disconfirming). Columns = hypotheses. + + For each cell, mark: + - CC (Strongly Consistent) — directly supports + - C (Consistent) — compatible + - N (Neutral) — irrelevant + - I (Inconsistent) — contradicts + - II (Strongly Inconsistent) — directly disproves + + | Evidence | H1 | H2 | H3 | + |----------|----|----|-----| + + Score each hypothesis by FEWEST inconsistencies (I + II). + The surviving hypothesis has the fewest inconsistencies, NOT the most consistencies. + + Then perform SENSITIVITY CHECK: + 1. Which single claim, if wrong, would change the conclusion? + 2. Are there linchpin claims from only one source? + 3. What new evidence would flip the answer? + + - id: self-critique + model: smart + prompt: | + Research question: {{clarify}} + Evidence matrix and sensitivity: {{evidence-matrix}} + + Self-critique your analysis: + 1. Did you genuinely try to disprove each hypothesis, or softball it? + 2. Are there perspectives you missed entirely? + 3. Are you over-weighting any single source? + 4. Would someone with the opposite view find this analysis fair? + 5. Are there claims treated as confirmed that are actually uncorroborated? + + If you find gaps, do ONE targeted search to fill them and report findings. + If the analysis is solid, say "CRITIQUE_COMPLETE" and summarize any minor adjustments. + loop: + max: 2 + until: CRITIQUE_COMPLETE + + - id: synthesize + model: smart + prompt: | + Research question: {{clarify}} + Evidence matrix: {{evidence-matrix}} + Self-critique: {{self-critique}} + + Write the final deep research synthesis: + + ## Deep Research: {question} + + ### Direct Answer + {answer with confidence: HIGH / MEDIUM / LOW} + + ### Competing Hypotheses Evaluated + For each hypothesis: + - Statement + - Status: REJECTED / SURVIVING / INCONCLUSIVE + - Supporting evidence with sources + - Disconfirming evidence with sources + - Inconsistency count + + ### Evidence Matrix Summary + {the matrix or prose summary} + + ### Key Findings + - CONFIRMED: {claim} — [source1], [source2] + - CONTESTED: {claim} — [source1] says X, [source2] says Y + - UNCORROBORATED: {claim} — [source] + + ### Sensitivity Analysis + - Linchpin evidence: {what changes the answer if wrong} + - Confidence fragility: HIGH (robust) / LOW (depends on 1-2 sources) + + ### What Would Change This Conclusion + {specific evidence or events that would flip the answer} + + ### Recommendation + {what to do and why, with calibrated confidence} diff --git a/workflows/research.yml b/workflows/research.yml index ba88c1e..b21011f 100644 --- a/workflows/research.yml +++ b/workflows/research.yml @@ -1,8 +1,8 @@ name: Research -description: Deep research — clarify with user, parallel search, analyze, synthesize +description: Research workflow — clarify, decompose, parallel search, summarize, corroborate, synthesize budget: - limit: 300000 + limit: 200000 downgrade: fast steps: @@ -18,69 +18,86 @@ steps: After getting their answer, restate the research question precisely. - - id: search-broad + - id: decompose model: general prompt: | Research question: {{clarify}} - Use web_search with a broad query to find general sources. + Break this into 3-5 sub-questions, each with an explicit retrieval goal. + + Format each as: + - Query: + Goal: + + Rules: + - Each query targets a DIFFERENT angle (definition, evidence, criticism, alternatives, recency) + - No two queries should return the same results + - Include at least one query seeking disconfirming evidence or criticism + + - id: search-1 + model: general + prompt: | + Execute web search for the FIRST sub-question from this decomposition: + {{decompose}} + Collect titles, URLs, and key snippets. - - id: search-specific + - id: search-2 model: general prompt: | - Research question: {{clarify}} + Execute web search for the SECOND sub-question from this decomposition: + {{decompose}} - Use web_search with specific technical queries targeting - documentation, benchmarks, or comparisons related to the question. - Use different search terms than a broad search would. + Collect titles, URLs, and key snippets. - - id: search-alternatives + - id: search-3 model: fast prompt: | - Research question: {{clarify}} + Execute web search for the THIRD sub-question from this decomposition: + {{decompose}} - Use web_search to find contrarian or alternative viewpoints. - Search for "problems with", "alternatives to", or "vs" queries. + Collect titles, URLs, and key snippets. - id: dispatch-searches - parallel: [search-broad, search-specific, search-alternatives] + parallel: [search-1, search-2, search-3] - - id: analyze + - id: summarize model: smart prompt: | Research question: {{clarify}} - Sources from broad search: - {{search-broad}} + Search results: + {{search-1}} + {{search-2}} + {{search-3}} - Sources from specific search: - {{search-specific}} + For the 3-5 most promising URLs, fetch clean content: + WebFetch https://r.jina.ai/{url} with header Accept: text/markdown - Alternative viewpoints: - {{search-alternatives}} + CRITICAL: Immediately summarize each page into 3-5 key CLAIMS with source attribution. + Do NOT carry raw page content forward. - Read the 3-5 most promising URLs in depth using Jina Reader for clean Markdown: - WebFetch https://r.jina.ai/{url} with header Accept: text/markdown - This strips boilerplate and returns clean article content. - If Jina fails for a URL, fall back to raw WebFetch on the original URL. - Extract key findings, compare approaches, note contradictions. + For each claim, note if it appears in multiple sources. + Mark claims as: + - CONFIRMED (2+ independent sources) + - UNCORROBORATED (1 source only) + - CONTESTED (sources disagree) - id: follow-up - model: smart + model: general prompt: | Research question: {{clarify}} - Initial analysis: - {{analyze}} + Current claims and corroboration status: + {{summarize}} - Are there gaps in the research? If the initial sources were weak - or missing key perspectives, use web_search and Jina Reader - (WebFetch https://r.jina.ai/{url}) to fill gaps. + Review claims marked UNCORROBORATED or CONTESTED. + For each, run one targeted search to confirm or resolve. + Summarize new sources immediately. - If the analysis is thorough enough, say "RESEARCH_COMPLETE". + If all key claims are resolved, say "RESEARCH_COMPLETE". loop: - max: 3 + max: 2 until: RESEARCH_COMPLETE - id: synthesize @@ -88,12 +105,12 @@ steps: prompt: | Research question: {{clarify}} - Full analysis: + Full analysis with corroboration: {{follow-up}} Write a clear synthesis: 1. **Direct answer** to the research question - 2. **Key findings** with source URLs + 2. **Key findings** with source URLs and corroboration status (CONFIRMED/UNCORROBORATED/CONTESTED) 3. **Tradeoffs** between approaches - 4. **Open questions** that couldn't be resolved - 5. **Recommendation** with reasoning + 4. **Open questions** — any CONTESTED claims without resolution + 5. **Recommendation** with confidence level (HIGH/MEDIUM/LOW) From 52fcd6f8744bc224a7e10a0f6a75918f25167e47 Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Sun, 5 Apr 2026 11:19:42 -0400 Subject: [PATCH 2/3] Disambiguate research vs deep-research skill triggers Add explicit NOT clause to research skill description so "deep research" and "validate" requests route to deep-research, not research. Caught by Claude reviewer in tri-review of PR #18. --- skills/research/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/research/SKILL.md b/skills/research/SKILL.md index c6fe0b1..81b94ea 100644 --- a/skills/research/SKILL.md +++ b/skills/research/SKILL.md @@ -1,6 +1,6 @@ --- name: research -description: Research workflow — use when asked to research a topic, do a deep dive, investigate options, compare approaches, or find the best solution to a technical question. For complex or high-stakes questions where correctness is critical, use deep-research instead. +description: Research workflow — use when asked to research a topic, investigate options, compare approaches, or find the best solution to a technical question. NOT for "deep research" or "validate" requests — those go to deep-research. For complex or high-stakes questions where correctness is critical, use deep-research instead. --- # Research Workflow From 45be34fe14df854dbf3acc9666eab82f982be1a1 Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Sun, 5 Apr 2026 11:25:13 -0400 Subject: [PATCH 3/3] Fix review findings: remove misplaced skill entry, add scratchpad to tree, fix hook count MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Remove deep-research from Coding Principles table (it's an auto-trigger workflow, not a coding principle — already in the correct table above) - Add scratchpad/SKILL.md to repo structure tree (was missing) - Fix ROADMAP hook consolidation count: 9 → 10 (stale from before PR #17) Caught by PR review toolkit (code-reviewer + comment-analyzer). --- README.md | 4 ++-- ROADMAP.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 73b80ac..4aa5e78 100644 --- a/README.md +++ b/README.md @@ -191,7 +191,6 @@ Loaded as reference material when relevant: | `devkit:yagni` | Build only what's needed, no speculative features or premature abstractions | | `devkit:dont-reinvent` | Use existing libraries, tools, and stdlib before building custom solutions | | `devkit:stuck` | Detect agent looping/failing, structured recovery — backtrack, simplify, escalate | -| `devkit:deep-research` | ACH-enhanced research — competing hypotheses, disconfirmation, evidence matrix | | `devkit:scratchpad` | Iteration memory protocol — prevents Groundhog Day loops by recording what was tried | ### Tools @@ -338,8 +337,9 @@ devkit/ │ ├── doc-gen/SKILL.md # Auto: "document this module" │ ├── test-gen/SKILL.md # Auto: "write tests for X" │ ├── onboard/SKILL.md # Auto: "explain this codebase" -│ ├── research/SKILL.md # Auto: "research X" +│ ├��─ research/SKILL.md # Auto: "research X" │ ├── deep-research/SKILL.md # Auto: "deep research", ACH pipeline +│ ├── scratchpad/SKILL.md # Iteration memory protocol │ └── scrape/SKILL.md # Auto: "scrape this URL" ├── agents/ # 6 agents │ ├── reviewer.md # Opus, worktree isolation diff --git a/ROADMAP.md b/ROADMAP.md index 8809976..8102b02 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -23,7 +23,7 @@ - **Iteration scratchpads** — Persistent memory across loop iterations to prevent repeated failures - **Cross-domain dirty-bit enforcement** — Blocks completion without test evidence per domain - **Language-universal hooks** — Consolidated language-specific hooks into `lang-review.sh` with Go, TypeScript, Rust, Python, and Shell support -- **Hook consolidation** — Merged 14 hooks into 9, reduced per-edit shell processes from 7 to 4 +- **Hook consolidation** — Merged 14 hooks into 10, reduced per-edit shell processes from 7 to 4 ## Retired