From d5a1a744bc266579b781637ebdade3a22a655280 Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Tue, 7 Apr 2026 10:39:34 -0400 Subject: [PATCH 1/3] Supplement existing commands with autoresearch-inspired patterns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add guard commands, investigation techniques, scenario expansion, and adversarial debate to four existing commands — no new files or structural changes. - autoloop: guard safety-net + escalation after repeated failures - tri-debug: investigation techniques + domain-specific checklists - tri-test-gen: scenario expansion techniques in generation prompt - deep-research: adversarial debate cycle for close hypotheses --- commands/autoloop.md | 25 +++++++++++++++++ commands/deep-research.md | 20 ++++++++++++++ commands/tri-debug.md | 58 +++++++++++++++++++++++++++++++++++++++ commands/tri-test-gen.md | 13 +++++++++ 4 files changed, 116 insertions(+) diff --git a/commands/autoloop.md b/commands/autoloop.md index cc1f50a..f419810 100644 --- a/commands/autoloop.md +++ b/commands/autoloop.md @@ -27,6 +27,7 @@ If the input doesn't contain a metric command, use `AskUserQuestion` to collect: 3. **Direction** — higher-is-better or lower-is-better 4. **Iterations** — how many cycles (default 10) 5. **Scope** — file/package constraints (optional) +6. **Guard** — optional safety-net command that must always pass (e.g., `npm test`, `tsc --noEmit`). Changes that improve the metric but break the guard are reverted. ## Step 2: Baseline @@ -51,11 +52,33 @@ Compare baseline vs measurement using the direction: - lower-is-better: new < old → IMPROVED - Equal or failed → REGRESSED +## Step 6.5: Guard Check (if guard command is set) + +If the metric improved, run the guard command: + +```bash +{guard_command} 2>&1 +``` + +- Guard passes (exit 0) → proceed to Step 7 as IMPROVED +- Guard fails → treat as REGRESSED regardless of metric improvement. Log: "Metric improved but guard failed — reverting to protect invariant." + ## Step 7: Keep or Revert - **IMPROVED** → stage only modified files + `git commit`, update scratchpad, update baseline to new number, loop back to Step 3 - **REGRESSED** → `git checkout -- ` (no `git clean -fd` — protect untracked work), update scratchpad with failure reason, loop back to Step 3 +### Escalation: Repeated Failures + +Track consecutive failures on the same target area. After 3 failed attempts at the same hypothesis or file: + +1. **Log** what was tried and why each approach failed +2. **Skip** — move the hypothesis to a "blocked" list in the scratchpad +3. **Pivot** — choose a completely different hypothesis targeting different files +4. **Report** — include blocked items in the final report with context for manual investigation + +Never loop on the same failing approach. Each attempt must use a materially different strategy. + ## Step 8: Report After all iterations or budget exhausted: @@ -79,3 +102,5 @@ After all iterations or budget exhausted: - Use the scratchpad to prevent repeating failures - The metric command must be identical in baseline and measure steps - Update the baseline number after each kept change (so the next cycle compares against the new state, not the original) +- Guard failures override metric improvements — the invariant is sacred +- 3 failures on the same target → skip and pivot, don't grind diff --git a/commands/deep-research.md b/commands/deep-research.md index 0ca813f..39a5a2c 100644 --- a/commands/deep-research.md +++ b/commands/deep-research.md @@ -71,6 +71,24 @@ Generate 2-4 mutually exclusive hypotheses. Include at least one contrarian hypo For EACH hypothesis, search specifically for evidence that would DISPROVE it. This is the critical ACH step — you're trying to kill each hypothesis, not confirm it. +## Step 7.5: Adversarial Debate (optional — use when hypotheses are close or stakes are high) + +When two or more hypotheses survive disconfirmation with similar evidence profiles, run an adversarial refinement cycle to stress-test them: + +1. **Advocate** — For each surviving hypothesis, write the strongest possible case. Cite specific evidence. Assume this hypothesis is correct and explain away contradictions. + +2. **Critic** — For each advocacy, write a targeted attack. Find the weakest link in the argument. Identify what the advocate glossed over or explained away too easily. Name the single observation that would kill this hypothesis. + +3. **Synthesize** — Given the advocacy and critique for all hypotheses, ask: Is there a composite hypothesis that accounts for more evidence than any individual one? Can the strongest elements be combined? + +4. **Blind Judge** — Present the surviving candidates (original + any composite) with randomized labels (Candidate A, B, C — not in hypothesis order). Evaluate on: + - Fewest unexplained observations + - Most falsifiable (can be tested) + - Least reliance on coincidence + Pick a winner. If no clear winner, note the deadlock and carry both forward with explicit uncertainty. + +Skip this step if: one hypothesis clearly dominates after Step 7, or the question is low-stakes. + ## Step 8: Build Evidence Matrix ``` @@ -128,3 +146,5 @@ Review for: genuine disconfirmation effort, missed perspectives, source over-wei - Self-critique before output - Cite everything - Be honest about uncertainty +- Use adversarial debate when hypotheses are close — don't just pick the first plausible one +- Blind judge evaluations use randomized labels to prevent ordering bias diff --git a/commands/tri-debug.md b/commands/tri-debug.md index c5a4451..2b34912 100644 --- a/commands/tri-debug.md +++ b/commands/tri-debug.md @@ -138,10 +138,68 @@ wait 1. ... ``` +## Investigation Techniques + +When building the debug prompt or analyzing results, apply the most appropriate technique: + +| Technique | When to Use | How | +|-----------|-------------|-----| +| **Binary search** | Bug exists somewhere in a range of changes/code | Bisect the search space — disable half, test, narrow | +| **Differential debugging** | "It worked before" | Compare working vs broken state — `git diff`, env diff, config diff | +| **Minimal reproduction** | Complex bug with many variables | Strip away everything unrelated until the simplest trigger remains | +| **Trace execution** | Control flow is unclear | Add logging or step through — follow the actual path, not the assumed one | +| **Working backwards** | You know the symptom but not the cause | Start at the error, trace data/control flow backwards to the source | +| **5 Whys** | Surface fix isn't enough | Ask "why?" at each layer: symptom → immediate cause → deeper cause → root cause → systemic issue | + +## Domain-Specific Debugging Checklists + +Include the relevant checklist in the agent prompt when the bug domain is identifiable: + +### API Bugs +- Does the route exist and match the HTTP method? +- Is auth middleware applied and in the correct order? +- Does the request body parse correctly (Content-Type header)? +- Are 4xx vs 5xx responses distinguishable? Is error shape consistent? +- Are query parameters validated and typed? + +### Database Bugs +- Is the query correct? Run it manually with `EXPLAIN ANALYZE` +- Are migrations up to date? Check for schema drift +- Connection pool exhaustion? Check pool size vs concurrent requests +- Transaction isolation — are reads seeing stale data? +- N+1 queries? Log SQL count per request + +### Auth/Authorization Bugs +- Token expired vs invalid vs missing — which case? +- Middleware ordering — does auth run before the handler that needs it? +- Role/permission check — is the check on the right resource? +- Session vs token mismatch after deploy? + +### Async/Concurrency Bugs +- Race condition? Can two operations interleave on shared state? +- Deadlock? Are locks acquired in inconsistent order? +- Unhandled promise rejection or missing `await`? +- Event listener leak? Check listener count over time + +### Performance Bugs +- Profile first — the slow part is almost never where you think +- Check for N+1 queries, missing indexes, unbounded loops +- Memory leak? Compare heap snapshots over time +- Connection pool or thread pool exhaustion? + +| Symptom | Likely Cause | Investigation | +|---------|--------------|---------------| +| Slow API response | N+1 queries | Log SQL count per request | +| Slow page render | Expensive recomputation | Profile render cycle | +| Gradual memory growth | Leak (listeners, connections) | Heap snapshots over time | +| Intermittent slowness | Lock contention / pool exhaustion | Connection pool metrics | + ## Rules - Claude always runs — Codex and Gemini are optional - If only Claude is available, still provide the full report format (just one perspective) - Report which agents participated - If agents disagree on root cause, present all hypotheses ranked by evidence +- When agents disagree, apply the 5 Whys to the consensus symptoms to find a deeper shared root cause +- Include the relevant domain checklist in each agent's prompt when the bug category is identifiable - Clean up temp files after diff --git a/commands/tri-test-gen.md b/commands/tri-test-gen.md index 98dd405..5ebe492 100644 --- a/commands/tri-test-gen.md +++ b/commands/tri-test-gen.md @@ -49,6 +49,19 @@ Generate a comprehensive test suite for the following code. - Match existing test patterns in the repo. - Write tests that actually run — no placeholder assertions. +For each public function/method, systematically apply these scenario expansion techniques: + +| Technique | Description | Example | +|-----------|-------------|---------| +| What-if | Change one variable from the happy path | "What if the input is empty string instead of valid?" | +| Boundary | Push values to limits | "0, -1, MAX_INT, empty array, single element" | +| Interruption | Inject failure mid-flow | "What if the network drops mid-request?" | +| Ordering | Change sequence of operations | "What if step 2 happens before step 1?" | +| Missing data | Remove expected input | "What if the required field is null/undefined?" | +| Stale data | Use outdated information | "What if the cached value changed between read and use?" | + +Prioritize: boundary conditions > missing/null data > error paths > ordering > interruption > stale data. + Target: {target_files} Code: {source_code} ``` From 2049bda36b853d24949752fe259e959cffd43c72 Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Tue, 7 Apr 2026 10:47:16 -0400 Subject: [PATCH 2/3] Address tri-agent review findings across all four commands autoloop: - Add baseline guard check in Step 2 (fail-fast if invariant broken) - Add timeout to guard execution (120s default) - Require guard commands to be side-effect free - Define escalation trigger concretely (same primary modified file) - Remove subjective "sacred" wording deep-research: - Move adversarial debate after evidence matrix (was Step 7.5, now 8.5) - Require matrix as input to judge evaluation - Relax "mutually exclusive" to "competing" hypotheses - Composites replace parent hypotheses and must be rescored - Acknowledge blind judge is a heuristic nudge, not true blinding - Add concrete skip threshold (2+ fewer inconsistencies) tri-debug: - Extract domain checklists to commands/references/debug-checklists.md - Fix contradictory "consensus symptoms" rule to distinguish symptom-agreement from full disagreement - Expand vague auth checklist item tri-test-gen: - Move scenario expansion to orchestrator guidance (Step 2.5) - Only inject applicable techniques per target, not full table - Add integration-test caveat for interruption/stale-data - Respect existing repo test conventions over default priority --- commands/autoloop.md | 16 +++++---- commands/deep-research.md | 42 ++++++++++++------------ commands/references/debug-checklists.md | 42 ++++++++++++++++++++++++ commands/tri-debug.md | 43 ++----------------------- commands/tri-test-gen.md | 33 +++++++++++-------- 5 files changed, 94 insertions(+), 82 deletions(-) create mode 100644 commands/references/debug-checklists.md diff --git a/commands/autoloop.md b/commands/autoloop.md index f419810..41e3953 100644 --- a/commands/autoloop.md +++ b/commands/autoloop.md @@ -27,12 +27,14 @@ If the input doesn't contain a metric command, use `AskUserQuestion` to collect: 3. **Direction** — higher-is-better or lower-is-better 4. **Iterations** — how many cycles (default 10) 5. **Scope** — file/package constraints (optional) -6. **Guard** — optional safety-net command that must always pass (e.g., `npm test`, `tsc --noEmit`). Changes that improve the metric but break the guard are reverted. +6. **Guard** — optional safety-net command that must always pass (e.g., `npm test`, `tsc --noEmit`). Changes that improve the metric but break the guard are treated as regressions and reverted. Guard commands must be side-effect free (no generated files, snapshots, or caches that persist between runs). ## Step 2: Baseline Run the metric command. Record the starting number and direction. +If a guard command is set, run it now. If the guard fails at baseline, stop and tell the user — the invariant must hold before the loop can start. + ## Step 3: Audit Analyze the codebase for the single highest-impact change. Read the scratchpad to avoid repeating failed approaches. Output one hypothesis with target files. @@ -54,14 +56,14 @@ Compare baseline vs measurement using the direction: ## Step 6.5: Guard Check (if guard command is set) -If the metric improved, run the guard command: +If the metric improved, run the guard command with a timeout: ```bash -{guard_command} 2>&1 +timeout 120 {guard_command} 2>&1 ``` - Guard passes (exit 0) → proceed to Step 7 as IMPROVED -- Guard fails → treat as REGRESSED regardless of metric improvement. Log: "Metric improved but guard failed — reverting to protect invariant." +- Guard fails or times out → treat as REGRESSED regardless of metric improvement. Log: "Metric improved but guard failed — reverting to protect invariant." ## Step 7: Keep or Revert @@ -70,7 +72,7 @@ If the metric improved, run the guard command: ### Escalation: Repeated Failures -Track consecutive failures on the same target area. After 3 failed attempts at the same hypothesis or file: +Track consecutive failures by primary modified file. After 3 failed attempts where the same file is the main edit target: 1. **Log** what was tried and why each approach failed 2. **Skip** — move the hypothesis to a "blocked" list in the scratchpad @@ -102,5 +104,5 @@ After all iterations or budget exhausted: - Use the scratchpad to prevent repeating failures - The metric command must be identical in baseline and measure steps - Update the baseline number after each kept change (so the next cycle compares against the new state, not the original) -- Guard failures override metric improvements — the invariant is sacred -- 3 failures on the same target → skip and pivot, don't grind +- Guard failures override metric improvements — never accept a change that breaks the guard +- 3 failures on the same file → skip and pivot, don't grind diff --git a/commands/deep-research.md b/commands/deep-research.md index 39a5a2c..5300a60 100644 --- a/commands/deep-research.md +++ b/commands/deep-research.md @@ -65,30 +65,12 @@ Fetch top 5-8 URLs with Jina Reader. Extract atomic claims (subject-predicate-ob ## Step 6: Generate Competing Hypotheses -Generate 2-4 mutually exclusive hypotheses. Include at least one contrarian hypothesis. +Generate 2-4 competing hypotheses. Include at least one contrarian hypothesis. ## Step 7: Directed Disconfirmation For EACH hypothesis, search specifically for evidence that would DISPROVE it. This is the critical ACH step — you're trying to kill each hypothesis, not confirm it. -## Step 7.5: Adversarial Debate (optional — use when hypotheses are close or stakes are high) - -When two or more hypotheses survive disconfirmation with similar evidence profiles, run an adversarial refinement cycle to stress-test them: - -1. **Advocate** — For each surviving hypothesis, write the strongest possible case. Cite specific evidence. Assume this hypothesis is correct and explain away contradictions. - -2. **Critic** — For each advocacy, write a targeted attack. Find the weakest link in the argument. Identify what the advocate glossed over or explained away too easily. Name the single observation that would kill this hypothesis. - -3. **Synthesize** — Given the advocacy and critique for all hypotheses, ask: Is there a composite hypothesis that accounts for more evidence than any individual one? Can the strongest elements be combined? - -4. **Blind Judge** — Present the surviving candidates (original + any composite) with randomized labels (Candidate A, B, C — not in hypothesis order). Evaluate on: - - Fewest unexplained observations - - Most falsifiable (can be tested) - - Least reliance on coincidence - Pick a winner. If no clear winner, note the deadlock and carry both forward with explicit uncertainty. - -Skip this step if: one hypothesis clearly dominates after Step 7, or the question is low-stakes. - ## Step 8: Build Evidence Matrix ``` @@ -101,6 +83,24 @@ CC=Strongly Consistent, C=Consistent, N=Neutral, I=Inconsistent, II=Strongly Inc Score by FEWEST inconsistencies (not most consistencies). +## Step 8.5: Adversarial Debate (optional — use when hypotheses are close or stakes are high) + +When two or more hypotheses survive with similar scores in the evidence matrix, run an adversarial refinement cycle to stress-test them. Use the completed matrix as input. + +1. **Advocate** — For each surviving hypothesis, write the strongest possible case. Cite specific evidence from the matrix. Assume this hypothesis is correct and explain away inconsistencies. + +2. **Critic** — For each advocacy, write a targeted attack. Find the weakest link in the argument. Identify what the advocate glossed over or explained away too easily. Name the single observation that would kill this hypothesis. + +3. **Synthesize** — Given the advocacy and critique for all hypotheses, ask: Is there a composite hypothesis that accounts for more evidence than any individual one? If so, the composite replaces its parent hypotheses — rescore it against the evidence matrix as a new candidate. + +4. **Judge** — Present the surviving candidates (original + any composite) with randomized labels (Candidate A, B, C — not in hypothesis order) as a heuristic to reduce anchoring bias. Note: in a single-agent context this is a nudge, not a true blind. Evaluate using the evidence matrix on: + - Fewest inconsistencies in the matrix + - Most falsifiable (can be tested) + - Least reliance on coincidence + Pick a winner. If no clear winner, note the deadlock and carry both forward with explicit uncertainty. + +Skip this step if: one hypothesis has 2+ fewer inconsistencies than the runner-up in the matrix, or the research question is informational rather than decision-driving. + ## Step 9: Sensitivity Check Identify linchpin evidence — what single fact, if wrong, would change the conclusion? @@ -146,5 +146,5 @@ Review for: genuine disconfirmation effort, missed perspectives, source over-wei - Self-critique before output - Cite everything - Be honest about uncertainty -- Use adversarial debate when hypotheses are close — don't just pick the first plausible one -- Blind judge evaluations use randomized labels to prevent ordering bias +- Use adversarial debate when hypotheses score similarly in the evidence matrix — don't just pick the first plausible one +- Judge evaluations use randomized labels as an anchoring-bias heuristic (not a true blind in single-agent context) diff --git a/commands/references/debug-checklists.md b/commands/references/debug-checklists.md new file mode 100644 index 0000000..b2bb1cd --- /dev/null +++ b/commands/references/debug-checklists.md @@ -0,0 +1,42 @@ +# Debug Checklists + +Domain-specific checklists for inclusion in debug agent prompts. Include only the relevant checklist based on the identified bug domain. + +## API Bugs +- Does the route exist and match the HTTP method? +- Is auth middleware applied and in the correct order? +- Does the request body parse correctly (Content-Type header)? +- Are 4xx vs 5xx responses distinguishable? Is error shape consistent? +- Are query parameters validated and typed? + +## Database Bugs +- Is the query correct? Run it manually with `EXPLAIN ANALYZE` +- Are migrations up to date? Check for schema drift +- Connection pool exhaustion? Check pool size vs concurrent requests +- Transaction isolation — are reads seeing stale data? +- N+1 queries? Log SQL count per request + +## Auth/Authorization Bugs +- Token expired vs invalid vs missing — which case? +- Middleware ordering — does auth run before the handler that needs it? +- Role/permission check — is the check on the right resource? +- Session store cleared on deploy while long-lived tokens persist? Token signing key rotated without invalidating existing tokens? + +## Async/Concurrency Bugs +- Race condition? Can two operations interleave on shared state? +- Deadlock? Are locks acquired in inconsistent order? +- Unhandled promise rejection or missing `await`? +- Event listener leak? Check listener count over time + +## Performance Bugs +- Profile first — the slow part is almost never where you think +- Check for N+1 queries, missing indexes, unbounded loops +- Memory leak? Compare heap snapshots over time +- Connection pool or thread pool exhaustion? + +| Symptom | Likely Cause | Investigation | +|---------|--------------|---------------| +| Slow API response | N+1 queries | Log SQL count per request | +| Slow page render | Expensive recomputation | Profile render cycle | +| Gradual memory growth | Leak (listeners, connections) | Heap snapshots over time | +| Intermittent slowness | Lock contention / pool exhaustion | Connection pool metrics | diff --git a/commands/tri-debug.md b/commands/tri-debug.md index 2b34912..4ba71a0 100644 --- a/commands/tri-debug.md +++ b/commands/tri-debug.md @@ -153,46 +153,7 @@ When building the debug prompt or analyzing results, apply the most appropriate ## Domain-Specific Debugging Checklists -Include the relevant checklist in the agent prompt when the bug domain is identifiable: - -### API Bugs -- Does the route exist and match the HTTP method? -- Is auth middleware applied and in the correct order? -- Does the request body parse correctly (Content-Type header)? -- Are 4xx vs 5xx responses distinguishable? Is error shape consistent? -- Are query parameters validated and typed? - -### Database Bugs -- Is the query correct? Run it manually with `EXPLAIN ANALYZE` -- Are migrations up to date? Check for schema drift -- Connection pool exhaustion? Check pool size vs concurrent requests -- Transaction isolation — are reads seeing stale data? -- N+1 queries? Log SQL count per request - -### Auth/Authorization Bugs -- Token expired vs invalid vs missing — which case? -- Middleware ordering — does auth run before the handler that needs it? -- Role/permission check — is the check on the right resource? -- Session vs token mismatch after deploy? - -### Async/Concurrency Bugs -- Race condition? Can two operations interleave on shared state? -- Deadlock? Are locks acquired in inconsistent order? -- Unhandled promise rejection or missing `await`? -- Event listener leak? Check listener count over time - -### Performance Bugs -- Profile first — the slow part is almost never where you think -- Check for N+1 queries, missing indexes, unbounded loops -- Memory leak? Compare heap snapshots over time -- Connection pool or thread pool exhaustion? - -| Symptom | Likely Cause | Investigation | -|---------|--------------|---------------| -| Slow API response | N+1 queries | Log SQL count per request | -| Slow page render | Expensive recomputation | Profile render cycle | -| Gradual memory growth | Leak (listeners, connections) | Heap snapshots over time | -| Intermittent slowness | Lock contention / pool exhaustion | Connection pool metrics | +When the bug domain is identifiable (API, database, auth, async, performance), read the relevant section from `commands/references/debug-checklists.md` and include it in each agent's prompt. Only include the matching domain — don't load the full file. ## Rules @@ -200,6 +161,6 @@ Include the relevant checklist in the agent prompt when the bug domain is identi - If only Claude is available, still provide the full report format (just one perspective) - Report which agents participated - If agents disagree on root cause, present all hypotheses ranked by evidence -- When agents disagree, apply the 5 Whys to the consensus symptoms to find a deeper shared root cause +- When agents agree on symptoms but disagree on root cause, apply the 5 Whys to the shared symptoms to converge on a deeper cause. If they disagree on symptoms too, prioritize verifying symptoms through additional logs or traces before root-cause analysis - Include the relevant domain checklist in each agent's prompt when the bug category is identifiable - Clean up temp files after diff --git a/commands/tri-test-gen.md b/commands/tri-test-gen.md index 5ebe492..c99e2bb 100644 --- a/commands/tri-test-gen.md +++ b/commands/tri-test-gen.md @@ -40,6 +40,25 @@ HAS_GEMINI_CLI=$(command -v gemini && echo "yes" || echo "no") Prefer plugin over CLI. +## Step 2.5: Scenario Expansion (orchestrator guidance — not injected into sub-agent prompts) + +When analyzing the target in Step 1, identify which scenario expansion techniques are highest-yield for each public function/method based on its behavior and risk profile. Use this to guide prompt construction — don't apply every technique to every function. + +| Technique | When high-yield | Example | +|-----------|-----------------|---------| +| Missing data | Functions with required parameters | "What if the required field is null/undefined?" | +| Boundary | Functions with numeric/collection inputs | "0, -1, MAX_INT, empty array, single element" | +| What-if | Functions with branching logic | "What if the input is empty string instead of valid?" | +| Ordering | Functions called in sequences or pipelines | "What if step 2 happens before step 1?" | +| Interruption | I/O or network-dependent functions | "What if the network drops mid-request?" | +| Stale data | Functions using caches or shared state | "What if the cached value changed between read and use?" | + +Prioritize unless existing test patterns in the repo establish a different convention: missing/null data > boundary conditions > error paths > ordering > interruption > stale data. + +Interruption and stale data scenarios often require integration-level test infrastructure (mocking I/O, time manipulation). Skip these when generating pure unit tests. + +Include only the applicable techniques in each agent's prompt — not the full table. + ## Step 3: Build the Prompt ``` @@ -48,19 +67,7 @@ Generate a comprehensive test suite for the following code. - Use {framework} conventions. - Match existing test patterns in the repo. - Write tests that actually run — no placeholder assertions. - -For each public function/method, systematically apply these scenario expansion techniques: - -| Technique | Description | Example | -|-----------|-------------|---------| -| What-if | Change one variable from the happy path | "What if the input is empty string instead of valid?" | -| Boundary | Push values to limits | "0, -1, MAX_INT, empty array, single element" | -| Interruption | Inject failure mid-flow | "What if the network drops mid-request?" | -| Ordering | Change sequence of operations | "What if step 2 happens before step 1?" | -| Missing data | Remove expected input | "What if the required field is null/undefined?" | -| Stale data | Use outdated information | "What if the cached value changed between read and use?" | - -Prioritize: boundary conditions > missing/null data > error paths > ordering > interruption > stale data. +{scenario_guidance — applicable techniques from Step 2.5, tailored to this target} Target: {target_files} Code: {source_code} From db857b3b56c7e4830e624ba3cd0c77171980a30c Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Tue, 7 Apr 2026 10:56:12 -0400 Subject: [PATCH 3/3] Address PR review-toolkit findings autoloop: - Document intentional guard skip on regressions (not a gap) - Integrate per-file failure counter into Step 7 REGRESSED flow deep-research: - Reorder judge criteria: evidence-fit primary, falsifiability tiebreaker tri-debug: - Fix cross-reference path to be relative to commands/ directory tri-test-gen: - Fix contradictory parenthetical on Step 2.5 injection scope - Replace phantom "error paths" with "what-if" in priority list debug-checklists: - Add sub-heading for performance symptom lookup table --- commands/autoloop.md | 4 ++-- commands/deep-research.md | 4 ++-- commands/references/debug-checklists.md | 2 ++ commands/tri-debug.md | 2 +- commands/tri-test-gen.md | 4 ++-- 5 files changed, 9 insertions(+), 7 deletions(-) diff --git a/commands/autoloop.md b/commands/autoloop.md index 41e3953..3f148b6 100644 --- a/commands/autoloop.md +++ b/commands/autoloop.md @@ -56,7 +56,7 @@ Compare baseline vs measurement using the direction: ## Step 6.5: Guard Check (if guard command is set) -If the metric improved, run the guard command with a timeout: +Guard is only evaluated when the metric improved. Regressions are already reverted in Step 7, so running the guard on them would be wasted work. If the metric improved, run the guard with a timeout: ```bash timeout 120 {guard_command} 2>&1 @@ -68,7 +68,7 @@ timeout 120 {guard_command} 2>&1 ## Step 7: Keep or Revert - **IMPROVED** → stage only modified files + `git commit`, update scratchpad, update baseline to new number, loop back to Step 3 -- **REGRESSED** → `git checkout -- ` (no `git clean -fd` — protect untracked work), update scratchpad with failure reason, loop back to Step 3 +- **REGRESSED** → `git checkout -- ` (no `git clean -fd` — protect untracked work), update scratchpad with failure reason, increment per-file failure counter (see Escalation below), loop back to Step 3 ### Escalation: Repeated Failures diff --git a/commands/deep-research.md b/commands/deep-research.md index 5300a60..c9b2796 100644 --- a/commands/deep-research.md +++ b/commands/deep-research.md @@ -94,9 +94,9 @@ When two or more hypotheses survive with similar scores in the evidence matrix, 3. **Synthesize** — Given the advocacy and critique for all hypotheses, ask: Is there a composite hypothesis that accounts for more evidence than any individual one? If so, the composite replaces its parent hypotheses — rescore it against the evidence matrix as a new candidate. 4. **Judge** — Present the surviving candidates (original + any composite) with randomized labels (Candidate A, B, C — not in hypothesis order) as a heuristic to reduce anchoring bias. Note: in a single-agent context this is a nudge, not a true blind. Evaluate using the evidence matrix on: - - Fewest inconsistencies in the matrix - - Most falsifiable (can be tested) + - Fewest inconsistencies in the matrix (primary) - Least reliance on coincidence + - Most falsifiable (tiebreaker — prefer hypotheses that can be tested) Pick a winner. If no clear winner, note the deadlock and carry both forward with explicit uncertainty. Skip this step if: one hypothesis has 2+ fewer inconsistencies than the runner-up in the matrix, or the research question is informational rather than decision-driving. diff --git a/commands/references/debug-checklists.md b/commands/references/debug-checklists.md index b2bb1cd..010f48c 100644 --- a/commands/references/debug-checklists.md +++ b/commands/references/debug-checklists.md @@ -34,6 +34,8 @@ Domain-specific checklists for inclusion in debug agent prompts. Include only th - Memory leak? Compare heap snapshots over time - Connection pool or thread pool exhaustion? +### Quick Symptom Lookup + | Symptom | Likely Cause | Investigation | |---------|--------------|---------------| | Slow API response | N+1 queries | Log SQL count per request | diff --git a/commands/tri-debug.md b/commands/tri-debug.md index 4ba71a0..7640bea 100644 --- a/commands/tri-debug.md +++ b/commands/tri-debug.md @@ -153,7 +153,7 @@ When building the debug prompt or analyzing results, apply the most appropriate ## Domain-Specific Debugging Checklists -When the bug domain is identifiable (API, database, auth, async, performance), read the relevant section from `commands/references/debug-checklists.md` and include it in each agent's prompt. Only include the matching domain — don't load the full file. +When the bug domain is identifiable (API, database, auth, async, performance), read the relevant section from `references/debug-checklists.md` (relative to this `commands/` directory) and include it in each agent's prompt. Only include the matching domain — don't load the full file. ## Rules diff --git a/commands/tri-test-gen.md b/commands/tri-test-gen.md index c99e2bb..f5d925f 100644 --- a/commands/tri-test-gen.md +++ b/commands/tri-test-gen.md @@ -40,7 +40,7 @@ HAS_GEMINI_CLI=$(command -v gemini && echo "yes" || echo "no") Prefer plugin over CLI. -## Step 2.5: Scenario Expansion (orchestrator guidance — not injected into sub-agent prompts) +## Step 2.5: Scenario Expansion (orchestrator selects applicable techniques — only selected items are injected into sub-agent prompts) When analyzing the target in Step 1, identify which scenario expansion techniques are highest-yield for each public function/method based on its behavior and risk profile. Use this to guide prompt construction — don't apply every technique to every function. @@ -53,7 +53,7 @@ When analyzing the target in Step 1, identify which scenario expansion technique | Interruption | I/O or network-dependent functions | "What if the network drops mid-request?" | | Stale data | Functions using caches or shared state | "What if the cached value changed between read and use?" | -Prioritize unless existing test patterns in the repo establish a different convention: missing/null data > boundary conditions > error paths > ordering > interruption > stale data. +Prioritize unless existing test patterns in the repo establish a different convention: missing/null data > boundary conditions > what-if > ordering > interruption > stale data. Interruption and stale data scenarios often require integration-level test infrastructure (mocking I/O, time manipulation). Skip these when generating pure unit tests.