diff --git a/commands/autoloop.md b/commands/autoloop.md index cc1f50a..3f148b6 100644 --- a/commands/autoloop.md +++ b/commands/autoloop.md @@ -27,11 +27,14 @@ If the input doesn't contain a metric command, use `AskUserQuestion` to collect: 3. **Direction** — higher-is-better or lower-is-better 4. **Iterations** — how many cycles (default 10) 5. **Scope** — file/package constraints (optional) +6. **Guard** — optional safety-net command that must always pass (e.g., `npm test`, `tsc --noEmit`). Changes that improve the metric but break the guard are treated as regressions and reverted. Guard commands must be side-effect free (no generated files, snapshots, or caches that persist between runs). ## Step 2: Baseline Run the metric command. Record the starting number and direction. +If a guard command is set, run it now. If the guard fails at baseline, stop and tell the user — the invariant must hold before the loop can start. + ## Step 3: Audit Analyze the codebase for the single highest-impact change. Read the scratchpad to avoid repeating failed approaches. Output one hypothesis with target files. @@ -51,10 +54,32 @@ Compare baseline vs measurement using the direction: - lower-is-better: new < old → IMPROVED - Equal or failed → REGRESSED +## Step 6.5: Guard Check (if guard command is set) + +Guard is only evaluated when the metric improved. Regressions are already reverted in Step 7, so running the guard on them would be wasted work. If the metric improved, run the guard with a timeout: + +```bash +timeout 120 {guard_command} 2>&1 +``` + +- Guard passes (exit 0) → proceed to Step 7 as IMPROVED +- Guard fails or times out → treat as REGRESSED regardless of metric improvement. Log: "Metric improved but guard failed — reverting to protect invariant." + ## Step 7: Keep or Revert - **IMPROVED** → stage only modified files + `git commit`, update scratchpad, update baseline to new number, loop back to Step 3 -- **REGRESSED** → `git checkout -- ` (no `git clean -fd` — protect untracked work), update scratchpad with failure reason, loop back to Step 3 +- **REGRESSED** → `git checkout -- ` (no `git clean -fd` — protect untracked work), update scratchpad with failure reason, increment per-file failure counter (see Escalation below), loop back to Step 3 + +### Escalation: Repeated Failures + +Track consecutive failures by primary modified file. After 3 failed attempts where the same file is the main edit target: + +1. **Log** what was tried and why each approach failed +2. **Skip** — move the hypothesis to a "blocked" list in the scratchpad +3. **Pivot** — choose a completely different hypothesis targeting different files +4. **Report** — include blocked items in the final report with context for manual investigation + +Never loop on the same failing approach. Each attempt must use a materially different strategy. ## Step 8: Report @@ -79,3 +104,5 @@ After all iterations or budget exhausted: - Use the scratchpad to prevent repeating failures - The metric command must be identical in baseline and measure steps - Update the baseline number after each kept change (so the next cycle compares against the new state, not the original) +- Guard failures override metric improvements — never accept a change that breaks the guard +- 3 failures on the same file → skip and pivot, don't grind diff --git a/commands/deep-research.md b/commands/deep-research.md index 0ca813f..c9b2796 100644 --- a/commands/deep-research.md +++ b/commands/deep-research.md @@ -65,7 +65,7 @@ Fetch top 5-8 URLs with Jina Reader. Extract atomic claims (subject-predicate-ob ## Step 6: Generate Competing Hypotheses -Generate 2-4 mutually exclusive hypotheses. Include at least one contrarian hypothesis. +Generate 2-4 competing hypotheses. Include at least one contrarian hypothesis. ## Step 7: Directed Disconfirmation @@ -83,6 +83,24 @@ CC=Strongly Consistent, C=Consistent, N=Neutral, I=Inconsistent, II=Strongly Inc Score by FEWEST inconsistencies (not most consistencies). +## Step 8.5: Adversarial Debate (optional — use when hypotheses are close or stakes are high) + +When two or more hypotheses survive with similar scores in the evidence matrix, run an adversarial refinement cycle to stress-test them. Use the completed matrix as input. + +1. **Advocate** — For each surviving hypothesis, write the strongest possible case. Cite specific evidence from the matrix. Assume this hypothesis is correct and explain away inconsistencies. + +2. **Critic** — For each advocacy, write a targeted attack. Find the weakest link in the argument. Identify what the advocate glossed over or explained away too easily. Name the single observation that would kill this hypothesis. + +3. **Synthesize** — Given the advocacy and critique for all hypotheses, ask: Is there a composite hypothesis that accounts for more evidence than any individual one? If so, the composite replaces its parent hypotheses — rescore it against the evidence matrix as a new candidate. + +4. **Judge** — Present the surviving candidates (original + any composite) with randomized labels (Candidate A, B, C — not in hypothesis order) as a heuristic to reduce anchoring bias. Note: in a single-agent context this is a nudge, not a true blind. Evaluate using the evidence matrix on: + - Fewest inconsistencies in the matrix (primary) + - Least reliance on coincidence + - Most falsifiable (tiebreaker — prefer hypotheses that can be tested) + Pick a winner. If no clear winner, note the deadlock and carry both forward with explicit uncertainty. + +Skip this step if: one hypothesis has 2+ fewer inconsistencies than the runner-up in the matrix, or the research question is informational rather than decision-driving. + ## Step 9: Sensitivity Check Identify linchpin evidence — what single fact, if wrong, would change the conclusion? @@ -128,3 +146,5 @@ Review for: genuine disconfirmation effort, missed perspectives, source over-wei - Self-critique before output - Cite everything - Be honest about uncertainty +- Use adversarial debate when hypotheses score similarly in the evidence matrix — don't just pick the first plausible one +- Judge evaluations use randomized labels as an anchoring-bias heuristic (not a true blind in single-agent context) diff --git a/commands/references/debug-checklists.md b/commands/references/debug-checklists.md new file mode 100644 index 0000000..010f48c --- /dev/null +++ b/commands/references/debug-checklists.md @@ -0,0 +1,44 @@ +# Debug Checklists + +Domain-specific checklists for inclusion in debug agent prompts. Include only the relevant checklist based on the identified bug domain. + +## API Bugs +- Does the route exist and match the HTTP method? +- Is auth middleware applied and in the correct order? +- Does the request body parse correctly (Content-Type header)? +- Are 4xx vs 5xx responses distinguishable? Is error shape consistent? +- Are query parameters validated and typed? + +## Database Bugs +- Is the query correct? Run it manually with `EXPLAIN ANALYZE` +- Are migrations up to date? Check for schema drift +- Connection pool exhaustion? Check pool size vs concurrent requests +- Transaction isolation — are reads seeing stale data? +- N+1 queries? Log SQL count per request + +## Auth/Authorization Bugs +- Token expired vs invalid vs missing — which case? +- Middleware ordering — does auth run before the handler that needs it? +- Role/permission check — is the check on the right resource? +- Session store cleared on deploy while long-lived tokens persist? Token signing key rotated without invalidating existing tokens? + +## Async/Concurrency Bugs +- Race condition? Can two operations interleave on shared state? +- Deadlock? Are locks acquired in inconsistent order? +- Unhandled promise rejection or missing `await`? +- Event listener leak? Check listener count over time + +## Performance Bugs +- Profile first — the slow part is almost never where you think +- Check for N+1 queries, missing indexes, unbounded loops +- Memory leak? Compare heap snapshots over time +- Connection pool or thread pool exhaustion? + +### Quick Symptom Lookup + +| Symptom | Likely Cause | Investigation | +|---------|--------------|---------------| +| Slow API response | N+1 queries | Log SQL count per request | +| Slow page render | Expensive recomputation | Profile render cycle | +| Gradual memory growth | Leak (listeners, connections) | Heap snapshots over time | +| Intermittent slowness | Lock contention / pool exhaustion | Connection pool metrics | diff --git a/commands/tri-debug.md b/commands/tri-debug.md index c5a4451..7640bea 100644 --- a/commands/tri-debug.md +++ b/commands/tri-debug.md @@ -138,10 +138,29 @@ wait 1. ... ``` +## Investigation Techniques + +When building the debug prompt or analyzing results, apply the most appropriate technique: + +| Technique | When to Use | How | +|-----------|-------------|-----| +| **Binary search** | Bug exists somewhere in a range of changes/code | Bisect the search space — disable half, test, narrow | +| **Differential debugging** | "It worked before" | Compare working vs broken state — `git diff`, env diff, config diff | +| **Minimal reproduction** | Complex bug with many variables | Strip away everything unrelated until the simplest trigger remains | +| **Trace execution** | Control flow is unclear | Add logging or step through — follow the actual path, not the assumed one | +| **Working backwards** | You know the symptom but not the cause | Start at the error, trace data/control flow backwards to the source | +| **5 Whys** | Surface fix isn't enough | Ask "why?" at each layer: symptom → immediate cause → deeper cause → root cause → systemic issue | + +## Domain-Specific Debugging Checklists + +When the bug domain is identifiable (API, database, auth, async, performance), read the relevant section from `references/debug-checklists.md` (relative to this `commands/` directory) and include it in each agent's prompt. Only include the matching domain — don't load the full file. + ## Rules - Claude always runs — Codex and Gemini are optional - If only Claude is available, still provide the full report format (just one perspective) - Report which agents participated - If agents disagree on root cause, present all hypotheses ranked by evidence +- When agents agree on symptoms but disagree on root cause, apply the 5 Whys to the shared symptoms to converge on a deeper cause. If they disagree on symptoms too, prioritize verifying symptoms through additional logs or traces before root-cause analysis +- Include the relevant domain checklist in each agent's prompt when the bug category is identifiable - Clean up temp files after diff --git a/commands/tri-test-gen.md b/commands/tri-test-gen.md index 98dd405..f5d925f 100644 --- a/commands/tri-test-gen.md +++ b/commands/tri-test-gen.md @@ -40,6 +40,25 @@ HAS_GEMINI_CLI=$(command -v gemini && echo "yes" || echo "no") Prefer plugin over CLI. +## Step 2.5: Scenario Expansion (orchestrator selects applicable techniques — only selected items are injected into sub-agent prompts) + +When analyzing the target in Step 1, identify which scenario expansion techniques are highest-yield for each public function/method based on its behavior and risk profile. Use this to guide prompt construction — don't apply every technique to every function. + +| Technique | When high-yield | Example | +|-----------|-----------------|---------| +| Missing data | Functions with required parameters | "What if the required field is null/undefined?" | +| Boundary | Functions with numeric/collection inputs | "0, -1, MAX_INT, empty array, single element" | +| What-if | Functions with branching logic | "What if the input is empty string instead of valid?" | +| Ordering | Functions called in sequences or pipelines | "What if step 2 happens before step 1?" | +| Interruption | I/O or network-dependent functions | "What if the network drops mid-request?" | +| Stale data | Functions using caches or shared state | "What if the cached value changed between read and use?" | + +Prioritize unless existing test patterns in the repo establish a different convention: missing/null data > boundary conditions > what-if > ordering > interruption > stale data. + +Interruption and stale data scenarios often require integration-level test infrastructure (mocking I/O, time manipulation). Skip these when generating pure unit tests. + +Include only the applicable techniques in each agent's prompt — not the full table. + ## Step 3: Build the Prompt ``` @@ -48,6 +67,7 @@ Generate a comprehensive test suite for the following code. - Use {framework} conventions. - Match existing test patterns in the repo. - Write tests that actually run — no placeholder assertions. +{scenario_guidance — applicable techniques from Step 2.5, tailored to this target} Target: {target_files} Code: {source_code}