diff --git a/README.md b/README.md index 553cfce..8bf12fc 100644 --- a/README.md +++ b/README.md @@ -134,6 +134,7 @@ Automated propose → measure → keep/discard → repeat cycles. | `/self:perf` | Hypothesis-driven performance investigation — evidence, hypotheses, one-at-a-time testing | | `/self:migrate` | Incremental migration (JS→TS, class→hooks, etc.) with test gate | | `/self:audit` | Codebase audit inspired by karpathy/autoresearch — measure everything, rank hypotheses, report only | +| `/devkit:autoloop` | Autonomous improvement loop — audit, fix, measure, keep or revert, repeat | ### Multi-Agent Commands (Claude + optional Codex/Gemini) @@ -312,7 +313,7 @@ devkit/ │ └── plugin.json # Plugin metadata (name, version, author) ├── ROADMAP.md # Implemented features and future plans ├── PREFERENCES.md # Agent behavior guidelines -├── commands/ # 22 slash commands (tab-completable) +├── commands/ # 23 slash commands (tab-completable) │ ├── tri-*.md # Multi-agent dispatch (5) │ ├── self-*.md # Self-improvement loops (6) │ ├── pr-ready.md # PR preparation pipeline @@ -325,7 +326,7 @@ devkit/ │ ├── audit.md # Project health audit │ ├── repo-map.md # AST-based symbol index │ └── status.md # Health check -├── skills/ # 16 context-activated skills +├── skills/ # 17 context-activated skills │ ├── executing/SKILL.md # Principle: methodical execution │ ├── clean-code/SKILL.md # Principle: readability │ ├── dry/SKILL.md # Principle: don't repeat yourself @@ -361,7 +362,7 @@ devkit/ │ ├── lang-review.sh # Language-aware code quality (Go/TS/Rust/Python/Shell) │ ├── subagent-stop.sh # Subagent work verification │ └── stop-gate.sh # Consolidated quality gate (cross-domain + vet/lint) -├── workflows/ # 14 YAML workflow definitions +├── workflows/ # 15 YAML workflow definitions ├── presets/ # Reserved for future use ├── .github/workflows/ # CI/CD │ ├── ci.yml # Build + test + vet on push/PR diff --git a/ROADMAP.md b/ROADMAP.md index 37ab273..cd9e9b1 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -2,8 +2,8 @@ ## Implemented -- **22 slash commands** — Lifecycle workflows, self-improvement loops, multi-agent dispatch, project health audit, post-PR monitoring, AST repo mapping, autoresearch-inspired self-audit -- **16 context-activated skills** — 7 auto-trigger workflows (test-gen, doc-gen, changelog, onboard, research, deep-research, scrape) + 6 coding principles (executing, clean-code, DRY, YAGNI, dont-reinvent, stuck) + 2 tools (gcli, creating-workflows) + 1 iteration memory (scratchpad) +- **23 slash commands** — Lifecycle workflows, self-improvement loops, multi-agent dispatch, project health audit, post-PR monitoring, AST repo mapping, autoresearch-inspired self-audit, autoloop +- **17 context-activated skills** — 8 auto-trigger workflows (test-gen, doc-gen, changelog, onboard, research, deep-research, scrape, autoloop) + 6 coding principles (executing, clean-code, DRY, YAGNI, dont-reinvent, stuck) + 2 tools (gcli, creating-workflows) + 1 iteration memory (scratchpad) - **6 agents** — Scoped tool access, worktree isolation, model assignment - **10 hooks** — Safety (destructive command blocking, edit-time security patterns, PR gate), observability (audit trail, slop detection, post-validation, subagent verification, language-aware code review), optimization (RTK token compression) - **Graceful degradation** — tri:* commands work with 1-3 agents depending on installed CLIs @@ -12,7 +12,7 @@ - **Early-exit conditions** — Self-improvement loops stop when goal is met, not just at max iterations - **Token budget guidance** — Per-command budget recommendations with model downgrade patterns - **RTK token optimization** — Optional PreToolUse hook compresses Bash output via RTK (60-90% savings) -- **14 YAML workflows** — Portable workflow definitions (feature, bugfix, refactor, research, deep-research, self-*, tri-*) +- **15 YAML workflows** — Portable workflow definitions (feature, bugfix, refactor, research, deep-research, autoloop, self-*, tri-*) - **Separate marketplace** — Multi-plugin marketplace at `5uck1ess/marketplace` - **Companion ecosystem** — Evaluated official marketplace, documented holistic setup with 7 complementary plugins - **Hypothesis-driven perf** — Evidence gathering, ranked hypotheses, one-at-a-time testing replaces blind benchmark loops @@ -34,6 +34,6 @@ Items below were on the roadmap but determined to be unnecessary — either alre | Stop hook redesign | Still fires every turn, but exits early with `approve` when no files are changed — near-instant on clean trees, so the performance concern is moot. Revisit only if it causes measurable latency. | | Cost event hooks | Budget enforcement already exists in the Go engine via `overBudget()` + `addCost()` callbacks with hard limits | | Execution registry | Step tracking already handled by SQLite via `lib.DB` with status, cost, and timing per step | -| Preset library | The 14 YAML workflows and 16 skills already serve this purpose | +| Preset library | The 15 YAML workflows and 17 skills already serve this purpose | | Framework-specific review checklists | `lang-review.sh` covers language-level patterns; framework-specific rules are better added per-project via hookify | | Conditional hook firing | Hooks already self-filter internally (extension checks, changed-file checks); a generic condition system adds complexity for no current need | diff --git a/commands/autoloop.md b/commands/autoloop.md new file mode 100644 index 0000000..cc1f50a --- /dev/null +++ b/commands/autoloop.md @@ -0,0 +1,81 @@ +--- +description: Autonomous improvement loop inspired by karpathy/autoresearch — audit, fix, measure, keep or revert, repeat. +--- + +# Autoloop + +Autonomous codebase improvement loop. Each cycle: audit → pick hypothesis → fix → measure → keep or revert → repeat. + +Inspired by [karpathy/autoresearch](https://github.com/karpathy/autoresearch): one metric, keep or discard, loop. + +## Step 0: Harness Detection + +```bash +if command -v devkit >/dev/null 2>&1; then + echo "Go harness detected — delegating to devkit workflow autoloop." + devkit workflow autoloop "{input}" + exit 0 +fi +``` + +## Step 1: Gather Inputs + +If the input doesn't contain a metric command, use `AskUserQuestion` to collect: + +1. **Objective** — what to improve +2. **Metric command** — how to measure (auto-detect from stack if not provided) +3. **Direction** — higher-is-better or lower-is-better +4. **Iterations** — how many cycles (default 10) +5. **Scope** — file/package constraints (optional) + +## Step 2: Baseline + +Run the metric command. Record the starting number and direction. + +## Step 3: Audit + +Analyze the codebase for the single highest-impact change. Read the scratchpad to avoid repeating failed approaches. Output one hypothesis with target files. + +## Step 4: Fix + +Make the recommended change. Minimal, focused, no unrelated refactoring. + +## Step 5: Measure + +Run the EXACT same metric command. Record the new number. + +## Step 6: Compare + +Compare baseline vs measurement using the direction: +- higher-is-better: new > old → IMPROVED +- lower-is-better: new < old → IMPROVED +- Equal or failed → REGRESSED + +## Step 7: Keep or Revert + +- **IMPROVED** → stage only modified files + `git commit`, update scratchpad, update baseline to new number, loop back to Step 3 +- **REGRESSED** → `git checkout -- ` (no `git clean -fd` — protect untracked work), update scratchpad with failure reason, loop back to Step 3 + +## Step 8: Report + +After all iterations or budget exhausted: +- Starting vs final metric +- List of kept changes with impact +- List of reverted attempts with failure reason +- Net improvement +- Recommendation for next steps + +## Budget + +- **Token budget:** ~500k tokens. Each cycle costs ~30-50k tokens. +- **Iteration limit:** User-specified (default 10). +- Budget or iteration limit, whichever hits first, stops the loop. + +## Rules + +- Every change must be measured — no skipping the metric step +- Never keep a regression — always revert +- One hypothesis at a time — no bundling +- Use the scratchpad to prevent repeating failures +- The metric command must be identical in baseline and measure steps +- Update the baseline number after each kept change (so the next cycle compares against the new state, not the original) diff --git a/skills/autoloop/SKILL.md b/skills/autoloop/SKILL.md new file mode 100644 index 0000000..8c29672 --- /dev/null +++ b/skills/autoloop/SKILL.md @@ -0,0 +1,75 @@ +--- +name: autoloop +description: Autonomous improvement loop inspired by karpathy/autoresearch — use when asked to run an autoloop, auto loop, autonomous improvement, or run experiments overnight on the codebase. +--- + +# Autoloop + +Autonomous codebase improvement inspired by [karpathy/autoresearch](https://github.com/karpathy/autoresearch). Audit the codebase, pick the highest-impact change, fix it, measure the result, keep or revert, repeat. + +## Before Starting + +Use `AskUserQuestion` to gather these inputs. Do NOT proceed without answers. + +### 1. Objective + +Ask: "What do you want to improve? (e.g., test coverage, lint errors, performance, security)" + +### 2. Metric Command + +Ask: "What command measures success? (e.g., `go test -cover ./...`, `npx jest --coverage`, `ruff check . | wc -l`)" + +If the user doesn't have one, detect the stack and suggest: + +| Stack | Default metric | Direction | +|-------|---------------|-----------| +| Go | `go test -cover ./...` | higher-is-better (coverage %) | +| TypeScript | `npx jest --coverage` | higher-is-better (coverage %) | +| Python | `pytest --cov` | higher-is-better (coverage %) | +| Rust | `cargo test` | higher-is-better (pass count) | +| Go (lint) | `go vet ./... 2>&1 \| wc -l` | lower-is-better (error count) | +| Any (lint) | ` . 2>&1 \| wc -l` | lower-is-better (error count) | + +Confirm with the user: "I'll use `` with . Correct?" + +### 3. Direction + +If not obvious from the metric, ask: "Is higher or lower better for this metric?" + +### 4. Iterations + +Ask: "How many improvement cycles? (default: 10, max recommended: 50)" + +### 5. Scope (optional) + +Ask: "Any scope constraints? (e.g., only `src/engine/`, only `.py` files, or everything)" + +If the user says "everything" or skips, leave scope open. + +## Invoke the Workflow + +Assemble the input as a single string and invoke: + +``` +devkit workflow autoloop " | metric: | direction: -is-better | iterations: | scope: " +``` + +Or if the Go harness is not available, follow the workflow steps manually: + +1. **Baseline** — run the metric command, record the starting number +2. **Audit** — analyze codebase, pick single highest-impact hypothesis +3. **Fix** — make the recommended change (minimal, focused) +4. **Measure** — run the same metric command again +5. **Compare** — IMPROVED or REGRESSED based on direction +6. **Keep** (if improved) — git commit, update scratchpad, loop back to audit +7. **Revert** (if regressed) — git checkout -- , update scratchpad, loop back to audit +8. **Report** — final summary with kept changes, reverted attempts, net improvement + +## Rules + +- Never skip the measurement step — every change must be measured +- Never keep a change that regressed the metric — always revert +- One hypothesis at a time — don't bundle changes +- Use the scratchpad (.devkit/scratchpads/current.md) to avoid repeating failed approaches +- Stop when budget is exhausted, iterations are done, or metric stops improving +- Be honest in the report — include reverted attempts, not just successes diff --git a/workflows/autoloop.yml b/workflows/autoloop.yml new file mode 100644 index 0000000..ec47236 --- /dev/null +++ b/workflows/autoloop.yml @@ -0,0 +1,208 @@ +name: Autoloop +description: Autonomous improvement loop inspired by karpathy/autoresearch — audit, fix, measure, keep or revert, repeat + +budget: + limit: 500000 + downgrade: fast + +steps: + - id: baseline + model: fast + prompt: | + Run the user's metric command to establish a baseline measurement. + + User input: {{input}} + + Extract the metric command from the input and run it via Bash. + Report the raw output and extract a single numeric score. + + You MUST output these two lines exactly: + SCORE: + DIRECTION: higher-is-better (or: DIRECTION: lower-is-better) + + If no metric command was provided, detect the stack and use the appropriate default: + - Go: go test -cover ./... (higher-is-better: coverage %) + - TypeScript: npx jest --coverage (higher-is-better: coverage %) + - Python: pytest --cov --cov-report term-missing (higher-is-better: coverage %) + - Rust: cargo test (higher-is-better: pass count) + - Lint: count of errors (lower-is-better) + + If the command fails or produces no numeric output, say "AUTOLOOP_ERROR: metric command failed" + and do NOT proceed. + + - id: audit + model: smart + prompt: | + Analyze the codebase and pick the SINGLE highest-impact improvement to make. + + User objective: {{input}} + Current baseline: {{baseline}} + + Read .devkit/scratchpads/current.md if it exists — it contains what was already tried. + Do NOT repeat any approach listed as "reverted" in the scratchpad. + + Measure what's relevant to the objective: + - If coverage objective: find the lowest-coverage package/file + - If lint objective: find the most common lint error category + - If performance objective: identify the hottest path + - If security objective: find the most critical vulnerability + + Output exactly ONE hypothesis to test: + - HYPOTHESIS: + - TARGET: + - EXPECTED IMPACT: + + Do NOT make any changes yet. Only analyze and recommend. + + - id: fix + model: smart + prompt: | + Make the change recommended by the audit. + + Objective: {{input}} + Hypothesis: {{audit}} + + Rules: + - Only change what the hypothesis recommends + - Do NOT refactor unrelated code + - Do NOT change tests to make them pass (fix the code, not the tests) + - Keep changes minimal and focused + - Check .devkit/scratchpads/current.md to avoid repeating failed approaches + - Track which files you modify — list them at the end + + After making changes, output: + - What you changed and why + - MODIFIED FILES: + + - id: measure + model: fast + prompt: | + Run the EXACT same metric command from the baseline to measure the result. + + Baseline step output: {{baseline}} + + Extract the metric command that was used in the baseline step and run it again via Bash. + Report the raw output and extract the numeric score. + + You MUST output this line exactly: + SCORE: + + - id: compare + model: fast + prompt: | + Compare the baseline to the current measurement. + + Baseline output: + {{baseline}} + + Current measurement: + {{measure}} + + Extract the SCORE from the baseline and the SCORE from the measurement. + Extract the DIRECTION from the baseline. + + Rules — follow these exactly: + - If DIRECTION is higher-is-better: current SCORE > baseline SCORE means IMPROVED + - If DIRECTION is lower-is-better: current SCORE < baseline SCORE means IMPROVED + - If scores are equal: REGRESSED (no change counts as failure) + - If either SCORE is missing or non-numeric: REGRESSED + + Output exactly one word on the first line: IMPROVED or REGRESSED + Then explain: "baseline SCORE was X, current SCORE is Y, direction is Z" + branch: + - when: "IMPROVED" + goto: keep + - when: "REGRESSED" + goto: revert + + - id: keep + model: fast + prompt: | + The change IMPROVED the metric. Keep it. + + What was changed: {{fix}} + Measurement: {{measure}} + Comparison: {{compare}} + + 1. Stage and commit only the modified files listed in the fix step: + git add && git commit -m "autoloop: " + Do NOT use git add -A (it stages untracked files that may not be yours). + 2. Update .devkit/scratchpads/current.md — append: + --- + Iteration: kept + Hypothesis: {{audit}} + Result: {{compare}} + 3. The baseline SCORE for the next cycle is now the current SCORE from {{measure}}. + + Count the iterations in the scratchpad. Extract the iteration limit from the user input: {{input}} + If the iteration count >= the limit (default 10), say "AUTOLOOP_COMPLETE". + Otherwise say "CONTINUE" to trigger the next audit cycle. + branch: + - when: "AUTOLOOP_COMPLETE" + goto: report + - when: "CONTINUE" + goto: audit + + - id: revert + model: fast + prompt: | + The change REGRESSED or had no effect. Revert it. + + What was attempted: {{fix}} + Measurement: {{measure}} + Comparison: {{compare}} + + 1. Revert only the modified files from the fix step: + git checkout -- + If you cannot determine the files, use: git checkout -- . (but do NOT run git clean -fd) + 2. Update .devkit/scratchpads/current.md — append: + --- + Iteration: reverted + Hypothesis: {{audit}} + Result: {{compare}} + Reason: + + Count the iterations in the scratchpad. Extract the iteration limit from the user input: {{input}} + If the iteration count >= the limit (default 10), say "AUTOLOOP_COMPLETE". + Otherwise say "CONTINUE" to try the next hypothesis. + branch: + - when: "AUTOLOOP_COMPLETE" + goto: report + - when: "CONTINUE" + goto: audit + + - id: report + model: smart + prompt: | + The autoloop session is complete. Write a final report. + + Original objective: {{input}} + Starting baseline: {{baseline}} + Final measurement: {{measure}} + + Read .devkit/scratchpads/current.md for the full iteration history. + + Report: + ## Autoloop Report + + ### Objective + {what the user wanted} + + ### Results + - Starting metric: {baseline SCORE} + - Final metric: {final SCORE} + - Net change: {delta} + - Iterations: {count kept + count reverted} + - Kept: {count} + - Reverted: {count} + + ### Changes Made (kept) + {list each kept change with what it improved} + + ### Failed Attempts (reverted) + {list each reverted attempt with why it failed} + + ### Recommendation + {what to do next — run another cycle, investigate a specific area, or stop} + + Clean up: delete .devkit/scratchpads/current.md