diff --git a/README.md b/README.md index df0b73c..d3804fa 100644 --- a/README.md +++ b/README.md @@ -310,7 +310,7 @@ devkit/ ├── skills/ # 18 context-activated skills ├── agents/ # 6 agents (reviewer, researcher, improver, ...) ├── hooks/ # 10 hooks (safety, security, quality gates) -├── workflows/ # 15 YAML workflow definitions +├── workflows/ # 16 YAML workflow definitions ├── resources/rules/ # Language-specific coding rules ├── presets/ # Reserved for future use ├── src/ # Go CLI harness diff --git a/ROADMAP.md b/ROADMAP.md index c52a165..83a42ea 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -12,7 +12,7 @@ - **Early-exit conditions** — Self-improvement loops stop when goal is met, not just at max iterations - **Token budget guidance** — Per-command budget recommendations with model downgrade patterns - **RTK token optimization** — Optional PreToolUse hook compresses Bash output via RTK (60-90% savings) -- **15 YAML workflows** — Portable workflow definitions (feature, bugfix, refactor, research, deep-research, autoloop, self-*, tri-*) +- **16 YAML workflows** — Portable workflow definitions (feature, bugfix, refactor, research, deep-research, autoloop, self-*, tri-*) - **Separate marketplace** — Multi-plugin marketplace at `5uck1ess/marketplace` - **Companion ecosystem** — Evaluated official marketplace, documented holistic setup with 7 complementary plugins - **Hypothesis-driven perf** — Evidence gathering, ranked hypotheses, one-at-a-time testing replaces blind benchmark loops @@ -34,6 +34,6 @@ Items below were on the roadmap but determined to be unnecessary — either alre | Stop hook redesign | Still fires every turn, but exits early with `approve` when no files are changed — near-instant on clean trees, so the performance concern is moot. Revisit only if it causes measurable latency. | | Cost event hooks | Budget enforcement already exists in the Go engine via `overBudget()` + `addCost()` callbacks with hard limits | | Execution registry | Step tracking already handled by SQLite via `lib.DB` with status, cost, and timing per step | -| Preset library | The 15 YAML workflows and 18 skills already serve this purpose | +| Preset library | The 16 YAML workflows and 18 skills already serve this purpose | | Framework-specific review checklists | `lang-review.sh` covers language-level patterns; framework-specific rules are better added per-project via hookify | | Conditional hook firing | Hooks already self-filter internally (extension checks, changed-file checks); a generic condition system adds complexity for no current need | diff --git a/commands/self-improve.md b/commands/self-improve.md index c61fbe1..307c0b0 100644 --- a/commands/self-improve.md +++ b/commands/self-improve.md @@ -1,104 +1,27 @@ --- -description: Self-recursive improvement loop — automated refactoring with test gate. Uses native improver agent in worktree isolation. Propose → measure → keep/discard → repeat. +description: Metric-gated improvement loop — run command, fix issues, repeat until passing. --- -# Self-Improve Loop +# Self-Improve -Autonomous, iterative improvement of a target file or directory. Each iteration: propose change → run metric → keep if better → discard if worse → repeat. +Deterministic improvement loop: run metric → fix issues → gate check → repeat until exit code 0. -## Parameters +## Invoke -1. **Target** — file or directory to improve (required) -2. **Metric** — shell command that exits 0 on success (required) -3. **Objective** — what to optimize for (required) -4. **Iterations** — max cycles (default: 10) -5. **Budget** — max USD (default: $2) - -## Budget & Early Exit - -- **Token budget:** ~300k tokens. If approaching limit, reduce remaining iteration count. -- **Early exit:** Stop the loop immediately if the metric output contains the objective target (e.g., "0 errors", "100%", or a user-specified success string). -- **Stuck detection:** If 3 consecutive iterations fail (revert), stop the loop and report. Don't keep trying the same failing approach. See the `stuck` skill. - -## Step 1: Establish Baseline - -```bash -git checkout -b self-improve/$(date +%Y%m%d-%H%M%S) -BASELINE=$({metric_command} 2>&1) -echo "$BASELINE" > /tmp/self-improve-baseline.txt -echo "$BASELINE" > /tmp/self-improve-best.txt -``` - -## Step 2: Run the Loop - -For each iteration, spawn the `improver` agent as a native background task with worktree isolation: - -``` -Task: Improve {target} toward objective: {objective} -Agent: improver -Context: - - Iteration history: (cat /tmp/self-improve-log.txt) - - Current best metric: (cat /tmp/self-improve-best.txt) - - Target file(s): {target} -``` - -The improver agent: -1. Reads the target and iteration history -2. Proposes ONE focused change -3. Applies it - -Then the orchestrator: -```bash -RESULT=$({metric_command} 2>&1) -EXIT_CODE=$? - -if [ $EXIT_CODE -eq 0 ]; then - echo "ITERATION $i: SUCCESS" >> /tmp/self-improve-log.txt - echo "$RESULT" > /tmp/self-improve-best.txt - git add -A && git commit -m "self-improve: iteration $i — passed" -else - echo "ITERATION $i: FAILED — reverting" >> /tmp/self-improve-log.txt - git checkout -- . -fi ``` - -## Step 3: Report - +devkit workflow run self-improve "{metric_command}" ``` -## Self-Improve Report - -**Target:** {target} -**Objective:** {objective} -**Iterations:** {completed} / {total} - -### Baseline → Final -{baseline} → {final} -### Log -| # | Result | Change | -|---|--------|--------| -| 1 | PASS | Refactored validation | -| 2 | FAIL | Caching — broke tests | -| 3 | PASS | Extracted helper | +If `devkit workflow` is not available, follow this manually: -### Next Steps -- Review: `git diff main...HEAD` -- Merge: `git checkout main && git merge self-improve/{branch}` -- Discard: `git checkout main && git branch -D self-improve/{branch}` -``` - -## Presets - -``` -/self:improve --target src/ --metric "npm test" --objective "fix failing tests" -/self:improve --target train.py --metric "python train.py" --objective "minimize val_bpb" --iterations 50 -``` +1. **Baseline** — Run the metric command and capture output +2. **Improve loop** — Analyze failures, make targeted fixes, re-run metric; stop when passing (max 10 iterations) +3. **Verify** — Run metric one final time +4. **Summary** — Report what was fixed, iteration count, final metric status ## Rules -- Uses native `improver` agent with worktree isolation (token-efficient) -- Always branches first — never modifies main -- One change per iteration -- Discard on failure — `git checkout -- .` -- Never modify the metric command -- Human merges at end — never auto-merges +- Only change what's needed to improve the metric +- Don't refactor unrelated code +- One group of related fixes per iteration +- The metric command must exit non-zero when improvement is still needed diff --git a/commands/self-lint.md b/commands/self-lint.md index 8265aca..916e64b 100644 --- a/commands/self-lint.md +++ b/commands/self-lint.md @@ -1,96 +1,27 @@ --- -description: Self-improvement loop targeting lint and type errors. Iteratively fixes issues until zero remain or iterations exhausted. +description: Run linter, fix violations deterministically, repeat until clean. --- -# Self-Improve: Lint / Type Errors +# Self-Lint -Automated loop that runs your linter or type checker, fixes issues one at a time, and verifies no regressions. +Deterministic lint-fix loop: run linter → fix violations → gate check → repeat until exit code 0. -## Parameters +## Invoke -1. **Lint command** — command that reports lint/type errors (required) -2. **Target** — file or directory scope (default: entire project) -3. **Iterations** — max cycles (default: 20) -4. **Budget** — max USD (default: $2) - -## Budget & Early Exit - -- **Token budget:** ~200k tokens. Lint fixes are usually cheap. -- **Early exit:** Stop immediately when error count reaches 0. -- **Stuck detection:** If 3 consecutive iterations show no improvement (error count doesn't decrease), stop and report. See the `stuck` skill. - -## Step 1: Establish Baseline - -```bash -git checkout -b self-lint/$(date +%Y%m%d-%H%M%S) -BASELINE=$({lint_command} 2>&1) -ERRORS=$(echo "$BASELINE" | grep -cE '(error|warning)') -echo "Baseline: $ERRORS issues" > /tmp/self-lint-log.txt -echo "$BASELINE" > /tmp/self-lint-baseline.txt -``` - -## Step 2: Run the Loop - -For each iteration, spawn the `improver` agent: - -``` -Task: Fix lint/type errors in {target}. -Agent: improver -Context: - - Iteration: {i} of {max} - - Remaining errors: {error_count} - - Current lint output: {lint_output} - - Iteration history: (cat /tmp/self-lint-log.txt) ``` - -The improver agent: -1. Reads the lint output -2. Picks the highest-priority error -3. Fixes ONE issue (or a group of related issues in one file) - -Then the orchestrator: -```bash -RESULT=$({lint_command} 2>&1) -NEW_ERRORS=$(echo "$RESULT" | grep -cE '(error|warning)') - -if [ $NEW_ERRORS -lt $PREV_ERRORS ]; then - echo "ITERATION $i: PASS — $PREV_ERRORS → $NEW_ERRORS issues" >> /tmp/self-lint-log.txt - git add -A && git commit -m "self-lint: iteration $i — $NEW_ERRORS remaining" - PREV_ERRORS=$NEW_ERRORS -else - echo "ITERATION $i: FAIL — no improvement or regression, reverting" >> /tmp/self-lint-log.txt - git checkout -- . -fi +devkit workflow run self-lint "{lint_command}" ``` -Stop early if zero errors remain. +If `devkit workflow` is not available, follow this manually: -## Step 3: Report - -``` -## Self-Lint Report - -**Lint command:** {lint_command} -**Errors:** {baseline_count} → {final_count} -**Iterations:** {completed} / {total} - -### Log -| # | Result | Errors | Fix | -|---|--------|--------|-----| -| 1 | PASS | 12→10 | Fixed unused imports in api.ts | -| 2 | PASS | 10→8 | Added missing return types | -| 3 | FAIL | 8→8 | Reverted — no improvement | - -### Next Steps -- Review: `git diff main...HEAD` -- Merge: `git checkout main && git merge self-lint/{branch}` -``` +1. **Baseline** — Run the lint command and capture output +2. **Fix loop** — Fix one group of related lint/type issues at a time; re-run linter after each fix; stop when clean (max 20 iterations) +3. **Verify** — Run linter one final time +4. **Summary** — Report what was fixed and what remains ## Rules -- Uses `improver` agent with worktree isolation -- Always branches first -- Error count must strictly decrease to keep a change -- Discard on regression or no improvement -- Stop early at zero errors +- Prioritize errors over warnings +- Fix one group of related issues at a time +- Don't change code behavior — only fix lint issues - Never disable lint rules — fix the underlying issue diff --git a/commands/self-migrate.md b/commands/self-migrate.md index 74d2805..a63d1c9 100644 --- a/commands/self-migrate.md +++ b/commands/self-migrate.md @@ -1,113 +1,27 @@ --- -description: Self-improvement loop for incremental codebase migrations. Iteratively migrates code with tests as the safety gate. +description: Incremental migration loop — migrate code one piece at a time with tests as safety gate. --- -# Self-Improve: Migration +# Self-Migrate -Automated, incremental migration loop. Each iteration migrates a small piece, runs tests, keeps if green, reverts if red. +Deterministic migration loop: run tests → migrate one piece → gate check → repeat until gate exits 0. -## Parameters - -1. **Target** — file or directory to migrate (required) -2. **Test command** — command that validates correctness (required) -3. **Migration** — what to migrate to (required, e.g., "TypeScript", "React hooks", "Python 3.12", "ES modules") -4. **Iterations** — max cycles (default: 20) -5. **Budget** — max USD (default: $3) - -## Budget & Early Exit - -- **Token budget:** ~400k tokens. Migrations can touch many files. -- **Early exit:** Stop when all target files are migrated — don't run remaining iterations. -- **Stuck detection:** If 3 consecutive iterations fail (tests break after migration), stop and report. The remaining files may need manual intervention. See the `stuck` skill. - -## Step 1: Establish Baseline - -```bash -git checkout -b self-migrate/$(date +%Y%m%d-%H%M%S) -{test_command} 2>&1 > /tmp/self-migrate-baseline.txt -``` - -All tests must pass before starting. Abort if baseline fails. - -## Step 2: Run the Loop - -For each iteration, spawn the `improver` agent: - -``` -Task: Migrate {target} toward {migration}. -Agent: improver -Context: - - Iteration: {i} of {max} - - Migration objective: {migration} - - Iteration history: (cat /tmp/self-migrate-log.txt) - - Remaining unmigrated files: {file_list} - - Target: {target} -``` - -The improver agent: -1. Identifies the next file or component to migrate -2. Performs ONE migration step (one file or one closely-related group) -3. Updates imports/references as needed - -Then the orchestrator: -```bash -RESULT=$({test_command} 2>&1) -EXIT_CODE=$? - -if [ $EXIT_CODE -eq 0 ]; then - MIGRATED=$(git diff --name-only) - echo "ITERATION $i: PASS — migrated $MIGRATED" >> /tmp/self-migrate-log.txt - git add -A && git commit -m "self-migrate: iteration $i — ${migration}" -else - echo "ITERATION $i: FAIL — reverting" >> /tmp/self-migrate-log.txt - git checkout -- . -fi -``` - -Stop early if all target files are migrated. - -## Step 3: Report +## Invoke ``` -## Self-Migrate Report - -**Target:** {target} -**Migration:** {migration} -**Iterations:** {completed} / {total} - -### Migrated Files -- src/utils.js → src/utils.ts ✓ -- src/api.js → src/api.ts ✓ -- src/parser.js → (failed, reverted) - -### Log -| # | Result | File(s) | -|---|--------|---------| -| 1 | PASS | utils.js → utils.ts | -| 2 | PASS | api.js → api.ts | -| 3 | FAIL | parser.js — type errors | - -### Next Steps -- Review: `git diff main...HEAD` -- Merge: `git checkout main && git merge self-migrate/{branch}` +devkit workflow run self-migrate "{test_command}" ``` -## Presets +If `devkit workflow` is not available, follow this manually: -``` -/self:migrate --target src/ --test "npm test" --migration "TypeScript strict mode" -/self:migrate --target app/ --test "pytest" --migration "Python 3.12 syntax (match statements, tomllib)" -/self:migrate --target components/ --test "npm test" --migration "React class components to hooks" -/self:migrate --target lib/ --test "go test ./..." --migration "Go 1.22 range-over-func" -``` +1. **Baseline** — Run tests to confirm green starting point +2. **Migrate loop** — Migrate one file or closely-related group per iteration; update imports/references; re-run gate command after each step (max 20 iterations). To detect migration completeness, use a gate command that exits non-zero until all files are converted (e.g., `npm test && ! grep -r 'require(' src/`). +3. **Verify** — Run tests one final time +4. **Summary** — Report which files were migrated and what remains ## Rules -- Uses `improver` agent with worktree isolation -- Always branches first - One file or closely-related group per iteration - Tests must pass to keep — no exceptions -- Discard on any test failure - Preserve existing behavior — migration, not refactoring - Update imports and references in the same iteration -- Stop when all target files are migrated diff --git a/commands/self-perf.md b/commands/self-perf.md index 65e2475..4c0bda1 100644 --- a/commands/self-perf.md +++ b/commands/self-perf.md @@ -1,250 +1,26 @@ --- -description: Hypothesis-driven performance investigation — analyze, hypothesize, test one theory at a time, measure against baseline. +description: Profile performance, optimize hot paths deterministically, verify improvement. --- -# Self-Improve: Performance +# Self-Perf -Structured performance optimization that forms hypotheses from evidence before changing code. Tests one theory at a time against a stable baseline. +Deterministic performance optimization: benchmark → optimize → gate check → repeat until benchmark exits 0. -## Parameters - -1. **Benchmark command** — command that outputs a measurable metric (required) -2. **Target** — file or directory scope (default: entire project) -3. **Metric name** — what the benchmark measures, e.g. "request latency ms" (required) -4. **Goal** — target metric value, e.g. "< 200" (optional) -5. **Iterations** — max investigation cycles (default: 10) -6. **Budget** — max USD (default: $5) - -## Budget & Early Exit - -- **Token budget:** ~400k tokens. Investigation is more expensive than blind optimization. -- **Early exit:** Stop when goal is met or no viable hypotheses remain. -- **Stuck detection:** If 3 consecutive hypotheses fail, escalate to user. See the `stuck` skill. - -## Step 1: Establish Baseline - -Run the benchmark at least 3 times for stability: - -```bash -git checkout -b perf/$(date +%Y%m%d-%H%M%S) - -echo "=== Baseline Run 1 ===" && {benchmark_command} -echo "=== Baseline Run 2 ===" && {benchmark_command} -echo "=== Baseline Run 3 ===" && {benchmark_command} -``` - -Record the median result as the baseline. Reject runs with >20% variance — the benchmark isn't stable enough for meaningful optimization. - -Save baseline: -```bash -mkdir -p .devkit/perf -cat > .devkit/perf/baseline.json << 'BASELINE' -{ - "metric": "{metric_name}", - "value": {median_value}, - "unit": "{unit}", - "runs": [{run1}, {run2}, {run3}], - "variance_pct": {variance}, - "commit": "{commit_hash}", - "timestamp": "{iso8601}" -} -BASELINE -``` - -## Step 2: Gather Evidence - -Before forming hypotheses, collect data from multiple sources: - -### 2a. Git History Analysis - -```bash -# Find performance-related commits -git log --all --oneline --grep="perf" --grep="slow" --grep="optimize" --grep="cache" --grep="latency" --grep="memory" | head -20 - -# Find recent changes to hot paths -git log --oneline -20 -- {target} - -# Find large commits that may have introduced regressions -git log --oneline --diff-filter=M --stat | head -30 -``` - -### 2b. Code Path Analysis - -Spawn the `researcher` agent: +## Invoke ``` -Task: Analyze the critical code paths in {target} for performance. -Agent: researcher -Focus on: - - Hot loops and recursive calls - - I/O operations (file, network, database) - - Memory allocation patterns (large objects, frequent allocations) - - Synchronous operations that could be async - - Missing caching opportunities - - N+1 query patterns - - Unnecessary serialization/deserialization - - Redundant computation -Report: list of suspicious code paths with file:line references +devkit workflow run self-perf "{benchmark_command}" ``` -### 2c. Profiling Data (if available) - -```bash -# Check for existing profiling tools -command -v perf >/dev/null 2>&1 && echo "perf available" -command -v hyperfine >/dev/null 2>&1 && echo "hyperfine available" -[ -f flamegraph.svg ] && echo "flamegraph found" -``` - -If profiling tools are available, run a quick profile to identify actual hotspots. - -## Step 3: Form Hypotheses - -Based on the evidence, form up to 5 ranked hypotheses: - -``` -## Hypotheses - -| # | Hypothesis | Evidence | Confidence | Expected Impact | -|---|-----------|----------|------------|-----------------| -| 1 | N+1 queries in getUserOrders | researcher found loop with individual DB calls at orders.ts:45 | High | 3-5x faster | -| 2 | Missing cache for config parsing | parseConfig called 12x per request, git shows it was recently changed | Medium | 20-30% faster | -| 3 | Synchronous file reads in middleware | blocking I/O in request path at middleware.ts:23 | Medium | 10-20% faster | -| 4 | Large JSON serialization in logging | JSON.stringify on full request objects at logger.ts:67 | Low | 5-10% faster | -| 5 | Regex compilation on every call | new RegExp() inside loop at validator.ts:12 | Low | 5% faster | -``` - -Rules for hypotheses: -- Each must cite specific evidence (file:line, git commit, profiler output) -- Each must predict the expected impact (not just "faster") -- Confidence is based on evidence strength, not gut feel -- Order by confidence * expected impact (highest first) - -## Step 4: Test Hypotheses (One at a Time) - -For each hypothesis, starting from highest-ranked: - -### 4a. Implement the Fix - -Spawn the `improver` agent: +If `devkit workflow` is not available, follow this manually: -``` -Task: Optimize {target} based on this hypothesis: - Hypothesis: {hypothesis_description} - Evidence: {evidence} - File: {file_path}:{line} -Agent: improver -Constraints: - - Change ONLY what this hypothesis addresses - - Do not refactor unrelated code - - Preserve all existing behavior - - Keep the change as small as possible -``` - -### 4b. Verify Correctness - -```bash -# Run tests first — optimization must not break anything -{test_command} || echo "TESTS FAILED — reverting" -``` - -If tests fail, revert and move to next hypothesis. - -### 4c. Measure Impact - -Run benchmark 3 times again: - -```bash -echo "=== Post-fix Run 1 ===" && {benchmark_command} -echo "=== Post-fix Run 2 ===" && {benchmark_command} -echo "=== Post-fix Run 3 ===" && {benchmark_command} -``` - -### 4d. Evaluate - -Compare median against baseline: - -```bash -IMPROVEMENT=$(( (BASELINE - NEW_MEDIAN) * 100 / BASELINE )) -``` - -Decision: -- **Improvement matches or exceeds prediction** → Keep. Commit. - ```bash - git add -A && git commit -m "perf: {hypothesis summary} ({improvement}% improvement)" - ``` -- **Improvement exists but below prediction** → Keep if >5% improvement, otherwise revert. -- **No improvement or regression** → Revert. Log why hypothesis was wrong. - ```bash - git checkout -- . - ``` - -Log the result: -```bash -echo "HYPOTHESIS {n}: {PASS|FAIL} — predicted {predicted}%, actual {actual}%" >> .devkit/perf/investigation.log -echo " Evidence: {evidence}" >> .devkit/perf/investigation.log -echo " Lesson: {why it worked or didn't}" >> .devkit/perf/investigation.log -``` - -### 4e. Update Baseline - -If the fix was kept, the new median becomes the baseline for subsequent hypotheses. - -## Step 5: Report - -``` -## Performance Investigation Report - -**Target:** {target} -**Metric:** {metric_name} -**Baseline:** {baseline_value} {unit} -**Final:** {final_value} {unit} -**Total improvement:** {total_improvement}% -**Goal:** {goal} — {met|not met} - -### Hypotheses Tested -| # | Hypothesis | Predicted | Actual | Result | -|---|-----------|-----------|--------|--------| -| 1 | N+1 queries in getUserOrders | 3-5x | 3.2x | PASS — kept | -| 2 | Missing cache for config | 20-30% | 22% | PASS — kept | -| 3 | Sync file reads | 10-20% | 2% | FAIL — below threshold, reverted | -| 4 | JSON serialization | 5-10% | — | SKIPPED — goal already met | - -### Investigation Log -{contents of .devkit/perf/investigation.log} - -### Commits -| Commit | Hypothesis | Improvement | -|--------|-----------|-------------| -| abc1234 | Batch N+1 queries | 3.2x | -| def5678 | Add config cache | 22% | - -### Remaining Hypotheses (untested) -- Large JSON serialization in logging — estimated 5-10% -- Regex compilation on every call — estimated 5% - -### Next Steps -- Review: `git diff main...HEAD` -- Merge: `git checkout main && git merge perf/{branch}` -``` - -## Presets - -``` -/self:perf --target src/api/ --benchmark "wrk -t4 -c100 -d5s http://localhost:3000" --metric "p99 latency ms" --goal "< 200" -/self:perf --target lib/parser.go --benchmark "go test -bench=. -benchtime=3s" --metric "ns/op" --goal "< 1000" -``` +1. **Baseline** — Run the benchmark command and capture metrics +2. **Optimize loop** — Identify bottleneck, make one targeted optimization, re-benchmark; stop when target met (max 5 iterations) +3. **Verify** — Run benchmark one final time +4. **Summary** — Report improvement (absolute and percentage) ## Rules -- Always branch first -- Never skip the evidence-gathering step — blind optimization is guessing -- Test ONE hypothesis at a time — never bundle changes -- Run benchmarks 3x minimum — single runs are unreliable -- Reject benchmarks with >20% variance -- Tests must pass before measuring — broken code isn't faster code -- Log every hypothesis outcome with the lesson learned -- Revert on regression or no improvement — don't keep dead changes -- Stop when goal is met — don't over-optimize -- The improver agent runs in worktree isolation -- The researcher agent runs in worktree isolation for code analysis +- The benchmark command must exit non-zero when the target is not met (wrap bare benchmarks in a threshold-checking script) +- One optimization at a time — no speculative refactoring +- Only change what impacts the metric diff --git a/commands/self-test.md b/commands/self-test.md index 3fa6479..da60638 100644 --- a/commands/self-test.md +++ b/commands/self-test.md @@ -1,110 +1,27 @@ --- -description: Self-improvement loop targeting test coverage. Iteratively generates and improves tests until a coverage target is met or iterations exhausted. +description: Run tests, fix failures deterministically, repeat until all pass. --- -# Self-Improve: Test Coverage +# Self-Test -Automated loop that generates tests, runs them, measures coverage, and iterates until the target is hit. +Deterministic test-fix loop: run tests → fix failures → gate check → repeat until exit code 0. -## Parameters +## Invoke -1. **Target** — file or directory to generate tests for (required) -2. **Test command** — command that runs tests and reports coverage (required) -3. **Coverage target** — percentage to aim for (default: 80) -4. **Iterations** — max cycles (default: 10) -5. **Budget** — max USD (default: $2) - -## Budget & Early Exit - -- **Token budget:** ~300k tokens. If approaching limit, reduce remaining iteration count. -- **Early exit:** Stop immediately when coverage target is met — don't run remaining iterations. -- **Stuck detection:** If 3 consecutive iterations fail (tests break), stop and report. See the `stuck` skill. - -## Step 1: Detect Test Framework - -```bash -# Auto-detect from package.json, pyproject.toml, go.mod, Cargo.toml, etc. -# Identify existing test files, patterns, and coverage tooling ``` - -If no test command provided, infer from project config. Ask user to confirm. - -## Step 2: Establish Baseline - -```bash -git checkout -b self-test/$(date +%Y%m%d-%H%M%S) -BASELINE=$({test_command} 2>&1) -echo "$BASELINE" > /tmp/self-test-baseline.txt +devkit workflow run self-test "{test_command}" ``` -Extract current coverage percentage from output. If no tests exist yet, baseline is 0%. - -## Step 3: Run the Loop - -For each iteration, spawn the `test-writer` agent as a background task: - -``` -Task: Generate or improve tests for {target} to increase coverage. -Agent: test-writer -Context: - - Iteration: {i} of {max} - - Current coverage: {current}% - - Target coverage: {coverage_target}% - - Iteration history: (cat /tmp/self-test-log.txt) - - Target file(s): {target} - - Existing tests: {test_files} -``` +If `devkit workflow` is not available, follow this manually: -The test-writer agent: -1. Reads the target source and existing tests -2. Identifies uncovered code paths -3. Writes or improves ONE test file - -Then the orchestrator: -```bash -RESULT=$({test_command} 2>&1) -EXIT_CODE=$? - -if [ $EXIT_CODE -eq 0 ]; then - COVERAGE=$(echo "$RESULT" | grep -oE '[0-9]+\.?[0-9]*%' | tail -1) - echo "ITERATION $i: PASS — coverage $COVERAGE" >> /tmp/self-test-log.txt - git add -A && git commit -m "self-test: iteration $i — coverage $COVERAGE" -else - echo "ITERATION $i: FAIL — tests broke, reverting" >> /tmp/self-test-log.txt - git checkout -- . -fi -``` - -Stop early if coverage target is met. - -## Step 4: Report - -``` -## Self-Test Report - -**Target:** {target} -**Coverage:** {baseline}% → {final}% (target: {coverage_target}%) -**Iterations:** {completed} / {total} - -### Log -| # | Result | Coverage | Change | -|---|--------|----------|--------| -| 1 | PASS | 45% | Added unit tests for parser | -| 2 | FAIL | — | Tests broke — reverted | -| 3 | PASS | 62% | Added edge case tests | - -### Next Steps -- Review: `git diff main...HEAD` -- Merge: `git checkout main && git merge self-test/{branch}` -- Discard: `git checkout main && git branch -D self-test/{branch}` -``` +1. **Baseline** — Run the test command and capture output +2. **Fix loop** — Fix one group of related failures at a time; re-run tests after each fix; stop when all pass (max 8 iterations) +3. **Verify** — Run tests one final time +4. **Summary** — Report what was fixed and iteration count ## Rules -- Uses `test-writer` agent with worktree isolation -- Always branches first — never modifies main -- One test file per iteration -- Discard on failure — `git checkout -- .` -- Stop early if target coverage reached -- Match existing test conventions (file naming, framework, patterns) -- Never modify source code — only test files +- One group of related fixes per iteration +- The bug might be in the test or the code under test +- Don't refactor unrelated code +- Match existing test conventions diff --git a/workflows/self-improve.yml b/workflows/self-improve.yml index c159813..79e8643 100644 --- a/workflows/self-improve.yml +++ b/workflows/self-improve.yml @@ -1,44 +1,39 @@ name: Self-Improve -description: Metric-gated improvement loop — run a metric, improve code, repeat until passing +description: Metric-gated improvement loop — run command, fix issues, repeat until passing steps: - id: baseline - model: fast - prompt: | - Run the metric command to establish a baseline: - - {{input}} - - Report what passed, what failed, and the current score. + command: "{{input}} 2>&1 || true" - id: improve model: smart prompt: | - Current state from last run: + Current metric output: {{baseline}} Analyze the failures and make targeted fixes. Only change what's needed to improve the metric. Don't refactor unrelated code. + ONE group of related fixes at a time. - After making changes, run the metric again and report results. - If everything passes, say "ALL_PASSING". + After making changes, say DONE. loop: max: 10 - until: ALL_PASSING + until: DONE + gate: "{{input}}" + + - id: verify + command: "{{input}} 2>&1 || true" - - id: report + - id: summary model: fast prompt: | - Improvement session complete. Summarize: + Improvement session complete. Starting state: {{baseline}} Final state: - {{improve}} + {{verify}} - Report: - - What was fixed - - How many iterations it took - - Current metric status + Report what was fixed, how many iterations it took, and current metric status. diff --git a/workflows/self-lint.yml b/workflows/self-lint.yml index 428649e..557fd23 100644 --- a/workflows/self-lint.yml +++ b/workflows/self-lint.yml @@ -20,7 +20,7 @@ steps: After fixing, say DONE. loop: max: 20 - until: "exit code: 0" + until: DONE gate: "{{input}}" - id: verify diff --git a/workflows/self-migrate.yml b/workflows/self-migrate.yml new file mode 100644 index 0000000..d584090 --- /dev/null +++ b/workflows/self-migrate.yml @@ -0,0 +1,40 @@ +name: Self-Migrate +description: Incremental migration loop — migrate code one piece at a time with tests as safety gate + +steps: + - id: baseline + command: "{{input}} 2>&1 || true" + + - id: migrate + model: smart + prompt: | + Test baseline: + + {{baseline}} + + Identify the next file or component to migrate. + Perform ONE migration step (one file or closely-related group). + Update imports and references in the same change. + Preserve existing behavior — migration, not refactoring. + + After making changes, say DONE. + loop: + max: 20 + until: DONE + gate: "{{input}}" + + - id: verify + command: "{{input}} 2>&1 || true" + + - id: summary + model: fast + prompt: | + Migration session complete. + + Starting state: + {{baseline}} + + Final state: + {{verify}} + + Report which files were migrated and what remains. diff --git a/workflows/self-perf.yml b/workflows/self-perf.yml index db71750..df7096b 100644 --- a/workflows/self-perf.yml +++ b/workflows/self-perf.yml @@ -1,14 +1,9 @@ name: Self-Perf -description: Profile performance, optimize hot paths, verify improvement +description: Profile performance, optimize hot paths deterministically, verify improvement steps: - id: baseline - model: fast - prompt: | - Profile the target and establish a performance baseline. - {{input}} - - Report timing, memory usage, or throughput — whatever the metric is. + command: "{{input}} 2>&1 || true" - id: optimize model: smart @@ -19,11 +14,16 @@ steps: Identify the bottleneck and make a targeted optimization. Only change what impacts the metric — no speculative refactoring. + ONE optimization at a time. - Re-run the benchmark. If the target metric is met, say "ALL_PASSING". + After making changes, say DONE. loop: max: 5 - until: ALL_PASSING + until: DONE + gate: "{{input}}" + + - id: verify + command: "{{input}} 2>&1 || true" - id: summary model: fast @@ -34,6 +34,6 @@ steps: {{baseline}} After: - {{optimize}} + {{verify}} Report the improvement (absolute and percentage). diff --git a/workflows/self-test.yml b/workflows/self-test.yml index 70bc51c..ecf94ad 100644 --- a/workflows/self-test.yml +++ b/workflows/self-test.yml @@ -1,29 +1,29 @@ name: Self-Test -description: Run tests, fix failures, repeat until all pass +description: Run tests, fix failures deterministically, repeat until all pass steps: - id: baseline - model: fast - prompt: | - Run the project's test suite and report results. - {{input}} - - Show which tests pass, which fail, and the error messages. + command: "{{input}} 2>&1 || true" - id: fix model: smart prompt: | - Test results: + Test output: {{baseline}} Fix the failing tests. The bug might be in the test or the code under test. Make targeted changes — don't refactor unrelated code. + ONE group of related fixes at a time. - Run tests again after fixing. If all pass, say "ALL_PASSING". + After fixing, say DONE. loop: max: 8 - until: ALL_PASSING + until: DONE + gate: "{{input}}" + + - id: verify + command: "{{input}} 2>&1 || true" - id: summary model: fast @@ -34,6 +34,6 @@ steps: {{baseline}} Final state: - {{fix}} + {{verify}} Summarize what was fixed and how many iterations it took.