Skip to content

PR 4: Convert remaining 12 commands to thin wrappers #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
5uck1ess merged 3 commits into from
Apr 9, 2026
Merged

PR 4: Convert remaining 12 commands to thin wrappers #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4fce8d0
Convert remaining 12 commands to thin wrappers
5uck1ess Apr 9, 2026
b690269
Address review findings: create missing workflows, fix references
5uck1ess Apr 9, 2026
7751346
Fix refactor loop/budget mismatch, document tri-* architecture diverg…
5uck1ess Apr 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -310,7 +310,7 @@ devkit/
├── skills/ # 18 context-activated skills
├── agents/ # 6 agents (reviewer, researcher, improver, ...)
├── hooks/ # 10 hooks (safety, security, quality gates)
├── workflows/ # 16 YAML workflow definitions
├── workflows/ # 18 YAML workflow definitions
├── resources/rules/ # Language-specific coding rules
├── presets/ # Reserved for future use
├── src/ # Go CLI harness
Expand Down
4 changes: 2 additions & 2 deletions ROADMAP.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
- **Early-exit conditions** — Self-improvement loops stop when goal is met, not just at max iterations
- **Token budget guidance** — Per-command budget recommendations with model downgrade patterns
- **RTK token optimization** — Optional PreToolUse hook compresses Bash output via RTK (60-90% savings)
- **16 YAML workflows** — Portable workflow definitions (feature, bugfix, refactor, research, deep-research, autoloop, self-*, tri-*)
- **18 YAML workflows** — Portable workflow definitions (feature, bugfix, refactor, research, deep-research, autoloop, self-*, tri-*)
- **Separate marketplace** — Multi-plugin marketplace at `5uck1ess/marketplace`
- **Companion ecosystem** — Evaluated official marketplace, documented holistic setup with 7 complementary plugins
- **Hypothesis-driven perf** — Evidence gathering, ranked hypotheses, one-at-a-time testing replaces blind benchmark loops
Expand All @@ -34,6 +34,6 @@ Items below were on the roadmap but determined to be unnecessary — either alre
| Stop hook redesign | Still fires every turn, but exits early with `approve` when no files are changed — near-instant on clean trees, so the performance concern is moot. Revisit only if it causes measurable latency. |
| Cost event hooks | Budget enforcement already exists in the Go engine via `overBudget()` + `addCost()` callbacks with hard limits |
| Execution registry | Step tracking already handled by SQLite via `lib.DB` with status, cost, and timing per step |
| Preset library | The 16 YAML workflows and 18 skills already serve this purpose |
| Preset library | The 18 YAML workflows and 18 skills already serve this purpose |
| Framework-specific review checklists | `lang-review.sh` covers language-level patterns; framework-specific rules are better added per-project via hookify |
| Conditional hook firing | Hooks already self-filter internally (extension checks, changed-file checks); a generic condition system adds complexity for no current need |
185 changes: 17 additions & 168 deletions commands/audit.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,182 +1,31 @@
---
description: Unified project health audit — dependencies, vulnerabilities, outdated packages, licenses, lint, and security in one report.
description: Unified project health audit — dependencies, vulnerabilities, outdated packages, licenses, lint, and security.
---

# Project Audit

Full-project health check in a single pass. Consolidates dependency vulnerabilities, outdated packages, license compliance, code quality, and security findings into one scored report.
Deterministic project health check: detect ecosystem → audit dependencies → check licenses → lint → security scan → report.

## Parameters

1. **Target** — directory to audit (default: project root)
2. **Budget** — max USD (default: $1)

## Budget

- **Token budget:** ~100k tokens. Mostly tool output from audit commands.
- Prefer `haiku` for parsing/summarizing tool output, `sonnet` for the final report.

## Step 1: Detect Ecosystem

Detect which package managers and tools are present:

```bash
echo "=== Ecosystem Detection ==="
[ -f package.json ] && echo "node: package.json found"
[ -f package-lock.json ] && echo "node: package-lock.json found"
[ -f yarn.lock ] && echo "node: yarn.lock found"
[ -f pnpm-lock.yaml ] && echo "node: pnpm-lock.yaml found"
[ -f requirements.txt ] && echo "python: requirements.txt found"
[ -f pyproject.toml ] && echo "python: pyproject.toml found"
[ -f Pipfile.lock ] && echo "python: Pipfile.lock found"
[ -f go.mod ] && echo "go: go.mod found"
[ -f Cargo.toml ] && echo "rust: Cargo.toml found"
[ -f Gemfile.lock ] && echo "ruby: Gemfile.lock found"
```

## Step 2: Dependency Vulnerabilities

Run the appropriate audit tool for the detected ecosystem:

| Ecosystem | Command |
|-----------|---------|
| Node (npm) | `npm audit --json 2>/dev/null` |
| Node (yarn) | `yarn audit --json 2>/dev/null` |
| Node (pnpm) | `pnpm audit --json 2>/dev/null` |
| Python | `pip-audit --format json 2>/dev/null` or `safety check --json 2>/dev/null` |
| Go | `govulncheck ./... 2>/dev/null` |
| Rust | `cargo audit --json 2>/dev/null` |
| Ruby | `bundle audit check 2>/dev/null` |

If the audit tool isn't installed, note it as "skipped — {tool} not installed" and continue.

Parse the JSON output and extract:
- Total vulnerabilities by severity (critical, high, moderate, low)
- Top 5 most severe findings with package name + advisory

## Step 3: Outdated Packages

```bash
# Node
npm outdated --json 2>/dev/null || true

# Python
pip list --outdated --format json 2>/dev/null || true

# Go
go list -m -u all 2>/dev/null | grep '\[' || true

# Rust
cargo outdated --format json 2>/dev/null || true
```

Categorize as:
- **Major** — breaking version behind (e.g., v3 → v5)
- **Minor** — feature versions behind
- **Patch** — only patch versions behind

## Step 4: License Compliance

Check for potentially problematic licenses in dependencies:

```bash
# Node
npx license-checker --json --production 2>/dev/null | head -200 || true

# Python
pip-licenses --format json 2>/dev/null | head -200 || true
```

Flag:
- **Copyleft** licenses (GPL, AGPL, LGPL) in non-copyleft projects
- **Unknown** or missing licenses
- License conflicts with the project's own license (read LICENSE or package.json license field)

If license tools aren't installed, skip with a note.

## Step 5: Code Quality Summary

Run whatever linter/type checker is configured:

```bash
# Detect and run
[ -f .eslintrc* ] || [ -f eslint.config.* ] && npx eslint . --format json 2>/dev/null | head -100
[ -f tsconfig.json ] && npx tsc --noEmit 2>&1 | tail -5
[ -f pyproject.toml ] && (ruff check . --output-format json 2>/dev/null || flake8 . --format json 2>/dev/null) | head -100
[ -f .golangci.yml ] && golangci-lint run --out-format json 2>/dev/null | head -100
```

Count total errors and warnings. Don't fix anything — just report.

## Step 6: Security Patterns

Spawn the `security-auditor` agent for a quick scan:
## Invoke

```
Task: Quick security scan of {target}. Check for:
- Hardcoded secrets, API keys, tokens
- SQL injection patterns
- XSS vulnerabilities
- Insecure dependencies usage
- Exposed debug/admin endpoints
Agent: security-auditor
Report: list of findings with severity and file:line
devkit workflow run audit
```

## Step 7: Generate Report

Produce a scored report:

```
## Project Audit Report

**Project:** {name}
**Date:** {date}
**Overall Score:** {score}/100

### Scoring
| Category | Score | Weight | Details |
|----------|-------|--------|---------|
| Vulnerabilities | {0-100} | 30% | {critical}C {high}H {moderate}M {low}L |
| Dependencies | {0-100} | 20% | {major_outdated} major, {minor_outdated} minor behind |
| Licenses | {0-100} | 15% | {issues} issues found |
| Code Quality | {0-100} | 20% | {errors} errors, {warnings} warnings |
| Security | {0-100} | 15% | {findings} findings |

### Critical Findings (fix immediately)
{list critical/high severity items across all categories}

### Warnings (fix soon)
{list moderate items}

### Informational
{list low-severity and suggestions}

### Skipped Checks
{list any tools that weren't available and how to install them}

### Recommended Actions
1. {highest priority fix}
2. {second priority}
3. {third priority}
```

## Scoring Guide

- **Vulnerabilities:** Start at 100. -25 per critical, -10 per high, -3 per moderate, -1 per low.
- **Dependencies:** Start at 100. -5 per major-version-behind package, -1 per minor.
- **Licenses:** Start at 100. -20 per copyleft in non-copyleft project, -10 per unknown license.
- **Code Quality:** Start at 100. -2 per error, -0.5 per warning. Floor at 0.
- **Security:** Start at 100. -25 per critical, -10 per high, -3 per moderate.
If `devkit workflow` is not available, follow this manually:

Overall = weighted average, rounded to nearest integer.
1. **Detect ecosystem** — Check for go.mod, package.json, requirements.txt/pyproject.toml, Cargo.toml
2. **Dependency vulnerabilities** — Run ecosystem audit tools (`npm audit`, `go vuln`, `pip-audit`, `cargo audit`)
3. **Outdated packages** — Check for outdated dependencies, report major/minor/patch updates available
4. **License compliance** — Scan dependency licenses, flag non-permissive or unknown licenses
5. **Code quality** — Run linters for detected ecosystems, count errors and warnings
6. **Security patterns** — Grep for common vulnerabilities: hardcoded secrets, SQL injection, command injection, path traversal
7. **Generate report** — Health score (A-F) with breakdown by category, actionable recommendations

## Rules

- Never fix anything — audit only, report only
- Always continue if a tool is missing — skip and note it
- Use `--json` output formats where available for reliable parsing
- Cap command output with `head` to avoid context bloat
- Run ecosystem-specific checks only for detected ecosystems
- The security-auditor agent runs in worktree isolation
- Report should be actionable — every finding needs a "what to do" line
- Auto-detect ecosystem — don't ask the user what language
- Run actual audit tools — don't guess at vulnerabilities
- Score overall health: A (excellent) through F (critical issues)
- Prioritize findings by severity
- Token budget: ~100k tokens
112 changes: 18 additions & 94 deletions commands/autoloop.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,108 +1,32 @@
---
description: Autonomous improvement loop inspired by karpathy/autoresearch — audit, fix, measure, keep or revert, repeat.
description: Autonomous improvement loop — audit, fix, measure, keep or revert, repeat.
---

# Autoloop

Autonomous codebase improvement loop. Each cycle: audit → pick hypothesis → fix → measure → keep or revert → repeat.
Autonomous iterative improvement: gather inputs → baseline → audit → fix → measure → compare → keep/revert → repeat.

Inspired by [karpathy/autoresearch](https://github.com/karpathy/autoresearch): one metric, keep or discard, loop.
## Invoke

## Step 0: Harness Detection

```bash
if command -v devkit >/dev/null 2>&1; then
echo "Go harness detected — delegating to devkit workflow autoloop."
devkit workflow autoloop "{input}"
exit 0
fi
```

## Step 1: Gather Inputs

If the input doesn't contain a metric command, use `AskUserQuestion` to collect:

1. **Objective** — what to improve
2. **Metric command** — how to measure (auto-detect from stack if not provided)
3. **Direction** — higher-is-better or lower-is-better
4. **Iterations** — how many cycles (default 10)
5. **Scope** — file/package constraints (optional)
6. **Guard** — optional safety-net command that must always pass (e.g., `npm test`, `tsc --noEmit`). Changes that improve the metric but break the guard are treated as regressions and reverted. Guard commands must be side-effect free (no generated files, snapshots, or caches that persist between runs).

## Step 2: Baseline

Run the metric command. Record the starting number and direction.

If a guard command is set, run it now. If the guard fails at baseline, stop and tell the user — the invariant must hold before the loop can start.

## Step 3: Audit

Analyze the codebase for the single highest-impact change. Read the scratchpad to avoid repeating failed approaches. Output one hypothesis with target files.

## Step 4: Fix

Make the recommended change. Minimal, focused, no unrelated refactoring.

## Step 5: Measure

Run the EXACT same metric command. Record the new number.

## Step 6: Compare

Compare baseline vs measurement using the direction:
- higher-is-better: new > old → IMPROVED
- lower-is-better: new < old → IMPROVED
- Equal or failed → REGRESSED

## Step 6.5: Guard Check (if guard command is set)

Guard is only evaluated when the metric improved. Regressions are already reverted in Step 7, so running the guard on them would be wasted work. If the metric improved, run the guard with a timeout:

```bash
timeout 120 {guard_command} 2>&1
devkit workflow run autoloop "{metric_command}"
```

- Guard passes (exit 0) → proceed to Step 7 as IMPROVED
- Guard fails or times out → treat as REGRESSED regardless of metric improvement. Log: "Metric improved but guard failed — reverting to protect invariant."

## Step 7: Keep or Revert

- **IMPROVED** → stage only modified files + `git commit`, update scratchpad, update baseline to new number, loop back to Step 3
- **REGRESSED** → `git checkout -- <modified files>` (no `git clean -fd` — protect untracked work), update scratchpad with failure reason, increment per-file failure counter (see Escalation below), loop back to Step 3

### Escalation: Repeated Failures

Track consecutive failures by primary modified file. After 3 failed attempts where the same file is the main edit target:

1. **Log** what was tried and why each approach failed
2. **Skip** — move the hypothesis to a "blocked" list in the scratchpad
3. **Pivot** — choose a completely different hypothesis targeting different files
4. **Report** — include blocked items in the final report with context for manual investigation

Never loop on the same failing approach. Each attempt must use a materially different strategy.

## Step 8: Report

After all iterations or budget exhausted:
- Starting vs final metric
- List of kept changes with impact
- List of reverted attempts with failure reason
- Net improvement
- Recommendation for next steps

## Budget
If `devkit workflow` is not available, follow this manually:

- **Token budget:** ~500k tokens. Each cycle costs ~30-50k tokens.
- **Iteration limit:** User-specified (default 10).
- Budget or iteration limit, whichever hits first, stops the loop.
1. **Gather inputs** — Identify metric command, objective, guard command (optional), and iteration count
2. **Baseline** — Run metric command, capture starting state
3. **Audit** — Analyze codebase for improvement opportunities
4. **Fix** — Make one targeted improvement
5. **Measure** — Re-run metric command
6. **Compare + guard** — Did the metric improve? If yes AND a guard command is set, run it. If guard fails, treat as regression and revert.
7. **Keep or revert** — Keep if improved and guard passes; revert otherwise. If 3+ consecutive failures, escalate.
8. **Report** — Summary of iterations, improvements kept, and final state

## Rules

- Every change must be measured — no skipping the metric step
- Never keep a regression — always revert
- One hypothesis at a time — no bundling
- Use the scratchpad to prevent repeating failures
- The metric command must be identical in baseline and measure steps
- Update the baseline number after each kept change (so the next cycle compares against the new state, not the original)
- Guard failures override metric improvements — never accept a change that breaks the guard
- 3 failures on the same file → skip and pivot, don't grind
- One change per iteration — don't bundle
- Always measure before and after
- Revert on regression or guard failure
- Stop after 3 consecutive failures and report
- Guard command (if set) must pass to keep a change
Loading
Loading