feat(selfaudit): add idle-time self-audit sweep + run_self_audit MCP tool - #199
Merged
Conversation
…audit tool (Closes #164) Adds a lightweight, LLM-driven review pass (bugs, missing test coverage, security holes) that runs when the Superintendent's Issue queue is empty, throttled to roughly hourly via a new SelfAuditSince cadence timestamp (mirroring RequirementsSweepSince), plus an on-demand run_self_audit MCP tool that shares the exact same filing logic. Design (Go-vs-LLM split, per Issue #164's open judgment call): - internal/selfaudit is the single implementation both call paths share. It cannot itself decide what "a bug" is (unlike run_requirements_sweep's test-command exit code, or run_health_checks' check command, there is no deterministic verdict to shell out to for "review this code"), so that judgment is delegated to the calling LLM via a shared Instructions text. - Everything mechanical is done in Go: cadence tracking (self_audit_since via get_loop_state/update_loop_state, following the existing pattern exactly), deduping a finding against BOTH open and closed Issues (a new ListIssuesAnyState client method, since a finding whose Issue was already filed and since closed must not be re-filed), and actually filing the Issue (self-audit label). The Superintendent never touches the GitHub API directly for this — it only produces structured findings. - run_self_audit is two-phase/stateless: called with no `findings` it returns instructions to follow; called again with `findings` it dedupes and files. This keeps the on-demand and cadence-driven paths identical. Also updates CLAUDE.md's step 9 (extended rather than renumbering the whole cycle) to describe the new cadence, the .claude/settings.json permissions allowlist (new mcp__hermit__run_self_audit entry), and the expected-tool-count assertions in cmd/hermit/serve_test.go (21 -> 22). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Owner
Author
HERMIT: Superintendentによる補足レビュー(HIGH risk)
変更内容の要約
評価
懸念点は見当たりません。 HIGH riskは変更量とパスの機微性によるフラグであり、実装内容としては安全にマージ可能と判断します。マージはポリシー通り人間の判断待ちとします。 |
Owner
Author
|
|
…sue-164 # Conflicts: # internal/mcp/tools.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #164.
internal/selfauditpackage implementing the lightweight self-audit sweep: dedupe findings against existing GitHub Issues and file (deduped) Issues for real ones.SelfAuditSincetostate.LoopState(mirrorsRequirementsSweepSinceexactly), exposed via the existingget_loop_state/update_loop_stateMCP tools.run_self_auditMCP tool ininternal/mcp/tools.go.github.Client.ListIssuesAnyState(open+closed) since dedup must not re-file an Issue that was already filed and since closed.mcp__hermit__run_self_auditto.claude/settings.json's permissions allowlist, and bumps the expected-tool-count assertions incmd/hermit/serve_test.go(21 → 22).Go-vs-LLM design rationale (explicit judgment call per the Issue)
run_requirements_sweepandrun_health_checksboth compute their pass/fail verdict deterministically in Go (a test command's exit code, a health-check command's exit code). "Review the codebase for bugs/missing tests/security holes" has no equivalent deterministic command to shell out to — so that judgment is delegated to the calling LLM (the Superintendent, or an on-demand caller), guided by a singleinternal/selfaudit.Instructionsstring that is the one source of truth for what the sweep looks for. CLAUDE.md's step 9 text mirrors this string in prose rather than duplicating the judgment logic.Everything else — cadence tracking, deduping a finding against existing Issues, and actually filing the Issue — is implemented in Go via
internal/selfaudit.File, the single function both the CLAUDE.md-driven idle path and the on-demandrun_self_auditMCP tool call. This means the LLM side of the boundary only ever produces structured{title, body}findings and never talks to the GitHub API directly, keeping "Superintendent never fixes, only files Issues" enforced in code rather than relying on prose alone.run_self_auditis two-phase/stateless between calls:findingsargument, it returns aninstructionsfield (the shared text above) and files nothing.findingsarray of{title, body}objects, it dedupes each against existing open and closed Issues (via a newTitlePrefix+normalized-title match, followinginternal/healthcheck's title-prefix dedup convention but checkingListIssuesAnyStateinstead of just open issues) and files a new Issue (labeledself-audit) for each non-duplicate.Dedup is intentionally a simple exact-match-after-normalization heuristic (lowercase, punctuation collapsed,
[self-audit]prefix stripped) rather than fuzzy/semantic matching — proportionate to a "lightweight" audit sweep, not a from-scratch duplicate-detection engine.Test plan
go build ./...go vet ./...go test ./...(all packages pass, including newinternal/selfaudittests: cadence-independentFile/dedupe logic,GitHubIssueClientadapter, and MCP-tool-levelrun_self_auditwiring tests)internal/stateround-trip test extended to coverSelfAuditSincecmd/hermittool-count assertions updated for the new tool (21 → 22)🤖 Generated with Claude Code