Idea: mine feedback.md git history to harvest doer mistakes and improve member skills

[kumaakh]

Idea

Fleet members build up two kinds of persistent memory over time, both of which are a rich signal for skill improvement:

1. feedback.md (reviewer-generated) — Every time a reviewer catches a mistake by a doer, the correction lands in the doer's feedback.md and is committed to git. Over time this accumulates a timestamped record of every mistake made and what the right behaviour was.

2. PM memories (pm-generated) — The PM also develops LLM-style memories across sessions: lessons learned about a project, patterns in how members fail or succeed, and corrections to its own planning. These are equally valuable as a signal.

Both sources are currently unused beyond the individual member or session that generated them.

What Could Be Built

A periodic or on-demand analysis pass over the git history of memory files that:

1. Extracts mistake patterns — what categories of errors recur? (wrong branch, skipped step, over-engineered solution, wrong tool used, etc.)
2. Ranks by frequency / recency — which mistakes happen most often, or are still happening?
3. Generates skill improvement suggestions — new rules to add to a member's CLAUDE.md / skill files, or updates to existing profiles in skills/profiles/
4. Cross-member aggregation — if multiple members make the same mistake, it is likely a gap in the shared skill/profile rather than an individual problem → fix the profile, not just the member

Why This Is Valuable

• Both feedback.md and PM memories are already written as a side-effect of normal fleet operation — no extra instrumentation needed
• Git history gives timestamps, so trends over sprints can be tracked
• Reviewer corrections today can proactively prevent the same mistake on a different member tomorrow
• PM memories capture higher-level patterns (project-level, planning-level) that complement the doer-level feedback
• Closes the feedback loop: reviewer corrections + PM learnings → skill improvement → fewer corrections needed

Possible Implementation

• A harvest-feedback command or fleet skill that takes a member (or --all) and a date range
• Reads git log of memory/feedback_*.md and PM memory files, diffs each version, extracts the delta (new rule added = new mistake caught)
• Groups by category, ranks by frequency
• Outputs a suggested patch to the relevant skill profile or CLAUDE.md

Notes

• Works best when feedback entries follow the structured format (rule + Why: + How to apply:) — parsing is straightforward
• Could also feed into onboarding: new members get pre-loaded with the most common mistake rules before they ever make them

[kumaakh]

Technical direction: Start simple — a read-only harvesting script, not a new MCP tool.

1. scripts/harvest-feedback.mjs: takes --member or --all, optional --since . Walks git log --diff-filter=A --follow -- 'memory/feedback_*.md' in each member's work folder to find new rule additions. Diffs each version to extract the delta (new rule text = new mistake caught).
2. Categorization: group rules by keyword buckets (branch, permissions, tool, formatting, etc.) — simple string matching is enough to start.
3. Output: ranked list of most common mistake categories + suggested additions to the relevant skills/profiles/*.md or pl-doer.md.
4. PM memory files: same approach — walk PM's memory/ dir git history for any file, extract additions.

The structured feedback format (Why: / How to apply:) makes parsing straightforward with regex. No need for an LLM pass in the harvesting script itself — the raw rules are already actionable text.