Idea: mine feedback.md git history to harvest doer mistakes and improve member skills

[kumaakh]

The Idea

Fleet members generate two kinds of persistent memory as a natural side-effect of normal operation:

1. feedback.md (reviewer-generated) — Every time a reviewer catches a mistake by a doer, the correction lands in the doer's feedback.md and is committed to git. Over time this accumulates a timestamped record of every mistake made and what the correct behaviour was.

2. PM memories (pm-generated) — The PM develops LLM-style memories across sessions: lessons learned about a project, patterns in how members fail or succeed, corrections to its own planning. These capture higher-level patterns that complement the doer-level feedback.

Today both sources are siloed — used only by the member or session that generated them. The insight a reviewer adds to fleet-dev's feedback.md never reaches fleet-dev2. A planning mistake the PM corrects in one sprint recurs in the next because nothing was extracted and generalized.

This idea proposes closing that loop.

---

The Feedback Loop Fleet Is Missing

Doer makes mistake
  → Reviewer catches it → feedback.md updated → committed
    → (today) nothing happens
    → (proposed) harvester runs → pattern extracted → skill/profile updated
      → New members pre-loaded with lesson → mistake does not recur

The data is already there. Git history provides timestamps and diffs. The only missing piece is the analysis pass that reads across members and surfaces generalizable rules.

---

What Could Be Built

A periodic or on-demand harvest pass that:

1. Extracts mistake patterns — what categories of errors recur? (wrong branch, skipped step, over-engineered solution, wrong tool used, etc.)
2. Ranks by frequency / recency — which mistakes happen most often, or are still happening?
3. Detects cross-member patterns — if multiple members make the same mistake independently, it is likely a gap in the shared skill or profile rather than an individual problem → fix the profile, not just the member
4. Generates skill improvement suggestions — new rules to add to a member's CLAUDE.md / skill files, or updates to existing profiles in skills/profiles/
5. Pre-loads new member onboarding — new members get the most common mistake rules baked in before they ever make them

---

Why This Scales

The value of this system increases superlinearly with fleet size. With 2 members, the signal is thin. With 20 members across multiple organizations running daily sprints, the signal is rich: thousands of reviewer corrections, hundreds of PM memory updates, covering dozens of languages, frameworks, and deployment environments. The harvester turns that accumulated experience into a continuously improving shared knowledge base.

The structured format that fleet already requires — rule + Why: + How to apply: — makes parsing tractable. Entries follow a consistent schema; diffs are clean; categories are recognizable.

---

Connection to Other Fleet Memory Sources

As fleet gains more memory artifacts, each becomes an additional harvest signal:

• Playbook repair_log.json (see discussion #194) — every script repair records why a generated script failed. Cross-member repair logs reveal which execution environments or tool versions cause the most churn, informing smarter playbook generation.
• PM planning memories — recurring planning mistakes (wrong tier assignment, underestimated parallelism, missed pre-flight checks) can be extracted and added to PM skill profiles.

---

Possible Shape of Implementation

• A harvest-feedback command or fleet skill tool that takes a member (or --all) and a date range
• Reads git log of feedback_*.md and PM memory files, diffs each version, extracts the delta
• Groups by category, ranks by frequency
• Outputs a suggested patch to the relevant skill profile or CLAUDE.md
• Works best when feedback entries follow the structured format (rule + Why: + How to apply:) — parsing is straightforward

---

Open Questions

1. Automated vs on-demand — should harvesting run automatically on a schedule, or be a PM-triggered command? Automated risks over-writing carefully crafted profiles; on-demand requires the PM to remember to run it.
2. Confidence threshold — how many occurrences of the same mistake across members before it becomes a profile-level fix rather than a member-level fix?
3. Human approval gate — should suggested patches require explicit PM approval before being applied to shared profiles? Probably yes for shared profiles; possibly automatic for individual member CLAUDE.md.
4. Privacy across organizations — if fleet is used across multiple organizations, cross-org aggregation of mistakes is not appropriate. The harvester should scope to a single fleet instance by default.

[kumaakh]

Technical direction: Start simple — a read-only harvesting script, not a new MCP tool.

1. scripts/harvest-feedback.mjs: takes --member or --all, optional --since . Walks git log --diff-filter=A --follow -- 'memory/feedback_*.md' in each member's work folder to find new rule additions. Diffs each version to extract the delta (new rule text = new mistake caught).
2. Categorization: group rules by keyword buckets (branch, permissions, tool, formatting, etc.) — simple string matching is enough to start.
3. Output: ranked list of most common mistake categories + suggested additions to the relevant skills/profiles/*.md or pl-doer.md.
4. PM memory files: same approach — walk PM's memory/ dir git history for any file, extract additions.

The structured feedback format (Why: / How to apply:) makes parsing straightforward with regex. No need for an LLM pass in the harvesting script itself — the raw rules are already actionable text.