FR: Commit-by-commit incremental processing for staying current

[benthomasson]

Problem

The current --since option for updating beliefs after code changes is fragile. Large changes (big refactors, multi-file renames, feature branches with many commits) don't get accurately processed when diffed as a single blob. Beliefs derived from stale code accumulate, and the manual update cycle can't keep up.

Observed in practice: agents-python-expert has 1,175 beliefs. When the codebase evolves, bulk re-processing misses the semantic intent of individual changes and produces incorrect interpretations or misses relevant updates entirely.

Proposed Solution

Process changes commit-by-commit, and within each commit, changed-file-by-changed-file:

1. Track the last-processed commit SHA
2. Walk the commit log from last-processed to HEAD
3. For each commit:
   • Read the commit message for intent/context
   • For each changed file in the commit:
     • Diff the file (not the whole repo)
     • Update/retract/add beliefs affected by that specific file change
     • Record the new source hash
4. Update the last-processed SHA

Why commit-by-commit?

• Each commit has a message that explains intent — the LLM can use this to understand why the change was made, not just what changed
• Small diffs are easier to process accurately than large aggregate diffs
• A rename + refactor in one commit is interpretable; the same changes squashed with 20 other commits are not
• Mirrors how human reviewers process changes (PR review is commit-by-commit)

Why file-by-file within a commit?

• Isolates the blast radius of each change
• A change to auth.py shouldn't cause re-processing of beliefs about pipeline.py
• Allows parallel processing of independent file changes
• Makes it clear which beliefs need updating based on which files changed

Acceptance Criteria

• Track last-processed commit SHA per knowledge base
• Walk commit log incrementally (not full re-scan)
• Process each commit's changed files individually
• Use commit message as context for belief updates
• Handle file renames/deletes (retract beliefs whose source file no longer exists)
• Idempotent — re-running on already-processed commits is a no-op