Main Tracker: RBForge updates

[DJLougen]

Current Release State

• v0.1.0: initial RBForge baseline before the RBMEM v0.4 integration pass.
• v0.2.0: RBMEM v0.4 JSON diagnostics/context integration and working-name cleanup.
• v0.3.0: first-class rbforge doctor CLI for health and metrics checks.
• v0.4.0: debugger metrics, debugger signal extraction, and RL reward shaping.
• v0.5.0: debugger-first eval benchmark with tweetable proof metrics.
• v0.6.0: broader 15-case debugger benchmark with per-family metrics.
• Current main: 96c20de

Done

• Removed the old working-name references from code, docs, examples, configs, tests, and generated graph reports.
• Renamed the internal prototype package to rbforge_core.
• Kept the public RBForge package/API available for users.
• Updated package metadata to 0.6.0.
• Added CHANGELOG.md.
• Updated persisted record schemas to rbforge.*.
• Updated docs to describe RBMEM v0.4 integration.
• Added rbforge doctor with text and JSON output.
• Added deterministic debugger signal extraction for logs, tracebacks, failing tests, suspect files, and exception types.
• Added debugger-specific doctor metrics.
• Added debugger-use reward shaping and a debugger RL trace fixture.
• Added rbforge eval debugger with replay fixtures for debugger-first vs no-debugger comparisons.
• Expanded the debugger eval fixture from 3 cases to 15 cases across 13 debugging families.
• Added per-family debugger eval metrics in JSON output.
• Tagged and pushed v0.1.0, v0.2.0, v0.3.0, v0.4.0, v0.5.0, and v0.6.0.

RBMEM v0.4 Integration

RBForge now uses the machine-readable Rust-Brain / RBMEM interfaces:

• RbmemStore.rbmem_version() calls rbmem --version.
• RbmemStore.doctor() calls rbmem hermes doctor --format json.
• RbmemStore.context(...) calls rbmem query --format json.
• Registry and tool lookup use structured query sections instead of regex over minified text.
• Forge results include rbmem_diagnostics and a task-specific rbmem_context_preview.

rbforge doctor

The doctor command checks:

• RBForge version
• RBMEM CLI version
• memory file health
• registry size
• number of forged tools
• validated tool count
• validation rate
• average success rate
• debugger tool count
• debugger validation rate
• debugger average success rate
• category metrics

Example live output against local Hermes memory:

RBForge doctor
rbforge-version: 0.6.0
rbmem-version: rbmem 0.4.0
memory-health: ok
registry-size: 1
forged-tools: 1
validated-tools: 1
validation-rate: 100.0%
average-success-rate: 100.0%
debugger-tools: 1
debugger-validation-rate: 100.0%
debugger-average-success-rate: 100.0%

rbforge eval debugger

The eval command compares replayed debugger-first trajectories against no-debugger baselines.

Current broad fixture output:

RBForge debugger eval
cases: 15
families: 13
debugger-use-rate: 100.0%
root-cause-hit-rate: 100.0%
baseline-root-cause-hit-rate: 40.0%
avg-turn-reduction: 44.3%
estimated-turns-saved: 47
reusable-debuggers-created: 9
avg-debugger-signal-score: 0.6207

The fixture now covers traceback, test triage, lock contention, config/env, data pipeline, API integration, cross-platform path, async, test isolation, CLI, cache/state, filesystem, and RBMEM graph debugging families.

JSON output also includes per-family metrics, so we can see where debugger-first behavior is helping most.

Debugger RL Signal

The config now rewards the model when it:

• uses a debugger before patching,
• extracts root cause signals from debugger output,
• reuses an existing debugger tool when available,
• forges a reusable debugger only when the missing capability is real.

It penalizes skipped available debuggers, ignored debugger output, and duplicate low-value debugger tools.

Verification

Python checks passed:

$env:PYTHONPATH='src'
python -m compileall -q src tests examples scripts
pytest -q
python -m ruff check .

CLI smoke tests passed:

python -m rbforge_core.cli eval debugger
python -m rbforge_core.cli eval debugger --format json

Live smoke test against local Rust-Brain binary passed:

$env:PYTHONPATH='src'
$rbmem='C:\Users\basbe\Documents\GitHub\AIresources\Rust-Brain\target\release\rbmem.exe'
python -m rbforge_core.cli doctor C:\Users\basbe\.hermes\MEMORY.rbmem --rbmem-cli $rbmem

mypy was not run because it is not installed in the active Python environment.

Next Refinement Ideas

• Add real Hermes/RBForge session cases alongside the synthetic replay fixture.
• Add a --compare mode that accepts two model trajectory JSONL files.
• Add direct JSON output examples for rbmem_diagnostics and rbmem_context_preview.
• Decide whether the legacy src/RBForge/forge_tool.py should delegate internally to rbforge_core to reduce duplication.
• Add a compatibility note for users importing the old internal package name.
• Re-run Graphify with code extraction when its local config is fixed; graphify update . currently sees only docs in this repo.