Summary
total_memories_extracted and l0_conversations_count in recall_checkpoint.json only increment. After any cleanup (deleting test pipeline states, running memory-cleaner, or manual JSONL pruning), the counters permanently overstate reality.
Root Cause
CheckpointManager (bundled in dist/index.mjs L8339, L8375):
cp.total_memories_extracted += memoriesExtracted; // L8339
cp.l0_conversations_count += 1; // L8375
No code path decrements these counters. memory-cleaner deletes records from JSONL and SQLite, but never updates the checkpoint.
Reproduction
- Run gateway for a few days, accumulate ~45 L1 records
- Clean up test data: remove 8
pipeline_states entries from recall_checkpoint.json and delete corresponding JSONL lines
- Observe:
total_memories_extracted=50 while actual JSONL has 42 records
- Counter will never self-correct
Actual Impact
The drifted counters cause:
- Incorrect status reporting ("50 memories extracted" when only 42 exist)
- The counter is used by L2/L3 pipeline logic for persona generation thresholds (
cp.total_processed, cp.memories_since_last_persona), which may trigger prematurely or never trigger
Suggested Fix
Add a recalibrate() method to CheckpointManager that recounts from actual data:
async recalibrate(): Promise<void> {
await this.mutate((cp) => {
// Count actual JSONL records
const actualExtracted = /* count lines across all .jsonl files */;
cp.total_memories_extracted = actualExtracted;
// Count actual L0 conversations from DB
cp.l0_conversations_count = /* SELECT COUNT(*) FROM l0_conversations */;
});
}Call once on gateway startup. ~20 lines.
Summary
total_memories_extractedandl0_conversations_countinrecall_checkpoint.jsononly increment. After any cleanup (deleting test pipeline states, running memory-cleaner, or manual JSONL pruning), the counters permanently overstate reality.Root Cause
CheckpointManager(bundled indist/index.mjsL8339, L8375):No code path decrements these counters.
memory-cleanerdeletes records from JSONL and SQLite, but never updates the checkpoint.Reproduction
pipeline_statesentries fromrecall_checkpoint.jsonand delete corresponding JSONL linestotal_memories_extracted=50while actual JSONL has 42 recordsActual Impact
The drifted counters cause:
cp.total_processed,cp.memories_since_last_persona), which may trigger prematurely or never triggerSuggested Fix
Add a
recalibrate()method toCheckpointManagerthat recounts from actual data:Call once on gateway startup. ~20 lines.