BACKLOG.md

BACKLOG.md — memharness improvement backlog (dogfood-derived)

Companion to DOGFOOD.md, which holds the dogfood gate (use it for real, stopping rules, the 2026-06-23 decision window, how to generate belief data). This file is the improve half: it turns the friction that daily use surfaces into a prioritized, evidence-backed backlog. Generated 2026-06-16 from four survey lenses (gaps, measurement, live-store, arch) over the repo + the live store; every row cites real path:line or live data.

1. The operational loop

Each stage names the actual tool/file that plays the role. (Decision/stopping criteria for whether to keep going live in DOGFOOD.md; this is the what to fix loop.)

Stage	What happens	Where it lives
Use	Agents call the MCP tools as memory during real work (global ~/.claude/CLAUDE.md mandates proactive recall/remember). The only source of real belief data.	packages/mcp/src/server.ts (7 tools); ~/.memharness/memory.db.
Capture friction	(a) usage.log — every op + counters, no content, logUsage packages/mcp/src/index.ts:13; (b) gardener — SessionStart hygiene pass (fragmentation + code-map bloat), packages/core/scripts/memory-gardener.mjs; (c) staleness bin — cold-path git pin-drift check, packages/mcp/src/staleness.ts; (d) write-time nudges — length nag + code-map smell, packages/mcp/src/server.ts:199-214.	usage.log, gardener, staleness bin, server nudges.
Measure	Eval harness replays a synthetic dataset across retrieval/feature arms; reports recall@k by category.	packages/eval/src/runner.ts (CONFIGS:16, runEval:110); eval-results.json.
Triage	Survey friction vs eval numbers; dedupe; sort by impact-then-effort (this file). Edge-case semantics in EDGE-CASES.md; phased plans in STALENESS-SIGNAL.md.	this file + EDGE-CASES.md.
Fix	Land in packages/core (schema/semantics) or packages/mcp (surface/instrumentation), pinning invariants in tests (test/property/*, revise.test.ts, recall.test.ts).	packages/core/src/{memory,sql}.ts, packages/mcp/src/*.
Re-eval	Add the new arm/metric first (so the win is visible), re-run eval + gardener + staleness bin, confirm usage.log signals move.	packages/eval/src/runner.ts, gardener, staleness bin.

The loop's weak link today is Measure → Triage: usage.log counts hits but not relevance, and runner.ts records only hit: boolean (line 31) over a synthetic dataset — so fragmentation, code-map rot, semantic dupes, and deployment skew have no metric that would catch a regression. Section 4 closes that.

2. Prioritized backlog (deduped, impact-then-effort)

Item	Problem	Evidence	Proposed change	Impact	Effort	Measure of done
Schema-skew signal on startup	m005 columns are in the tree but the live DB is user_version=4; built changes are inert until the server restarts + migrates. remember with source_commit writes against a missing column. No "you're behind" signal.	Live PRAGMA user_version=4 (code expects 5); migrations/index.ts length 5; DB opened at packages/mcp/src/index.ts; logUsage:13 has no startup op.	On Memharness.open() compare user_version vs MIGRATIONS.length; emit catchable SCHEMA_MISMATCH (warn, don't crash); add open({autoMigrate}); log startup{schema_version,expected_version}; eval asserts parity.	high	S→M	usage.log startup shows actual==expected; eval refuses on skew; zero silent-NULL pins.
Code-map bloat: harden the nudge	Large structural facts pinned to stale commits read like maps Explore could re-derive; the looksLikeCodeMap nudge is advisory and ignored, so they land and rot (~2/3 of project:outerport-backend were code-map snapshots).	server.ts:48-59 (looksLikeCodeMap); server.ts:206-213 (nudge never blocks); MAX_FACT_CHARS=1500 server.ts:23; gardener memory-gardener.mjs:74-86.	Hard-reject when len > MAX_FACT_CHARS AND looksLikeCodeMap (detector is low-FP); message: "split into decision/rationale/version-pin/known-failure, or regenerate structure with explore."	high	S	eval compound_fact_fraction; usage.log rejected{reason:"codemap_too_long"} > 0; gardener smelly → 0.
Recall relevance unmeasured	Hits, not relevance: hits:5 could be 1 correct + 4 distractors. No precision/MRR/rank-of-first-hit.	usage.log only {op:"recall",hits:N}; runner.ts:31 ProbeOutcome = hit: boolean; runner.ts:135.	Extend ProbeOutcome with rank-of-first-gold + precision@k; compute MRR/precision@k in runEval; optional rated field on recall usage; offline label 10–20 live queries.	high	M	eval reports MRR + precision@k per arm; ranking regressions become visible.
Subject fragmentation: detect + repair	One project under many keys; gardener flags but never fixes; concurrent producers keep writing the wrong key. No dedupe-coverage metric.	Live project:TakoVM(20) vs project:tako(1); history tako(10)/tako-vm(4)/TakoVM(23); memory-gardener.mjs:50-58.	(a) subject_fragmentation_ratio + alias_precision in eval. (b) subject-aliases.json loaded on open(), canonicalize subject in remember/revise before store; normalize-subjects backfill.	high	M	gardener fragmented==0; ratio tracked; zero new non-canonical writes.
purge (Phase 4) vs chain/stable-id invariants	Hard delete breaks I4 ("never delete"): why() walk breaks on a missing row; (tx_at,id) tie-break assumes no holes; purge-by-sourceRef misses revisions' own source_ref.	EDGE-CASES.md Phase 4; why() memory.ts:516-517; chain walk memory.ts:436-441.	Define purge semantics before Phase 4: repair links or refuse mid-chain; return destroyed ids; follow-the-chain purge for privacy.	high	L	invariant tests for mid-chain purge + repair; why() never truncates; purge removes all revisions' source_refs.
Phase 3 staleWeight demotion un-validated	Demotion multiplier pre-wired neutral (1.0), never enabled; zero measurement of whether it helps precision or harms useful old facts.	sql.ts:71,151 no STALENESS_FACTOR; ranking.ts no staleWeight; STALENESS-SIGNAL.md §6–7.	Add staleWeight to DEFAULT_RANKING; multiply staleFactor into score() + both SQL queries; recall param; add hybrid-staleWeight=0.5 arm + staleness-resilience category. Calibrate after Phase 2 verdicts.	med	S	new arm shows stale ranks below equal-current; resilience precision@k up, recall flat.
get-fact by id	No fetch-by-id; surgical work forces recall→search→find (e.g. "consolidate #83 and #115").	RecallInput types.ts:90-102 no factId; GET_FACT already used memory.ts:431.	mem.getFact(id) over existing GET_FACT; MCP tool get-fact. ~1 line, no ranking.	med	S	tool returns the fact; usage.log get-fact ops replace multi-step workarounds.
Staleness pipeline depth (Phase 2/3)	bin has --check/--flag only; no --revise to decay confidence; no recall freshness filter; no consolidate() pass.	staleness.ts:29-42; sql.ts:30 has KIND_FILTER not FRESHNESS_FILTER; no consolidate in core.	--revise (chain-preserving confidence decay, never auto-retract); freshness param + FRESHNESS_FILTER; consolidate(opts) reporting dup + decay candidates (no auto-merge).	med	L	tests: why()+ranking reflect downgrades; recall honors freshness='current'; consolidate idempotent, never mutates unprompted.
Semantic dedup gap	Same claim, different words (#83/#115); only subject-string normalization, no semantic detector.	EDGE-CASES.md; embeddings already at write (setEmbedding); no cosine-dedup.	Advisory post-recall dedup: cluster cosine ~0.85–0.92 within subject → "may be the same; revise one to supersede." Read-only. Add dedup_recall/dedup_precision.	med	M	eval dedup P/R on labeled clusters; advisory clusters surfaced; gardener gains dedup flag.
E3: revise vs forget semantics	revise checks superseded_by not retracted_at; forget retracts heads + mid-chain indiscriminately. Revising a retracted fact yields a live head with disavowed ancestry.	EDGE-CASES.md E3; memory.ts:433-446; sql.ts:15 RETRACT_BY_ID.	Pick one invariant ("never true" vs "stop surfacing"); pin both paths in tests.	med	M	invariant tests for revise-of-retracted + forget-of-mid-chain.
E4: refile subject/predicate without severing chain	revise copies subject/predicate verbatim; misfiling fixable only by forget+remember, losing the chain.	EDGE-CASES.md E4; memory.ts:465-466.	Validate frequency first; then optional subject/predicate override on revise (default off) or a refile op.	med	M	misfiled corrections keep why() chain; forget+remember refile pairs drop.
Concurrent-write coordination	Parallel sessions wrote project:tako during a cleanup burst, unaware of each other. No session id; source_agent unused.	usage.log cleanup burst then concurrent writes; source_agent free text.	Log session_id per op; track write-conflict rate; gardener warns "another session wrote {subject} since your last check." Metadata-only.	med	M	conflict rate tracked; concurrency warning fires; correlatable session ids.
Cross-repo pin scoping	Global DB spans ≥3 repos; a source_commit resolves in one worktree not another; bin takes one --repo → over-flags unresolved.	staleness.ts:30; live pins across outerport-backend/memharness/TakoVM; STALENESS-SIGNAL.md §9.1.	Measure cross-repo pin frequency; if rare, document per-repo runs; if common, additive repo column (m006) + filter STALENESS_TARGETS.	med	M	frequency measured; per-repo unresolved rate; no false current.
Compound-fact length nag unmeasured	>1500ch rejected, >280ch nagged, but the rate is untracked; ~1500ch facts couldn't be revised piece-by-piece.	server.ts:23,182-184,200-205; usage.log no rejected op.	Log rejected{reason,length}; eval fact_length_distribution + compound_fact_fraction; splitting probe.	low	S	rejection rate visible; length percentiles; split pieces individually recalled.
Phase 3 freshness filter on recall	Can't ask recall for only-current/only-stale when auditing.	types.ts:90-102 no freshness; sql.ts:30 KIND_FILTER only.	Optional freshness param + FRESHNESS_FILTER mirroring KIND_FILTER.	low	S	recall freshness='current' excludes stale in tests.

3. Top 3 next moves

1. Startup schema-skew signal (S→M). The live DB is silently at user_version=4 while code expects 5 — so source-commit pins, the staleness bin, and the gardener's hasPin path all sit dormant and unannounced. Making skew loud (and optionally auto-migrating) turns the whole staleness pipeline from inert to live. Cheapest unblocker of everything else.
2. Harden the code-map nudge to a hard rejection (>1500ch AND looksLikeCodeMap) (S). Highest impact per line: detector already low-FP, ceiling already exists, live store proves the advisory nudge is ignored. Stops the dominant bloat source at write time instead of a week later in the gardener.
3. Measure recall relevance, not hit counts (M). Every ranking lever (staleWeight, hybrid fusion, importance) is invisible while ProbeOutcome is hit: boolean. precision@k / MRR / rank-of-first-hit in runner.ts is what makes the rest of this backlog testable and regression-guarded.

4. New evals / metrics

• Ranking quality (runner.ts): ProbeOutcome gains rankOfFirstGold, goldHitsInTopK; compute MRR/precision@k/recall@k per config. Foundation move.
• staleWeight arm + staleness-resilience category in CONFIGS:16, gold mixing fresh+stale facts.
• Subject-fragmentation metrics: subject_fragmentation_ratio, alias_precision; mirror gardener norm (memory-gardener.mjs:50).
• Code-map rot metric: tag facts code-map vs semantic via looksLikeCodeMap (server.ts:48); for pinned code-maps compute freshness; report code_map_rot_rate, stale_reference_rate.
• Semantic-dedup metric: cluster live facts within subject at cosine ~0.85–0.92; report dedup_recall/dedup_precision.
• logUsage extensions (index.ts:13): startup{schema_version,expected_version}; rejected{reason,length,subject}; optional rated{relevant_count,relevant_positions} on recall; session_id on every op; fact_length on remember.
• Schema-parity assertion in the harness before runEval.
• Offline labeled live-query set: 10–20 real queries sampled from usage.log, scored hit@k / precision@k / MRR — the bridge from synthetic eval to real dogfood relevance.