Repository navigation
hash_migrate: server-side DB writes on the read-only handle; failures logged as collisions; ghost duplicates share a hash #215
Copy link
Copy link
Closed
Description
Problem
cmd/server/hash_migrate.go (migrateContentHashesAsync) issues UPDATE/DELETE statements from the server process. The server opens SQLite with mode=ro, so this breaks the read/write separation invariant (Kpa-clawbot#1283). The current readonly_invariant_test.go does not catch it, because it only asserts three method names.
Found during the #209 review, using reviewer-only probes against master and the #209 merged tree:
- Every write fails on the production handle. With
OpenDB(mode=ro), two stale-hash duplicates, and the migration run, everyUPDATEfails.- Each failure is logged as
collides — merging duplicate, which is misleading. - The DB is left unchanged, with both rows and the old hash, so the migration repeats on every server start.
- Each failure is logged as
- Ghost duplicates in memory. Both transmissions take the new content hash in memory and both stay in
s.packets.byHashpoints to only one of them, andQueryPacketsreturns both. - Shared
nodeHashesentry.nodeHashesis keyed by hash. Evicting one ghost removes the sharednodeHashes[pk][hash]entry that the surviving transmission still relies on. This is inferred from the code, not probed. - Stale
nodeHasheskeys. Keys created before a migration are never removed. After both duplicates are evicted,byNodeis empty, butnodeHashesstill holds the relay keys under the old hashes. This applies to everybyNodesource, and it is bounded by migrated transmissions × relays. - Stale
chargedBytes. After migration,chargedBytesis off by the hash length difference. It is credited consistently, so there is no drift. - Untested constant.
perFallbackRelayBytes(added in fix(store): charge the path_json fallback and pin the hash-migration accounting (#202) #209) is not pinned by any default-run test. The mutantperFallbackRelayBytes = 0survives every_202/_113test, and the heap comparison that would catch it is env-gated (CORESCOPE_PERF_202).
Proposed direction
- Move the DB part of the content-hash migration (rehash, collision merge, duplicate delete) to
cmd/ingestor, like every other write. - The server should only rehash in memory. On a collision, it should merge in memory: move the duplicate's observations to the survivor, then remove the duplicate from
s.packets,byTxID,byHashand the node indexes, crediting each charge exactly once. - Re-key or remove the
nodeHashesentries under the old hash during migration. - Extend
readonly_invariant_test.go, or add a test, so that a write statement fromcmd/serverfails the build or a test. One option is a probe that runs the migration on amode=rohandle and asserts that no write is attempted. - Add a default-run bound for
perFallbackRelayBytes.
Acceptance
- No
UPDATE/DELETE/INSERTfromcmd/serverfor the hash migration, and a test enforces it. - Migration on a DB with stale-hash duplicates converges once (in the ingestor) and is not repeated on the next start.
- In memory, migrated duplicates leave exactly one transmission per hash, with all observations, and
QueryPacketscardinality is correct. - After migration and eviction,
byNode,nodeHashesandtrackedBytesall return to the baseline exactly, without relying on the clamp. -
StoreTxstays at 320 bytes. No newmap[string]interface{}. - A test pins
perFallbackRelayBytes, so the= 0mutant is killed.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
test(hash-migrate): cancel and resume, targeted rename, earliest firs…
fix(store): resumable ingestor migration, targeted rename, full merge…
test(hash-migrate): dest/src keys, list order, scope fill, lock-hold …
fix(hash-migrate): fill allow-list, write-lock hold summary, document…
Merge pull request #222 from dborup/codex/issue-215-hash-migrate-read…
dborup commentedon Oct 4, 2026
Fixed by #222 (merged as
97542417).The server no longer writes to the DB for the hash migration. A recording-driver probe on the
mode=rohandle, a source guard and the write-SQL ratchet now enforce this.The ingestor runs a one-time migration,
content_hash_formula_v1. It rehashes in id-ordered batches underwriterMu. When two rows collide:first_seentakes the MIN andlast_seenthe MAX;route_maskvalues are unioned;COALESCE;ping_triggersfollows the survivor.A cancelled run resumes to the same state and is logged as
cancelled (will resume).The server merges duplicates in memory over the whole store after the startup load. Each charge is credited once, and
nodeHashesis re-keyed through the transmission. The run logs its max write-lock hold per batch.perFallbackRelayBytesis now pinned by a default-run test.The deploy plan is in the PR description:
_async_migrationscontent_hash_formula_v1 = done.Remaining, documented in #222:
Merge remote-tracking branch 'origin/master' into codex/issue-223-rem…
test(server): one packet-table write guard; drop dead PayloadJSON (#231…