fix: unhang index builds on pathological tokens, and name the file when one hangs

.coldstart/notebook/.raw/how-a-hung-index-build-names-the-file-that-hung-it.jsonl

@@ -0,0 +1 @@
+{"title":"how a hung index build names the file that hung it","summary":"A build that spins the CPU inside one synchronous call cannot report anything through the normal channels — the event loop is dead, so a watchdog timer, a progress interval, a signal handler, and any async write are all permanently unable to run; the keeper log simply stops after its last phase line. The only thing that survives is a record written SYNCHRONOUSLY BEFORE the work starts, which is what markInflight does, and a record that outlives the process that wrote it is itself the diagnosis (readStaleInflight): the indexer began that unit of work and never finished it. The record is cleared on clean completion precisely so its presence means something. It is surfaced three ways — `coldstart status` prints `working on:` for a live pid and `LAST SEEN: ... died during ...` for a dead one, the next keeper logs a WARNING naming the prime suspect at startup, and that event is appended to repair.jsonl as `died-in-progress` because startup is the last moment the evidence exists before the next build overwrites it.","identityAliases":["hung build diagnosis","inflight record","stuck index file","died in progress"],"incidentAliases":["index build hangs silently","log stops at parsing files","which file is stuck"],"steps":[{"path":"src/inflight.ts","symbols":["markInflight","readStaleInflight","clearInflight"],"role":"the record: synchronous write before work, cleared on completion, stale = died here"},{"path":"src/indexer/parser.ts","symbols":["parseFile"],"role":"invokes the mark AFTER its awaits, immediately before the synchronous work"},{"path":"src/index.ts","symbols":["buildIndex","runKeeper"],"role":"marks each phase + hands parseFile the per-file mark; detects and announces a stale record at keeper start"},{"path":"src/status.ts","role":"renders live work vs a record that outlived its process"}],"invariants":["The mark is written synchronously (writeFileSync) — an async write may still be queued when the event loop stops, and queued means lost.","There must be NO await between a mark and the synchronous work it announces, or a concurrent file overwrites it and the record names an innocent file.","A record is cleared on clean completion, so its survival is what distinguishes a death from ordinary progress."],"verified":["src/inflight.ts","src/indexer/parser.ts","src/index.ts","src/status.ts"],"id":"how-a-hung-index-build-names-the-file-that-hung-it","type":"flow","op":"put","v":1,"ts":"2026-08-31T18:33:10.363Z","head":"c78eb6211f36","anchors":[{"path":"src/inflight.ts","hash":"sha256:79d91d4f62ce","head":"c78eb6211f36"},{"path":"src/indexer/parser.ts","hash":"sha256:81674e6509c7","head":"c78eb6211f36"},{"path":"src/index.ts","hash":"sha256:69f50f897b4f","head":"c78eb6211f36"},{"path":"src/status.ts","hash":"sha256:6e76b0cf5c91","head":"c78eb6211f36"}]}

.coldstart/notebook/.raw/how-coldstart-gets-created-and-how-it-silently-never-does.jsonl

@@ -0,0 +1 @@
+{"title":"how ~/.coldstart gets created, and how it silently never does","summary":"The coldstart home (~/.coldstart, or COLDSTART_HOME) is created lazily by READER processes only — there is no background service that recreates it, so after `rm -rf ~/.coldstart` it stays absent through any amount of editing, uninstalling, or reinstalling until a reader command runs. ensureKeeper is called from find/gs/index, non-hook kb commands, init, and MCP startup; the hooks that run during ordinary editing pass --hook and deliberately skip the spawn for latency, so normal work never recreates it. The actual mkdir happens inside tryAcquireSpawnLock (recursive, in the reader process) BEFORE the keeper is spawned, so the directory + daemon/ appear even when the keeper later dies or hangs. Two silent failure modes verified this session: (1) if the home path is not writable (EACCES — e.g. a root-owned remnant, or a coldstart previously run under sudo so homedir() differed), ensureKeeper swallows the error entirely at its catch-all and `find` still exits 0 with a normal-looking answer, the only trace being one `warn: could not save cache: EACCES ... mkdir` line, after which every run rebuilds in-process and persists nothing; (2) a leftover <root>.spawn lock file (reader killed between O_EXCL acquire and release) makes tryAcquireSpawnLock return null forever, so ensureKeeper returns at its `another reader is already spawning it` branch and NO keeper is ever spawned again — recovery is deleting that file, which nothing surfaces.","identityAliases":["coldstart home creation","coldstart cache directory","ensureKeeper spawn path","spawn lock"],"incidentAliases":["coldstart folder not recreated","index never builds","keeper never spawns"],"steps":[{"path":"src/keeper.ts","symbols":["ensureKeeper"],"role":"entry: the only thing that spawns a keeper; swallows every failure best-effort"},{"path":"src/daemon-lock.ts","symbols":["tryAcquireSpawnLock","daemonDir","isDaemonAlive"],"role":"recursive-mkdirs the home, then O_EXCL spawn lock with no staleness check"},{"path":"src/init.ts","role":"calls ensureKeeper then blocks on waitForKeeperCache, so `init` warms the cache at setup"},{"path":"src/constants.ts","symbols":["coldstartHome"],"role":"resolves COLDSTART_HOME override else homedir()/.coldstart, per call"}],"invariants":["Nothing recreates the coldstart home in the background — only a reader command does.","tryAcquireSpawnLock has no staleness check: a leftover .spawn file blocks keeper spawn permanently and silently."],"verified":["src/keeper.ts","src/daemon-lock.ts","src/init.ts","src/constants.ts"],"id":"how-coldstart-gets-created-and-how-it-silently-never-does","type":"flow","op":"put","v":1,"ts":"2026-08-31T18:14:09.468Z","head":"c78eb6211f36","anchors":[{"path":"src/keeper.ts","hash":"sha256:dd3f5cc515f2","head":"c78eb6211f36"},{"path":"src/daemon-lock.ts","hash":"sha256:4d06b20383c3","head":"c78eb6211f36"},{"path":"src/init.ts","hash":"sha256:25e0376d8287","head":"c78eb6211f36"},{"path":"src/constants.ts","hash":"sha256:db1ac57f315e","head":"c78eb6211f36"}]}

.coldstart/notebook/.raw/src-index-ts-c5fb8502.jsonl

@@ -3,3 +3,4 @@
 {"target":{"kind":"alias","key":"--help hangs"},"reason":"fixed — --help is now handled first","id":"src-index-ts-c5fb8502","type":"file","op":"retract","v":1,"ts":"2026-08-02T17:39:02.842Z","head":"bc2fe2ab8c2d"}
 {"aliasesVerified":true,"id":"src-index-ts-c5fb8502","type":"file","op":"put","anchors":[{"path":"src/index.ts","symbols":["main"]}],"verified":[],"v":1,"ts":"2026-08-02T18:19:44.450Z","head":"8d5cc74d3cb9"}
 {"identityAliases":["command dispatch","graph building","index accumulation"],"facets":[{"symbol":"main","detail":"Dispatch order is load-bearing, not stylistic. Everything past the subcommand block is treated as a flag-driven invocation (--daemon / --no-daemon / --probe / the default stdio MCP reader), and the unknown-command guard deliberately rejects only a first arg that does NOT start with '-'. So any bare flag falls through and BOOTS THE MCP READER. Fixed by handling --version/-v and --help/-h first by exact match."},{"symbol":"buildIndex","detail":"Lines 221–223: allSymbolEdgesBuilt = buildSymbolEdges() accumulates into symbolEdges array, then assigned to CodebaseIndex.symbolEdges at line 274. Lines 225–268: isBarrel flag set on TS/JS files (reexportRatio > 0.5 && importedByCount > 1 && exports.length > 0), then domain-map cleaned and transitiveImportedByCount computed for graph traversal. These setup cycle-guard structures: isBarrel + reexportRatio + transitiveImportedByCount are load-bearing for traversal algorithms that need to bypass re-export hubs."}],"id":"src-index-ts-c5fb8502","type":"file","op":"put","anchors":[{"path":"src/index.ts","symbols":["main","buildIndex"],"hash":"sha256:d4b06f304d61","head":"8ec92c1b94cb"}],"verified":["src/index.ts"],"character":"hub","v":1,"ts":"2026-08-18T13:25:59.573Z","head":"8ec92c1b94cb"}
+{"identityAliases":["command hangs","unknown command","MCP reader boots unexpectedly","command dispatch","graph building","index accumulation","keeper startup"],"facets":[{"symbol":"main","detail":"Dispatch order is load-bearing, not stylistic. Everything past the subcommand block is treated as a flag-driven invocation (--daemon / --no-daemon / --probe / the default stdio MCP reader), and the unknown-command guard deliberately rejects only a first arg that does NOT start with '-'. So any bare flag falls through and BOOTS THE MCP READER. Fixed by handling --version/-v and --help/-h first by exact match."},{"symbol":"buildIndex","detail":"allSymbolEdgesBuilt = buildSymbolEdges() accumulates into a symbolEdges array assigned to CodebaseIndex.symbolEdges; isBarrel is set on TS/JS files (reexportRatio > 0.5 && importedByCount > 1 && exports.length > 0), then the domain map is cleaned and transitiveImportedByCount computed. These are cycle-guard structures: isBarrel + reexportRatio + transitiveImportedByCount are load-bearing for traversal that must bypass re-export hubs. It also marks each phase (walk/parse/resolve/graph) in the inflight record and hands parseFile a per-file mark, then clears the record before returning — a record surviving that clear means the process died mid-build.","flows":["how-a-hung-index-build-names-the-file-that-hung-it"]},{"symbol":"runKeeper","detail":"At startup, before any build, it reads the inflight record left by a PREVIOUS process and — if that pid is gone — logs a WARNING naming the file as prime suspect and appends `died-in-progress` to repair.jsonl. This has to happen here because keeper start is the last moment that evidence exists: the build about to run overwrites it.","flows":["how-a-hung-index-build-names-the-file-that-hung-it"]}],"id":"src-index-ts-c5fb8502","type":"file","op":"put","anchors":[{"path":"src/index.ts","symbols":["main","buildIndex","runKeeper"],"hash":"sha256:69f50f897b4f","head":"c78eb6211f36"}],"verified":["src/index.ts"],"character":"hub","v":1,"ts":"2026-08-31T18:33:10.436Z","head":"c78eb6211f36"}

.coldstart/notebook/.raw/src-indexer-content-tokens-ts-21b52eaf.jsonl

@@ -0,0 +1 @@
+{"summary":"Extracts identifier-shaped tokens from file bodies for the Related-files link channel via isShapedToken's SNAKE/CAMEL/PASCAL/ALL_CAPS shape gate. SHAPE_SNAKE/CAMEL/ALL_CAPS previously nested a quantified group inside a trailing `+` (e.g. `(?:[A-Z][a-zA-Z0-9]*)+$`), a catastrophic-backtracking shape that hung `coldstart index` indefinitely on any file containing one long mixed-case run (an embedded hash, base64 blob, or signature) — an 82-char adversarial token alone took 110+ seconds. Fixed by rewriting each as a single-quantifier regex plus a linear trailing-check/substring test (verified mathematically equivalent to the originals against the full test corpus), with MAX_TOKEN_LENGTH=100 kept on top as defense-in-depth; SHAPE_PASCAL was already safe (disjoint character classes) and untouched.","identityAliases":["content token shape gate","isShapedToken","related files extraction"],"incidentAliases":["index rebuild hangs forever","coldstart index freezes","regex catastrophic backtracking"],"anchors":[{"path":"src/indexer/content-tokens.ts","symbols":["isShapedToken","SHAPE_SNAKE","SHAPE_CAMEL","SHAPE_ALL_CAPS","MAX_TOKEN_LENGTH"],"hash":"sha256:dcc9a76b95be","head":"c78eb6211f36"}],"id":"src-indexer-content-tokens-ts-21b52eaf","type":"file","op":"put","verified":["src/indexer/content-tokens.ts"],"character":"single","v":1,"ts":"2026-08-31T16:37:25.402Z","head":"c78eb6211f36"}

.coldstart/notebook/.raw/src-indexer-parser-ts-ddced573.jsonl

@@ -0,0 +1 @@
+{"summary":"parseFile takes an optional `onWork` callback that it invokes immediately before each synchronous section (the tree-sitter parse and the content-token extraction), and it MUST be invoked here rather than at the call site. Verified this session by reproducing a real hang: parseFile awaits a grammar load and a file read, while the caller runs a batch of 100 through Promise.all, so every file in the batch gets past the call site before any of them reaches the CPU-bound work — a mark set out there named an innocent file while a different one span at 100% CPU. The rule the signature depends on is that no await may sit between an onWork call and the work it announces.","identityAliases":["parseFile onWork","parser sync section","batch parse attribution"],"incidentAliases":["inflight names wrong file","mark set at call site"],"anchors":[{"path":"src/indexer/parser.ts","symbols":["parseFile","parseByLanguage"],"hash":"sha256:81674e6509c7","head":"c78eb6211f36"}],"id":"src-indexer-parser-ts-ddced573","type":"file","op":"put","verified":["src/indexer/parser.ts"],"character":"single","v":1,"ts":"2026-08-31T18:33:10.403Z","head":"c78eb6211f36"}

.coldstart/notebook/.raw/src-inflight-ts-668f1959.jsonl

@@ -0,0 +1 @@
+{"summary":"Records what the indexer is working on right now, as the one signal that survives a synchronous CPU spin: markInflight does a synchronous writeFileSync BEFORE each unit of work, because once a pathological input blocks the event loop no timer, signal handler, or async flush can ever run to report it. The record is overwritten in place (it answers 'what now', not 'what happened') and cleared by clearInflight on clean completion, which is what makes readStaleInflight meaningful — a record whose pid is no longer alive names the exact unit of work that killed the process. Per-file granularity was a measured choice: ~0.04ms/file, 86ms across a 2000-file build, under 3% of a build that takes seconds, in exchange for the exact path instead of a 100-file window to bisect by hand.","identityAliases":["inflight record","current file being parsed","markInflight","stale inflight"],"incidentAliases":["hung build names no file","silent index hang"],"anchors":[{"path":"src/inflight.ts","symbols":["markInflight","clearInflight","readInflight","readStaleInflight","describeInflight","inflightPath"],"hash":"sha256:79d91d4f62ce","head":"c78eb6211f36"}],"id":"src-inflight-ts-668f1959","type":"file","op":"put","verified":["src/inflight.ts"],"character":"single","v":1,"ts":"2026-08-31T18:33:10.393Z","head":"c78eb6211f36"}

.coldstart/notebook/.raw/src-keeper-state-ts-cc715659.jsonl

@@ -0,0 +1 @@
+{"summary":"Holds keeper-state.json (pid, startedAt, inProgress, last reconcile/patch/rebuild/save) and repair.jsonl beside the cache, all writes best-effort so observability can never take the keeper down. The load-bearing contract is on KeeperWork: a lingering inProgress stamp from a keeper that died mid-rebuild is declared harmless BECAUSE readers check the pid is alive before believing it — a rule waitForCacheHead honoured while status.ts did not, which is how a killed keeper produced a permanent 'IN PROGRESS'. RepairEvent's event union is closed and gained `died-in-progress` for a previous indexer that vanished mid-work (see the inflight flow).","identityAliases":["keeper state file","repair log","KeeperWork contract","inProgress stamp"],"anchors":[{"path":"src/keeper-state.ts","symbols":["KeeperState","KeeperWork","RepairEvent","updateKeeperState","readKeeperState","appendRepairLog"],"hash":"sha256:b3c119bd07c6","head":"c78eb6211f36"}],"id":"src-keeper-state-ts-cc715659","type":"file","op":"put","verified":["src/keeper-state.ts"],"character":"single","v":1,"ts":"2026-08-31T18:33:10.414Z","head":"c78eb6211f36"}

.coldstart/notebook/.raw/src-status-ts-8c69c9ca.jsonl

@@ -3,3 +3,5 @@
 {"target":{"kind":"alias","key":"had to uninstall and rewire"},"reason":"inaccurate — the documented remedy is `coldstart restart`/`coldstart init`, not uninstall+rewire","id":"src-status-ts-8c69c9ca","type":"file","op":"retract","v":1,"ts":"2026-08-02T17:44:09.823Z","head":"bc2fe2ab8c2d"}
 {"aliasesVerified":true,"id":"src-status-ts-8c69c9ca","type":"file","op":"put","anchors":[{"path":"src/status.ts","symbols":["runStatus","wiredHookPaths","indexFreshness","deriveRoot","relativeAge","fileSize"]}],"verified":[],"v":1,"ts":"2026-08-02T18:19:44.586Z","head":"8d5cc74d3cb9"}
 {"summary":"Renders `coldstart status`: a network-free table of every keeper (liveness via lockfile PID check, index freshness from cache meta.json mtime, keeper-state.json/repair.jsonl activity). The install-health section diagnoses the 'npm update didn't take' class — it compares each keeper's lock version to getCurrentVersion() and flags drift with a `coldstart restart` remedy, and via wiredHookPaths() scans .claude/settings.json as RAW TEXT (structure-agnostic regex for /…/hooks/<name>.mjs) to flag dangling wired hook paths with a `coldstart init` remedy. The per-root activity block also prints a moves-together line (2026-08-23) for a specific reason: co-change is silent on shallow clones and non-git dirs, and without a line saying how many commits were scanned that silence is indistinguishable from a bug.","identityAliases":["npm update didn't take","keeper running old version","install health","wired hook path missing","version drift","moves-together not derived"],"incidentAliases":["gs shows no moves together","co-change silent shallow clone"],"anchors":[{"path":"src/status.ts","symbols":["runStatus","wiredHookPaths","indexFreshness","deriveRoot","relativeAge","fileSize"],"hash":"sha256:d9ea936f4f43","head":"8ec92c1b94cb"}],"id":"src-status-ts-8c69c9ca","type":"file","op":"put","verified":["src/status.ts"],"character":"single","v":1,"ts":"2026-08-22T19:02:12.589Z","head":"8ec92c1b94cb"}
+{"summary":"Renders keeper liveness/freshness per root from keeper-state.json + repair.jsonl beside the cache. Its inProgress stamp is now gated on isDaemonAlive(state.pid): KeeperWork's own contract in keeper-state.ts says a lingering stamp from a keeper that died mid-rebuild is harmless BECAUSE readers check the pid first, and waitForCacheHead in keeper.ts honoured that while this renderer did not — so a killed keeper left a permanent 'IN PROGRESS: rebuild' that status reported as current forever. Verified this session that the catastrophic-backtracking parse hang writes exactly that stamp and nothing else, so killing the stuck keeper (the only recovery) was guaranteed to produce the stale reading during the very incident status is used to diagnose.","identityAliases":["coldstart status command","keeper freshness output","IN PROGRESS stamp"],"incidentAliases":["status shows stale in progress","rebuild stuck forever"],"anchors":[{"path":"src/status.ts","symbols":["runStatus","wiredHookPaths","indexFreshness","deriveRoot","relativeAge","fileSize","renderStatus","stampLine"],"hash":"sha256:26d79b2192ab","head":"c78eb6211f36"}],"id":"src-status-ts-8c69c9ca","type":"file","op":"put","verified":["src/status.ts"],"character":"single","v":1,"ts":"2026-08-31T18:14:09.497Z","head":"c78eb6211f36"}
+{"summary":"Renders keeper liveness/freshness per root from keeper-state.json + repair.jsonl + the inflight record beside the cache. Two pid-liveness rules live here and both were added after being wrong: the inProgress stamp is gated on isDaemonAlive(state.pid), because KeeperWork's contract in keeper-state.ts says a stamp outliving its keeper is harmless only if readers check the pid first (waitForCacheHead did; this renderer did not, so a killed keeper reported 'IN PROGRESS' forever); and the inflight line reads the opposite way on purpose — a live pid prints `working on: <phase> <file>`, while a record whose pid is GONE prints `LAST SEEN: PID N died during <phase> <file>`, which is the whole diagnostic for a build that hung and had to be killed.","identityAliases":["coldstart status command","keeper freshness output","IN PROGRESS stamp","LAST SEEN line"],"incidentAliases":["status shows stale in progress","status cannot explain a hang"],"anchors":[{"path":"src/status.ts","symbols":["runStatus","wiredHookPaths","indexFreshness","deriveRoot","relativeAge","fileSize","renderStatus","stampLine"],"hash":"sha256:6e76b0cf5c91","head":"c78eb6211f36"}],"id":"src-status-ts-8c69c9ca","type":"file","op":"put","verified":["src/status.ts"],"character":"single","v":1,"ts":"2026-08-31T18:33:10.424Z","head":"c78eb6211f36"}