Make gortex daemon stop finish in seconds without SIGKILL: tolerant teardown, bounded close

Why

Every graceful stop in the benchmarks gets force-killed. gortex daemon stop takes 18.1-19.5 s and prints "graceful shutdown timed out — force-killing", even on an idle restart that wrote 0 bytes. After a successful ack, the CLI waits daemonExitGrace (15 s) for the PID to go away. The process never exits in that window, so the CLI kills it. The known contributors:

On Unix, ProcessAlive uses kill(pid, 0), and that call reports a zombie as alive. Some parents do not reap the daemon right away: a test harness, an IDE task runner, a Popen supervisor. In that case the daemon has already exited, but the CLI still waits out the whole grace and then SIGKILLs a dead process.
The contract-analysis close has a 30 s budget. If it fails or times out, teardown returns before the store close. The daemon keeps running with the store open and the lock held, and the CLI reports "shutdown rejected". This regression is on main only and came in with Improve incremental indexing and resumable contract analysis #806.
After a cold index, the store's final close runs a TRUNCATE checkpoint whose published estimate reached about 16 s. When the last SQLite connection closes, SQLite runs that checkpoint again, and nothing can interrupt it.
An idle stop also gets killed, so part of the time goes after the store close. That points at teardown steps after Serve returns that have no bound.

A stop that takes 18 s and ends in a SIGKILL slows every restart and upgrade. A stranded daemon blocks the next daemon start until someone kills it by hand.

Current state

cmd/gortex/daemon.go runDaemonStart: the closeShared closure passed to installDaemonTeardown calls contractAnalysis.Close with a 30 s context. It returns that error before state.shared.Close(). newDaemonTeardown runs stopWatcher and then closeShared once under a sync.Once, with no per-step timing or bound. After Serve returns, the deferred calls unwind with no bound: retirementWorker.Stop, runTeardown, the reconcile janitor and the health broadcaster.
internal/daemon/server.go: in the ControlShutdown case, an error from Controller.Shutdown returns controlErr(ErrInternal, ...). The connection loop schedules go s.Shutdown() only when req.Kind == ControlShutdown && resp.OK. A teardown error therefore leaves the daemon running.
cmd/gortex/daemon.go waitForDaemonExitWithin polls the daemonProcessAlive seam (platform.ProcessAlive). It detects exit only when the PID disappears, with daemonAckedExitGrace = daemonExitGrace = 15 s after an ack. There is no store-lock probe and no goroutine dump.
internal/platform/platform_unix.go ProcessAlive is syscall.Kill(pid, 0) == nil. PR fix(platform): treat EPERM from kill(pid, 0) as a live process #978 added an EPERM-tolerant ProcessExists and left ProcessAlive strict, so liveness is still not zombie-aware. A zombie-liveness fix is planned separately for v0.64.11. If it has not merged when this work starts, do it here.
internal/graph/store_sqlite/store.go Close → wal_close.go closeCheckpointWAL runs a final wal_checkpoint(TRUNCATE). By default it has no deadline: closeCheckpointDeadline applies one only when GORTEX_SQLITE_CLOSE_CHECKPOINT_MAX_S is set. Its comment notes that SQLite's close of the last connection then runs the full checkpoint anyway. The driver is modernc.org/sqlite.
internal/indexer/checkout_lifecycle.go CheckoutLifecycle.Close calls coordinator.Close() on each registered coordinator in turn, in a sequential loop.

What to do

Failure-tolerant teardown. Each teardown step logs its error and the sequence continues. A contract-close failure never skips the store close. ControlShutdown answers OK with warnings, and Shutdown() is always scheduled. As a result the store closes, the flock is released, and the socket, PID and runtime-state files are removed. The once-guarded error is still returned to later callers for reporting.
Exit detection by "store free". Before sending the stop, read the store path from the runtime state. The exit wait succeeds when a non-blocking TryLock on <store>.lock succeeds, and the probe releases the lock within the same poll. Process liveness only decides whether a kill is needed. If the lock is free and the PID is still visible, print "store free, process not yet reaped by parent", with no SIGKILL and no "timed out" line. Add a daemonStoreFree seam next to daemonProcessAlive / daemonKillProcess. A daemon without an on-disk store falls back to waiting for the socket and PID file to be removed.
Zombie-aware liveness, if it has not merged yet: on darwin, sysctl kern.proc.pid with p_stat == SZOMB; on linux, /proc/<pid>/stat state Z; other Unixes keep kill(pid, 0).
Bounded stop order. Log each phase with its duration: watcher stop, contract close, stack close, CheckoutLifecycle.Close, and each deferred call. Bound each post-close step at 2 s: janitor, health broadcaster, retirement worker, LSP child reaping. Cancel the warmup goroutine's context at shutdown. If the timing shows coordinator joins dominate, cancel all coordinator lifetimes in parallel and then join them.
Exit watchdog. It may run only after the shared store close has returned successfully and the PID, socket and runtime-state files are gone. Then it writes a goroutine dump (runtime/pprof "goroutine", debug=2) to the daemon log, flushes the log and calls os.Exit(0) within 3 s. The dump shows which join blocked, so it can be fixed. Separately, add an opt-in SIGQUIT dump when the CLI grace expires (GORTEX_STOP_DUMP=1).
Skip or bound the close-time TRUNCATE checkpoint. The WAL is durable and gets recovered on the next open, so the close does not have to drain it. By default, either skip the final TRUNCATE or cap it with a short deadline. Also stop SQLite from rerunning the checkpoint on the last connection close, which cannot be interrupted: use SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE through the driver's connection hook, or an equivalent. Keep today's full drain as an opt-in. Check that the next open replays a large WAL correctly and still meets the warm-restart budget.
Fix the bench and validation harness so it reaps the daemon in a background thread or starts it with --detach. Before changing the product, capture ps -o stat= during one stop to confirm the zombie diagnosis.

Out of scope

Steady-state passive checkpoint and WAL reclaim tuning, including keeping the close-checkpoint estimate under 2 s at steady state. That is the WAL work in WAL checkpoint path appears to stall on large stores, forcing full re-index far below documented throughput #843. The open PR fix(store_sqlite): grow the passive WAL checkpoint window after execute-phase timeouts #950 targets code that main has since replaced.
Draining in-flight MCP handlers and refusing new calls with a daemon_stopping error during stop.
Windows liveness, which already checks STILL_ACTIVE.
Offline gortex untrack after a stop.

Done when

How to verify

Run go test -race ./cmd/gortex/ -run 'Teardown|Stop|Watchdog', go test -race ./internal/platform/ ./internal/indexer/ -run 'Zombie|LockProbe|LifecycleClose' and go test -race ./internal/graph/store_sqlite/ -run Close.
On the reference cgroup host, run the bench harness for: 20 idle stops; a stop right after a cold index; a stop right after the 60-minute edit churn run; stops with 6 repos, 29 repos, 16 worktrees and 32 worktree coordinators. Record stop wall time, SIGKILL count and the CLI and daemon log lines. Repeat the idle run once with the harness parent deliberately not reaping, to exercise the zombie path.
Grep the daemon log for the per-phase timing lines. Confirm the store-close line comes ≤2 s before the CLI returns and that no goroutine dump was written.

Dependencies

Blocked by (must land first):

Treat zombie processes as dead in ProcessAlive so daemon stop stops force-killing #995 — The stop work builds on the zombie-as-dead liveness fix.
Blocks:
Test and document the runaway-warmup escape: daemon stop, then offline untrack #1029 (needs this)

Related issues

Planning

Milestone: v0.64.12
Priority: latency (first priority)
Estimate: ~5.5 engineer-days

Starting this issue in a separate worktree; #995 is closed and its zombie-liveness fix is already on main. The project item is now In Progress. Implementation will use one PR with atomic commits, with pinned formatting/lint and relevant automated checks before pushing.

A fresh local observation is relevant to the teardown work: during a real installed-client acceptance case on source 186e1c160759e0a1954246ffdc36612748199067, the private daemon took 49.447 seconds to drain after SIGTERM while its repository was still loading. It exited within the existing five-minute allowance without a forced kill; all recorded owned processes/groups and socket/PID files were gone, and the existing store lock was independently reacquired. The final checkpoint reported zero remaining WAL bytes, followed by asynchronous warmup sql: database is closed errors during teardown.

This is a direct-signal observation, not a measurement of gortex daemon stop, a reference-cgroup result, proof of a zombie, or proof of which teardown phase caused the delay. The raw log and cleanup receipts are retained locally with client acceptance case c3. The source review will distinguish already-landed liveness work from remaining cancellation, phase ordering, close/checkpoint and post-close join work; the issue's actual stop/lock/reopen gates still need to be run.

Pre-change baseline captured on clean main 24ce36c4a, using the actual gortex daemon stop CLI and three fresh private stores on the native Apple M5 Pro host:

Case	CLI stop	First successful symbol query after observed restart readiness
Small repository, reaped parent	190 ms	71 ms
Small repository, deliberately unreaped parent	193 ms	70 ms
gortex repository while loading	50.060 s	98 ms

The unreaped case includes a matching native ps zombie witness with the existing store lock already free, before the parent called Wait. The earlier zombie fix is working in this capture.

The loading capture reproduces a remaining problem: startup continued parsing/indexing for 49.804 seconds after stop was launched, then attempted resolve/derived work and watcher startup after the database closed. The final explicit TRUNCATE logged 1 ms in this capture. This chronology does not assign the full delay to a particular internal close/join phase; phase timing is needed in the fix.

All three immediate restarts succeeded. Every daemon exited successfully; no actual force-kill message or controller cleanup signal occurred. Recorded processes/groups were gone, runtime files were removed, and each actual existing store lock was independently acquired and released without creating a replacement lock file. Raw output, stores, build/helper pins, and cleanup receipts are retained locally.

These are single native diagnostic captures, not the repeated reference-profile acceptance matrix. The loading latency target remains unmet. Implementation is starting with cancellation of daemon-owned startup work and a race-safe watcher handoff; the remaining teardown and acceptance criteria stay open.

The shutdown implementation is progressing in an isolated worktree through small commits; the current committed head is 15350502d7fa44564ac05985581e63b3eb444763.

Startup, watcher, checkout and contract producers now receive cancellation before their actual joins. A timeout remains an undrained warning, rather than permission to close the backend while a producer still owns it.
LSP shutdown sends the protocol shutdown/exit sequence, interrupts blocked owned-pipe writes, and reaps the exact owned child once. Concurrent callers share the real completion/error. Passive IDE connections only disconnect.
Provider/router shutdown now closes admission permanently, captures initializing clients before the initialize RPC, prevents late publication/reconnect, and joins constructors, duplicate/eviction closers and reapers. All captured providers begin closing before any one is joined. The LRU logging callback now runs outside the router lock.

The latest focused race census passed all 67 tests; seven new ownership cases also passed three repetitions. Actual child-wait receipts matched all 54 recorded helpers in that validation, with no surviving owned test processes. Earlier focused producer/client slices also passed. A test-only compile failure was retained and corrected; formatting/diff checks pass. Full pinned pre-push lint and integrated daemon acceptance are still pending, so no PR or push is claimed yet.

Next is connecting the semantic manager's cancellation and real completion to the shared server, before backend close. Co-change ownership, SQLite close/checkpoint handling, CLI lock/runtime proof and the final native shutdown matrix remain unfinished. The previously measured ~50-second loading shutdown remains the baseline; these unit results do not establish the final shutdown-time target.

Shutdown implementation progress (not acceptance yet):

Background analysis and co-change mining now have owned cancellation and joins. SQLite analysis writes and checked batches observe cancellation while waiting for the writer gate, connection pool and SQL work. They retain the physical transaction until rollback and connection release actually finish, before another writer can enter. A canceled batch also removes its temporary builtin admission, so a later retry can materialize the stub correctly.

Validation on the isolated issue branch:

Analysis writes at cd5ef5213: 38 focused race tests passed, plus eight new controls repeated three times.
Checked batches at a5e5f766b: 58 focused race tests passed, plus six new controls repeated three times; zero skips. These cover actual writes followed by rollback, a competing writer, pool/statement cancellation, JSONB/placeholder parity, receipts, counters and selected-generation bulk connections.
The initial checked-batch compilation failure (missing import) is retained in the local evidence and fixed in its own commit; the successful results belong to the corrected head.

This still does not establish the shutdown time targets. Native immediate BEGIN can wait for SQLite's busy timeout after entry; completed commits, individual encoding work and bulk index sealing still have to finish. Disabling SQLite's implicit close checkpoint is the next implementation step. Explicit close policy, CLI store-lock exit detection, watchdog behavior and the full native/reference-host acceptance runs remain pending. Full pinned lint and final required checks will run before push.

The issue stays In Progress.

Shutdown work has progressed through the worker-ownership and CLI prerequisites. The default close policy now skips the final WAL checkpoint; real-store tests verify committed data survives reopening and the explicit full-drain option still works. Background health, janitor and telemetry owners are cancelled and joined before backend teardown; admitted reconciliation remains owned until it actually finishes.

Focused validation passed: 48 worker tests plus 18 repeated concurrency executions; native process-incarnation controls plus Linux/Windows platform cross-builds; and 10 lock/ACK tests plus 6 repeated concurrency executions. The lock probe uses the captured existing file identity, releases ownership within each poll, and rejects missing/replaced/symlink/error cases. A real socket test reads the shutdown ACK after runtime removal, allowing removal of the fixed 100 ms delay.

CLI integration and the post-close watchdog are still in progress. These are local correctness results, not the required reference-host stop-time, warm-reopen or cross-platform runtime acceptance; the issue remains In Progress.

The captured-owner CLI stop path is implemented and its focused Darwin checks passed at 691a5f9f7: 59 meaningful tests plus two helper entrypoints, followed by 27 executions of nine concurrency cases. There were no skips or failures in the corrected batch; all 43 actual helper children were reaped.

The tests exercise an actual released store lock with a still-visible process, already-reaped stale PID cleanup, a lock released during the final termination checks, failed/ineffective signals, replaced runtime files, and a real legacy-schema control endpoint whose matching PID probe permits the intended fallback shutdown while an unrelated child survives. New runtime records carry native birth identity; an unreachable older PID-only record without positive ownership evidence now returns an explicit error rather than risking a signal to a reused PID. Normal legacy control-socket shutdown remains supported.

The advisory probe has one dial/handshake/request deadline. Skip-close policy avoids the expensive status query, and a failed probe does not trigger a second decorative status attempt. The actual shutdown acknowledgement remains readable after runtime cleanup.

The first batch's fixture failures are retained separately: the real server stamps its own uptime, and the child test sandbox replaces inherited runtime paths. The correction changed only test expectations and explicit owned fixture-path transfer; it did not bypass sandbox isolation or change production behavior.

Still open: cached physical runtime-cleanup results, the post-store/runtime exit watchdog, the optional diagnostic signal, full pre-push checks and native/reference acceptance. These focused tests do not establish the 20-stop, churn, large-workspace or overall stop-latency gates. Linux boot-identity and Windows runtime checks also remain separate from this Darwin result.

Four new six-repository shutdown observations (M09–M12) passed at b061d65b1a43a9f8db0be1a9238a133cf7f01fa5 in #1161.

Row	Complete stop	Backend close → CLI return	Matching query after ready
M09	907.837 ms	51.680 ms	156.668 ms
M10	103.400 ms	58.456 ms	149.498 ms
M11	118.124 ms	59.523 ms	150.636 ms
M12	111.983 ms	52.586 ms	150.304 ms

The first observation used a fresh six-repository store after all six exact roots were ready and loaded, global enrichment completed, and the existing contract/co-change/quiet conditions passed. The three following observations used immediate same-store warm reopens. Each observation passed the five-second complete-stop limit, the broader two-second post-backend limit, and the 1.5-second matching-query limit. Actual CLI endpoints and unique successful backend events were independently checked with integer nanosecond arithmetic.

The existing 8-CPU / 16-GiB memory / 8-GiB swap-allowance profile was unchanged. There were no force-kills, watchdog dumps, OOM kills or product retries, and no swap was used. Memory peaked at 16,710,320,128 bytes. The initial non-root administrative invocation failed its lock-permission check before starting the helper or product; that failure is preserved separately.

Raw and full-runtime archives are hash-pinned. The independent runtime monitor recorded 10,141 lock observations with no monitor error. Owned processes, groups, cgroups, sockets, PID/state files and handles are absent; the frozen seed verifier passed, the active cache is absent, and the VM stopped normally. Public completion/quiet observations do not claim that every internal worker was idle.

The exact issue-prescribed selected race batches also passed at this head: 117 top-level executions, zero skips. Hosted CI is terminal green: 24 successful checks and two intentional skips.

These are four additional logical rows. The ledger now has passing observations for 29 of 48 unique rows across their recorded source versions; old observations are not relabeled as b061 runs, and the original M07 failure remains recorded. The 19 unmeasured rows are full29 (4), 16-worktree (4), 32-worktree (10), and churn-60 (1). This is not full issue acceptance: keep #1022 In Progress and #1161 draft.

Worktree shutdown validation passed on source b061d65b1a43a9f8db0be1a9238a133cf7f01fa5 (the build's unchanged wire version is v0.64.11): four consecutive observations with 16 live linked coordinators and ten with 32. The primary checkout was additional in both cases. Every counted stop had an exact checkout-identity join to the registry-backed live census.

Across these 14 observations:

Maximum complete stop: 244.215 ms (limit 5 s).
Maximum backend-close log to actual CLI return: 60.527 ms (limit 2 s).
Maximum first matching query after observed restart readiness: 129.013 ms (limit 1.5 s).
No forced kills, watchdog dumps or OOM kills; every same-store restart and cleanup check passed.

The 8-CPU/16-GiB workload profile was unchanged. The 16-worktree cohort reached the memory limit and recorded 4,404 memory-max events; swap usage and OOM kills remained zero. This pressure is retained in the evidence.

An earlier fixture attempt passed M17, then stopped before M18 because a query of a persisted ready route did not activate a dormant coordinator. That failed preparation is preserved. The corrected local fixture explicitly requests the existing public checkout reindex operation and waits for the same strict live census. No product code changed; the repeated M17 receives no extra logical credit.

Cumulative coverage is now 43/48 distinct rows with passing observations across the recorded source versions. M13–M16 (full29) and M47 (the genuine one-hour churn run) remain. This issue and PR #1161 remain in progress; full acceptance is pending.

Raw evidence archive: 9,046,889 bytes, SHA-256 3edcc8beaa0b69ed11d5612d0a5b7896bebf245ecf3fdccccde3728808f486b0.
Full owned-runtime archive: 1,952,313,125 bytes, SHA-256 e863b07b90e7e9f2d0946028e6c724f8b49362198908e84d9eedabd3ee33bc24.
The raw timing intervals and live-identity censuses were independently reviewed. Seed/cache/process/lock/runtime-file audits passed, owned runtime copies were archived and retired, and the VM stopped normally with no remaining image writers.

The genuine one-hour edit-churn observation (M47) passed on b061d65b1a43a9f8db0be1a9238a133cf7f01fa5, using the same pinned Linux artifact as the recent shutdown observations. Its reported wire version remains v0.64.11; source attribution is recorded separately.

Offered workload: 3,600.003 seconds, 581 edits and 3,498 attempts; all edits became visible, with no visibility-censored edits or workload errors.
Complete shutdown: 303.580 ms. Backend-close completion to actual CLI return: 60.276 ms.
The stop started 1.496 ms after client quiescence, with no post-churn enrichment wait.
Immediate same-store restart succeeded. The first matching graph query completed 110.400 ms after observed readiness; a separate query found the final acknowledged edit marker in 35.023 ms. The helper verified all 581 acknowledged markers in the reopened owned source.
No force-kill, watchdog dump or OOM. The owned cgroup peaked at 14,728,134,656 bytes with zero swap and zero memory-cap events. The VM had 80 KiB of unrelated residual swap usage.
Process, group, cgroup, lock, runtime-file and handle cleanup checks passed; 75,523 independent lock observations had no monitor errors. The frozen compiler seed was verified, owned runtime data was archived, and the VM stopped normally.

Root review independently recomputed the raw backend-to-CLI and quiescent-to-stop intervals, checked the full-hour timestamps, 581 committed ACKs, final marker query and successful daemon exits. Raw archive SHA-256: c8bda6054a1eed07790b28c2299a439f436ac00648f4dc01eae4cd7a0716c361 (5,100,398 bytes). Full runtime archive SHA-256: 58f319ce8727f4a517eff834c7846ac5670e609f8abb851311813016c9196263 (413,827,389 bytes). Local receipts and raw evidence remain outside the repository.

This adds one unique row: 44/48 shutdown scenarios now have passing observations across their recorded source versions. Only M13–M16, the four full-workspace observations, remain. Their isolated source export is in progress. This issue and PR #1161 remain open/In Progress until those results and overall acceptance are reviewed; this is not milestone acceptance.

Full-workspace preparation update: the isolated host export is verified, and guest staging is next. The four native observations M13–M16 remain unrun; the shutdown ledger is still 44/48.

The export preserves all 29 configured repositories and four original linked worktrees. It contains 67,951 selected primary files (7,741,014,632 bytes), 9,774 linked files (96,993,688 bytes), the copied Git metadata and one retained internal symlink. Omissions follow the verified product root-ignore policy; policy/control files required to preserve that behavior are retained. All selected primary bytes match their historical SHA-256 manifests. Linked files are checked against their recorded Git identities, with fresh byte hashes recorded separately.

The first export attempt stopped on a PowerShell file whose required CRLF checkout representation differs from Git's canonical LF blob. The failed attempt is retained. The corrected exporter accepts only the two exact qualified entries, preserves their original bytes and requires both their exact raw SHA-256 and canonical Git identity. Its focused race controls, vet, formatting and build passed; the fresh export then passed in 650.33 seconds. Every other content check is unchanged.

No source original was modified and no product code changed. Guest staging will rebind copied Git paths before Git/product use, verify the transferred files and check the actual disk budget before the fixed four-observation run. This preparation result is not native runtime or issue acceptance.

The first full-workspace attempt is archived and cleaned up. It was canceled at the disk-availability cutoff during preparation, with 5,863,698,432 bytes free, before any M13–M16 shutdown observation. The ledger remains 44/48.

All 29 declared roots were ready and loaded before cancellation; global enrichment remained active. Post-abort allocation was 18,866,868,224 bytes for the runtime, dominated by store.sqlite (7,159,537,664 bytes) and its WAL (11,700,561,072 bytes). These are post-abort sizes, not an exact cutoff-time sample. The cgroup reached its 16-GiB memory cap with no OOM kill or swap use.

The helper had been waiting for global quiet before capturing its store identity, so cancellation cleanup used TERM to the verified owned daemon group. No CLI stop-latency observation was made. Independent auditing found no remaining owned processes/groups or held store lock; runtime/input/cache handle audits passed. After archiving and verifying the immutable compiler seed, only the owned copies were retired. The VM stopped normally, with unchanged configuration/image pins and no image writers.

Retained private archives:

Raw evidence: 390,622 bytes; SHA-256 d4a4e1370ad016d26e73a107be4b2b80003c6ab1a3497f5eabd94bc31a13c89f.
Full owned state: 6,931,954,061 bytes; SHA-256 b94df2fcf965f8d8d992fba7a9ae4124224ebdca6fd90f92d2af5fc3e38d5112.

The issue's full29 workspace scenario does not require global enrichment or contract quiet. The next run will therefore use exact readiness/loading of all 29 roots, capture the store identity at that point, and record the actual enrichment state. Initial-index proof, same-store restarts, the 5-second complete-stop / 2-second post-backend / 1.5-second graph-call limits, and ownership/lock/no-force/no-watchdog checks remain required. The six-repository scenario is unchanged.

The focused readiness control passed, including a ready-but-still-enriching case and refusal of a foreign PID. Fresh staging and the four observations are next. Product source remains b061d65b1a43a9f8db0be1a9238a133cf7f01fa5; there is no filesystem resize, source omission or automatic retry.

The 29-repository run found a product failure at b061d65b1a43a9f8db0be1a9238a133cf7f01fa5. M13 took 9.026 seconds for complete stop against the 5-second limit. All 29 declared roots were ready and loaded before stop; background enrichment was still running, which is recorded explicitly. M14–M16 were not run after the failure.

The warmup join took 8.926 seconds. Raw daemon logs show the master resolver returning context canceled, followed by another 8.042 seconds of test-edge reconciliation (1,454 files / 51,628 edges). Backend close itself took 14.369 ms, and backend-close-to-CLI return was about 53.556 ms. The identified cancellation gap is in the retargeted-test projection following resolution.

The fix and regression controls are being validated in this PR: skip that projection when cancellation is already observed, and check cancellation between complete file groups while keeping each graph eviction/replacement together. Shutdown still joins its owners before closing their backend. The restart repair behavior is under review before publishing the new head.

There was no forced kill, watchdog dump, OOM or capacity abort. Independent process/runtime/lock cleanup checks passed; the frozen seed was verified, owned working copies were retired, and the VM stopped normally. Raw logs and relevant runtime/store/WAL evidence are retained locally. Raw archive SHA-256: 20c112951efa589520273fc26841135699a44c4fe990fd156fe8242fbd70d747; runtime archive SHA-256: 25788a1c9df152de6e2e020d6b37bd021b96450dfde86d5c8fd6a4398ce9e5f0.

The ledger remains 44/48 distinct rows with passing observations across their recorded source versions. This failed M13 attempt adds no passing credit. #1022 stays In Progress and PR #1161 stays draft pending the fix, fresh validation and acceptance review.

The cancellation fix is pushed to PR #1161 at 5e223d667e9eca8d23e07f51787ddf21883362f1.

Canceled warmup no longer starts the retargeted-test projection, and cancellation during projection stops between complete file groups. Interrupted files are recorded durably and repaired before readiness on an unchanged restart. The real SQLite close/reopen regression also exercises failed publication: pending work survives, startup does not report completion, and the following restart repairs it. Successful ordinary resolution adds no pending-work writes.

The focused race controls passed, including shutdown ownership and cross-group test-edge correctness. Pinned Go 1.27.3 / golangci-lint 2.14.0, formatting, affected vet, CGO/workflow guards, tidy and module verification passed before push. Earlier fixture setup/lint failures are retained with their fixes; they are not counted as passing runs.

Fresh Linux full-workspace validation is next. The previous 9.026-second M13 failure remains recorded; 512 files bounds each work group, not its wall time. The startup metadata census cost will also be measured. No new passing matrix rows are claimed yet, and #1022 remains In Progress.

The four full-workspace observations passed at 5e223d667e9eca8d23e07f51787ddf21883362f1 with the same input, readiness conditions and limits as the failed run.

Row	Complete stop	Backend close → CLI return	Matching graph call after ready
M13	694.117 ms	57.142 ms	324.658 ms
M14	3,221.588 ms	53.288 ms	137.849 ms
M15	3,440.358 ms	53.534 ms	138.747 ms
M16	3,465.815 ms	60.603 ms	138.469 ms

Limits remain 5 s / 2 s / 1.5 s respectively. All 29 declared roots were ready and loaded before each stop; enrichment was still running. Every daemon exited normally, every same-store restart succeeded, and every query returned the expected indexed symbol. The previous M13 result of 9.026 s remains retained as a failure, not overwritten.

The new startup phase took 10.554 s on the first restart to repair interrupted work before readiness. Its initial clean census took 115.905 ms; subsequent clean censuses took 96.919–99.721 ms. That startup cost is explicit and is not part of the graph-call-after-ready measurements.

Independent raw-log/actual-CLI timing review passed. No forced kill, watchdog dump, OOM or capacity abort occurred. Peak memory was 14,581,719,040 bytes; swap was zero. All process/runtime/lock checks passed (6,127 lock observations, no monitor errors), the frozen seed was verified, owned copies were retired, and the VM stopped normally. Raw and relevant runtime/store/WAL archives are retained locally, with SHA-256 916fc417dd760dfb9ffcbe92f1a8f5e752c710d0cacf1485f9a135d470a5095f and 03c1a2ac6ed8810e6af0107985656b116fa8fa6be447fc7912b6e95fefdc7803.

The ledger now has passing observations for 48/48 distinct rows across their recorded source versions. This is not a claim that every row was rerun at this head. #1022 remains In Progress: current CI still needs the capability-checklist entry and relay-fixture cleanup correction. Both fixes stay in PR #1161; #1175 was closed as duplicate tracking after correcting its initial scope assessment. Acceptance and merge wait for green final-head CI.

The remaining CI failure on the previous head was in the catalog test fixture: it could finish a direct-controller assertion and shut down before the Serve goroutine entered. The fixture now completes a bounded real control handshake before returning. Shutdown's lifecycle guard remains unchanged.

Pushed ffbec2b21bde2c3ca991aed6e2847b2870424bfc after 30 race repetitions of the failing control, 15 neighboring top-level passes and 18 subtest passes, with zero failures/skips. Pinned lint, formatting, vet, CGO/workflow checks, module tidy and module verification passed. The previous head's other substantive CI jobs passed; the new head still requires its own green CI before merge.

This correction changes only a test file. Product source remains the measured 5e223d667, so the completed native evidence retains its existing attribution. The issue remains In Progress until final CI and merge.

Acceptance complete after the final required CI checks and PR #1161 merge.

The native ledger contains passing evidence for all 48 distinct required observations, with source attribution retained for each run.
The final product implementation at 5e223d667 passed the four 29-repository observations: complete stop 0.694–3.466 s; backend close to CLI return 53.288–60.603 ms; matching graph calls 137.849–324.658 ms after readiness. The later commits corrected test fixtures only.
Earlier evidence covers the one-hour churn run, live worktrees, repair and six-repository scenarios, with actual store-lock release, normal process exit and post-restart queries. Recorded failures remain visible alongside the fixes and passing follow-ups.
Interrupted test-edge projection is retained as repair debt and repaired before readiness. The first affected full-workspace restart needed 10.554 s of repair; clean census work took 96.919–115.905 ms. This startup cost remains explicit for Prove startup repo_loading across warm restart, cold start and upgrade, and close the gaps #1023.
Required correctness tests, pinned lint/formatting/build checks and final hosted CI passed. No native row is being relabeled as a final-head rerun.

The issue is closed and moved to Done. #1029 can now finish its own main-targeted CI and acceptance.

fix(daemon): cancel and join startup before backend teardown

Navigation Menu

Repository navigation

Make `gortex daemon stop` finish in seconds without SIGKILL: tolerant teardown, bounded close #1022

Why

Current state

What to do

Out of scope

Done when

How to verify

Dependencies

Related issues

Planning

10 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Make gortex daemon stop finish in seconds without SIGKILL: tolerant teardown, bounded close #1022

Description

Why

Current state

What to do

Out of scope

Done when

How to verify

Dependencies

Related issues

Planning

Activity

zzet commented on Oct 27, 2026

zzet commented on Oct 27, 2026

zzet commented on Oct 27, 2026

zzet commented on Oct 27, 2026

zzet commented on Oct 27, 2026

zzet commented on Oct 27, 2026

10 remaining items

zzet commented on Oct 29, 2026

zzet commented on Oct 29, 2026

zzet commented on Oct 29, 2026

zzet commented on Oct 29, 2026

zzet commented on Oct 29, 2026

zzet commented on Oct 29, 2026

zzet commented on Oct 29, 2026

zzet commented on Oct 29, 2026

zzet commented on Oct 29, 2026

zzet commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Make `gortex daemon stop` finish in seconds without SIGKILL: tolerant teardown, bounded close #1022