Skip to content

fix(daemon): cancel shutdown producers and verify store-free stop - #1161

Merged
zzet merged 74 commits into
mainfrom
fix/daemon-stop-teardown
Oct 29, 2026
Merged

zzet merged 74 commits into
mainfrom
fix/daemon-stop-teardown

Conversation

@zzet

@zzet zzet commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Daemon shutdown could leave startup work running, skip store close after a contract-drain error, or wait for an already-exited PID until the CLI forced a kill. Shutdown now cancels and joins its producers, closes their stores, and verifies completion against the captured store lock and runtime identities.

  • Continue cleanup after contract-drain errors, return warnings with the shutdown ACK, and schedule shutdown even if ACK delivery fails.
  • Join watcher/warmup, checkout, analysis and semantic work before closing their backend. Preserve completion barriers rather than treating a timeout as completion.
  • Skip the final TRUNCATE checkpoint by default and disable SQLite's implicit close checkpoint on every physical graph/sidecar connection. Keep full-drain opt-in and mandatory bulk sealing; recover committed data from WAL on reopen.
  • Detect completion using the captured store and process incarnation. Do not kill an exited, unreaped process; require matching ownership before signaling or cleanup. Arm the exit watchdog only after successful shared close and runtime cleanup.
  • Stop canceled warmup test-edge projection between complete file groups. Persist its interrupted file frontier and repair it before readiness on unchanged restart. Keep each eviction/replacement together, retain repair debt on publication failure, and prevent incomplete repair from reporting enrichment complete.

Focused race controls for the product at 5e223d667e9eca8d23e07f51787ddf21883362f1 cover cancellation, SQLite close/reopen through unchanged startup, failed publication with retained repair debt, cross-group test-edge correctness and shutdown ownership. The pinned Go 1.27.3 / golangci-lint 2.14.0, formatting, affected vet, CGO/workflow guards, tidy and module verification passed. The test-only CI corrections are pushed at ffbec2b21bde2c3ca991aed6e2847b2870424bfc. Capability/debt controls, 20 repeated relay-restart runs and protocol/no-replay neighbors passed under race. A later CI run exposed a catalog fixture shutting down before Serve began; a bounded real handshake now establishes admission. The failing control passed 30 race repetitions, with 15 neighboring top-level passes and 18 subtest passes. All prescribed pinned checks passed again before push. Final-head hosted CI passed: 24 successful checks and two skipped jobs. Only three test files differ from the measured product.

Fresh 29-repository validation passed all four observations at 5e223d667: complete stops 0.694–3.466 s, backend-close-to-CLI 53.288–60.603 ms, and matching graph calls 137.849–324.658 ms after readiness. All declared roots were ready/loaded; enrichment was still running. The first restart repaired interrupted projection work in 10.554 s before readiness; clean census phases took 96.919–115.905 ms. No forced kill, watchdog dump or OOM occurred, and evidence/ownership/lock/cleanup checks passed. The VM stopped normally.

The ledger now has passing observations for 48/48 distinct rows across their recorded source versions. Earlier results retain their source attribution, including one-hour churn and live-worktree stops; every row is not being relabeled as a final-head rerun. The earlier 9.026-second full-workspace failure remains recorded. Acceptance is complete; PR merged as 9797334420c47b2892652d8f14a2e84bc85dfc10.

Closes #1022; prerequisite #995 is merged. #1022 is closed and Done; #1029 can complete its own CI.

zzet added 30 commits October 27, 2026 18:03
Give warmup a daemon-owned context, join its producer before stopping the final watcher and closing shared state, and reject watcher handoffs after stop. Preserve canceled batch cleanup and forward cancellation to startup resolution and derived passes. Add deterministic parser, handoff and teardown-order regressions. Refs #1022.
@zzet
zzet marked this pull request as ready for review October 29, 2026 13:07
@zzet
zzet merged commit 9797334 into main Oct 29, 2026
26 checks passed
@zzet
zzet deleted the fix/daemon-stop-teardown branch October 31, 2026 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make gortex daemon stop finish in seconds without SIGKILL: tolerant teardown, bounded close

1 participant