Test and document the runaway-warmup escape: daemon stop, then offline untrack #1029

Why

Sometimes the daemon starts warming a repository the user did not mean to track, such as a home directory, a monorepo parent or a huge vendored tree. That warmup can run for many minutes or hours and keep the machine busy. We do not yet promise that untracking the repo works while the daemon is still warming it. The supported way out is:

1. stop the daemon;
2. remove the repo from the config while the daemon is down;
3. start the daemon again.

Today no test covers this path from start to finish, and no doc describes it.

Stop itself is slow right now. Every gortex daemon stop is force-killed after about 18 s: a 15 s exit grace plus up to 15 s for the checkpoint on close. A separate shutdown issue sets the stop budget:

• ≤3 s when idle;
• ≤5 s after a cold index or after churn;
• 0 SIGKILLs in 20 runs;
• the store is closed and its lock released.

This issue holds the escape hatch to that same budget. The v0.64.12 upgrade note will tell users: "to stop a runaway warmup: gortex daemon stop, then offline gortex untrack". This issue makes that sentence true and keeps it true.

Current state

Verified on main:

• Stop (cmd/gortex/daemon.go, runDaemonStop):
  • It records a sticky stop intent (daemon.MarkStopIntent), so an editor's gortex mcp proxy does not respawn the daemon.
  • When systemd or launchd owns the daemon (serviceActive()), it stops it through serviceStop. Otherwise it calls stopRunningDaemon, which waits for the process to exit and release the store lock.
  • daemonStopExitGrace sets how long it waits before a force-kill: 15 s after an ack, 2 min with no ack, 5 min at most.
• Untrack (cmd/gortex/track.go, runUntrack):
  • It asks untrackDaemonIndex whether a running daemon owns the target (daemon.IsRunning() plus daemonOwnsRepo in cmd/gortex/cli_daemon.go).
  • If no daemon owns it, it falls back to editing the config only: config.LoadGlobal → RemoveRepo → Save. It prints [gortex] untracked <path> (config only) and does not open the store.
• Gotcha: untrack by prefix needs the daemon. RemoveRepo (internal/config/global.go) makes its argument absolute and matches it by path identity. A bare repo prefix such as gortex untrack backend, which the docs show, therefore resolves against the current directory when the daemon is down. It fails with repository not found unless the user runs it from the repo's parent. Offline untrack only works reliably with a path.
• Next start (cmd/gortex/daemon_state.go): warmup runs an orphan-prefix purge first (orphan_maintenance, then orphan_prefix_scan, then orphan_prefix_purge). It purges every stored repo prefix that no tracked repo in the config maps to. A repo removed from the config while the daemon was down therefore has its rows purged on the next start instead of being indexed.
• Tests: cmd/gortex/track_offline_test.go covers only offline track (TestTrack_OfflineSucceeds, TestTrack_OfflineIdempotent). No test covers offline untrack. No test covers stop → untrack → start against a real daemon in mid-warmup.
• Docs: docs/cli.md says track and untrack fall back "to config-only when no daemon" is running. docs/onboarding.md has a Troubleshooting section, but it says nothing about recovering from a runaway warmup.

What to do

• Gated end-to-end test. Skip it unless an opt-in env var is set (for example GORTEX_E2E=1), so the default go test ./... stays fast. The test:
  1. builds or reuses the gortex binary and runs it against an isolated HOME, config, socket and store;
  2. tracks a small fixture repo A and a large synthetic repo B, then starts the daemon. B must be big enough that its cold warmup is still running when the test acts;
  3. waits until gortex daemon status shows warmup of B in progress, then runs gortex daemon stop and records its wall-clock time;
  4. checks that the process exited without SIGKILL, the socket is gone and the store lock is released (a fresh store open succeeds);
  5. runs gortex untrack <absolute path of B> and checks that it took the config-only path, exited 0, removed B from the global config and kept A;
  6. starts the daemon again and checks that A becomes ready, B is neither warmed nor indexed, and B's repo prefix has no rows left after the orphan purge;
  7. logs each stop duration and whether a SIGKILL happened, so a repeated run can be audited.
• Unit tests for the offline runUntrack path, next to the existing offline-track tests, with no daemon running:
  • removing a tracked path deletes that entry and keeps the others;
  • removing an unknown path returns a clear repository not found error and leaves the config unchanged;
  • a bare prefix given from a directory where it does not resolve returns the same clear error and does not delete some other entry.
• Docs. Add a short "Recovering from a runaway warmup" section to the Troubleshooting part of docs/onboarding.md, and link it from docs/cli.md. It covers:
  • gortex daemon stop, and that it goes through the service manager when one is installed;
  • gortex untrack <path> with the daemon down, using a path rather than a bare prefix;
  • gortex daemon start;
  • what to expect: the repo's rows are purged on the next start;
  • how to confirm, with gortex daemon status.
• Upgrade note. Use the same steps and wording in the v0.64.12 upgrade note.

Out of scope

• Making stop itself fast. The separate shutdown issue owns that, and the stop-timing targets below can only pass once it is merged.
• Supporting untrack against a daemon that is still warming the target.
• Making offline untrack accept a bare repo prefix (it is documented and tested as a clear error here, not fixed).
• Windows service-manager specifics beyond what the existing stop path already does.

Done when

• The gated end-to-end test exists and is skipped when the opt-in env var is unset.
• With the env var set, the test passes on macOS and Linux in 20 consecutive runs (-count=20).
• In those runs, gortex daemon stop during an active cold warmup returns in ≤5 s wall clock every time. There are 0 SIGKILLs across all 20 runs, and the store lock is released before the next step.
• Offline gortex untrack <path> returns in ≤1 s, exits 0 and removes only the target entry from the global config.
• After the restart:
  • the untracked repo is not warmed and gortex daemon status does not list it;
  • its repo prefix has 0 rows in the store;
  • the other repo reaches ready.
• Unit tests for offline runUntrack cover the three cases above (tracked path, unknown path, unresolvable bare prefix) and pass under go test -race ./cmd/gortex/.
• The troubleshooting section is merged in docs/onboarding.md with a link from docs/cli.md, and the v0.64.12 upgrade note uses the same steps.
• The gated test is part of the release-gate run on the reference Linux host and the reference Mac.

How to verify

• go test -race -run 'Untrack.*Offline' ./cmd/gortex/
• GORTEX_E2E=1 go test -run TestOfflineUntrackEscapeHatch -count=20 ./cmd/gortex/, then read the logged stop durations and SIGKILL count. Run this on macOS and on Linux.
• By hand:
  1. Track a large directory and run gortex daemon start.
  2. Wait for warmup to begin, then run time gortex daemon stop.
  3. Run gortex untrack /abs/path/to/dir, then gortex daemon start.
  4. Run gortex daemon status: the directory is gone and its rows were purged (look for the orphan_prefix_purge step in the daemon log).

Dependencies

Blocked by (must land first):

• Make gortex daemon stop finish in seconds without SIGKILL: tolerant teardown, bounded close #1022 — The offline-untrack test assumes daemon stop finishes within the stop budget.

Related issues

• Related: Catalog primary graph keeps a stale repo_prefix: worktrees never get a coordinator, untrack fails forever with "cannot serve checkout … yet" #898, daemon: deferred passes lost when a cold warmup dies after the bulk flush — next start sees repos unchanged and never re-arms them #670, The daemon cannot be stopped by 'gortex daemon stop' as expected #68

Planning

• Milestone: v0.64.12
• Priority: enabler
• Estimate: ~0.5 engineer-day
• Isolated enough for a community contributor.