Skip to content

Stop a stuck server, and check documents for a deploy without holding writes - #60

Merged
abdullah-bl merged 1 commit into
mainfrom
feat/watchdog-and-deploy-checks
Nov 16, 2026
Merged

abdullah-bl merged 1 commit into
mainfrom
feat/watchdog-and-deploy-checks

Conversation

@abdullah-bl

Copy link
Copy Markdown
Owner

Two ways one action could hold up the whole server.

1. A watchdog for a stuck main thread

Queries and mutations run on the server's one thread, and a time limit is a timer on that same thread. A function that loops without awaiting (while (true) {}) stops the timer along with everything else: the server keeps accepting connections and answers none, until someone notices.

src/engine/watchdog.ts adds a second thread that only watches whether the first is still turning. When it has been stuck for 30 seconds, it prints which function started last and kills the process, so whatever supervises it starts a fresh one:

onebase: the server's main thread has been stuck for 30s, so this process is stopping for its supervisor to restart it.
  The last function to start was notes:spin.
  A query or mutation that loops without awaiting does this: time limits can't interrupt it. Find the loop, or move heavy work to an action.
  $ONEBASE_WATCHDOG sets how long to wait (e.g. 2m), and "off" turns this off.
  • On by default for onebase serve (and so for onebase dev), at 30s. ONEBASE_WATCHDOG=2m or off changes it. startApp() leaves it off unless watchdogMs is given, so embedders and tests are unaffected.
  • It stops with SIGKILL. A stuck thread can't run a shutdown handler. Nothing committed is lost.
  • It needs a supervisor. Without one (a bare onebase serve in a terminal), the server stops and stays stopped, which is still better than hanging silently, but worth knowing.
  • "The last function to start" is a best guess. With several calls in flight, the stuck one is usually, not always, the last to start.
  • Sleep doesn't trip it. It counts its own checks, not the clock, so a laptop waking up doesn't look like a 30-second stall.
  • No build changes. The thread's program is a blob, like the vector workers, so it's in compiled binaries as they are. I checked a bun build --compile --minify binary: it stops and prints the message.

2. A deploy's document check no longer holds up writes

When schema.ts changes, a deploy validates every stored document. That happened inside the deploy's write transaction, so every write waited for the whole scan, and on SQLite so did everything else on the thread.

On SQLite the scan now runs before the transaction: 500 documents per read, with the event loop turning between reads. While it runs, each write records the ids it touched, and the transaction then re-checks only those. A document that stops matching during the scan is still caught.

  • Dry runs are checked the same way, instead of in one long read that held up every query.
  • Shared databases keep the old check. Other instances write too, and their writes can't be collected in time.
  • Startup is unchanged: nothing is being served yet.
  • If more than 100,000 documents are written during one scan, the transaction falls back to checking everything.

Measured

A schema-changing deploy on 100,000 documents, with queries and writes running against the server:

Before After
Longest stall 98 ms 3 ms
Deploy took 108 ms 167 ms

So the stall was about 1 ms per 1,000 documents: small here, about a second at a million documents. The deploy itself takes a little longer.

Tests

  • tests/watchdog.test.ts starts a real server in a child process: an idle server is left alone, a mutation that never returns gets it killed and named, and with the watchdog off it stays stuck.
  • tests/deploy-check.test.ts: the scan happens in short reads and never inside a write transaction; a dry run does the same and changes nothing; a document made invalid while the scan runs still stops the deploy.
  • All three deploy-check tests fail against the old code.
  • bun run typecheck is clean; bun test: 575 pass, 3 skip, 0 fail on SQLite. Not run locally on Postgres or libSQL, where the deploy check is unchanged.

🤖 Generated with Claude Code

… writes

Watchdog. A query or mutation that loops without awaiting stops the
server's one thread, and with it the timer that is its time limit. A
second thread now watches whether the first is still turning. After 30
seconds stuck it prints which function started last and kills the
process, so a supervisor starts a fresh one. `onebase serve` turns it
on; $ONEBASE_WATCHDOG sets the wait or turns it off. The thread's
program is a blob, so it is in compiled binaries without build changes.

Deploy check. A deploy that changes the schema validated every document
inside its write transaction, so every write waited for the whole scan.
On SQLite the scan now runs first, 500 documents per read with the
event loop turning in between, while writes record the ids they touch;
the transaction re-checks only those. A dry run is checked the same
way instead of in one long read. Shared databases keep the old check:
other instances write too, and their writes can't be collected in time.

Measured on 100,000 documents with queries and writes running: the
longest stall during the deploy went from 98 ms to 3 ms.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@abdullah-bl
abdullah-bl merged commit acf28ae into main Nov 16, 2026
1 check passed
@abdullah-bl
abdullah-bl deleted the feat/watchdog-and-deploy-checks branch November 16, 2026 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants