Repository navigation
Stop a stuck server, and check documents for a deploy without holding writes - #60
Merged
Merged
Conversation
… writes Watchdog. A query or mutation that loops without awaiting stops the server's one thread, and with it the timer that is its time limit. A second thread now watches whether the first is still turning. After 30 seconds stuck it prints which function started last and kills the process, so a supervisor starts a fresh one. `onebase serve` turns it on; $ONEBASE_WATCHDOG sets the wait or turns it off. The thread's program is a blob, so it is in compiled binaries without build changes. Deploy check. A deploy that changes the schema validated every document inside its write transaction, so every write waited for the whole scan. On SQLite the scan now runs first, 500 documents per read with the event loop turning in between, while writes record the ids they touch; the transaction re-checks only those. A dry run is checked the same way instead of in one long read. Shared databases keep the old check: other instances write too, and their writes can't be collected in time. Measured on 100,000 documents with queries and writes running: the longest stall during the deploy went from 98 ms to 3 ms. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two ways one action could hold up the whole server.
1. A watchdog for a stuck main thread
Queries and mutations run on the server's one thread, and a time limit is a timer on that same thread. A function that loops without awaiting (
while (true) {}) stops the timer along with everything else: the server keeps accepting connections and answers none, until someone notices.src/engine/watchdog.tsadds a second thread that only watches whether the first is still turning. When it has been stuck for 30 seconds, it prints which function started last and kills the process, so whatever supervises it starts a fresh one:onebase serve(and so foronebase dev), at 30s.ONEBASE_WATCHDOG=2moroffchanges it.startApp()leaves it off unlesswatchdogMsis given, so embedders and tests are unaffected.SIGKILL. A stuck thread can't run a shutdown handler. Nothing committed is lost.onebase servein a terminal), the server stops and stays stopped, which is still better than hanging silently, but worth knowing.bun build --compile --minifybinary: it stops and prints the message.2. A deploy's document check no longer holds up writes
When
schema.tschanges, a deploy validates every stored document. That happened inside the deploy's write transaction, so every write waited for the whole scan, and on SQLite so did everything else on the thread.On SQLite the scan now runs before the transaction: 500 documents per read, with the event loop turning between reads. While it runs, each write records the ids it touched, and the transaction then re-checks only those. A document that stops matching during the scan is still caught.
Measured
A schema-changing deploy on 100,000 documents, with queries and writes running against the server:
So the stall was about 1 ms per 1,000 documents: small here, about a second at a million documents. The deploy itself takes a little longer.
Tests
tests/watchdog.test.tsstarts a real server in a child process: an idle server is left alone, a mutation that never returns gets it killed and named, and with the watchdog off it stays stuck.tests/deploy-check.test.ts: the scan happens in short reads and never inside a write transaction; a dry run does the same and changes nothing; a document made invalid while the scan runs still stops the deploy.bun run typecheckis clean;bun test: 575 pass, 3 skip, 0 fail on SQLite. Not run locally on Postgres or libSQL, where the deploy check is unchanged.🤖 Generated with Claude Code