fix: give log-node and archive-node real health-check margin - #613
Merged
Conversation
All four services already had Fly health checks and all four were passing, so this is not about adding coverage. It is about two grace periods that were set below, or barely above, how long the service actually takes to start. Measured 2026-07-29 from the Fly logs, "Starting init" to "listening": log-node 5s startup, 5s grace no margin archive-node 4s startup, 5s grace 1s directory-node 3s grace 15s 5x, left alone grace_period is the window before a failing check counts, and neither service binds its port until startup finishes. At zero margin any slower boot marks a healthy service unhealthy. Both numbers also grow. log-node replays the persisted log into its in-memory Merkle tree at startup, so it scales with tree_size, ~115k entries replaying at roughly 23k/sec on the measurement date. archive-node replays archived record bodies, currently 13, which is why its tight value has not bitten yet. log-node goes to 45s and archive-node to 30s, roughly 9x and 7x measured boot. directory-node keeps 15s and gains the measurement in a comment so the next person can re-check rather than re-derive. Every file now records the ceiling: Fly clamps grace_period at 60s and lowers anything larger without failing, so past that the fix is a cheaper startup rather than a longer grace. That clamp is the same constraint P059 records for graph-node, where archive replay time is what bounds machine sizing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Item #5 from the graph-node incident follow-ups. The premise turned out to be wrong, and the real finding is different from what I expected.
Correction first
I previously reported that "no service has Fly health checks configured at all." That was wrong. I had grepped for
checks]]— the[[http_service.checks]]array syntax — which misses the top-level[checks]form the other three services use.Actual state: all four services have health checks and all four are passing.
graph-node genuinely had none, so #606 was correct. But "add checks to the other three" was not a real task.
What is real
Measuring startup to justify grace periods surfaced an actual problem. From the Fly logs, "Starting init" to "listening":
grace_periodis the window before a failing check counts, and none of these bind their port until startup completes. log-node has zero margin — any slower boot marks a healthy service unhealthy.Both numbers also grow, which is what makes this the same latent class as the graph-node outage rather than a one-off:
[[vm]]comment says so), so it scales withtree_size— ~115k entries replaying at roughly 23k/sec today.Change
5s -> 45s(~9x measured boot, covers order-of-a-million entries)5s -> 30s(~7x measured boot)Every file now records the ceiling: Fly clamps
grace_periodat 60s and silently lowers anything larger. Past that the fix is a cheaper startup, not a longer grace — the same constraint P059 records for graph-node, where replay time is what bounds machine sizing.Timeouts and intervals are deliberately untouched. I have no evidence of flapping, and changing them would be the guessing this PR is meant to replace.
Verification
flyctl config validateon all three: valid, no clamp warnings (45s and 30s are under 60s). No application code changed, so no test impact.Scope note
This PR touches only the three
fly.tomlfiles. Items #2 (splittinglog-smokeout ofdeploy-log-node) and #3 (deploy concurrency) are being handled in a concurrent session that is already editingdeploy-services.yml; I've coordinated with it directly rather than opening a competing PR. One thing I flagged there: a single workflow-level concurrency group cancels earlier pending runs, and since change detection is per-commit, that can drop a service's only deploy — job-level groups avoid it.