Skip to content

fix: give log-node and archive-node real health-check margin - #613

Merged
creatornader merged 1 commit into
mainfrom
fix/deploy-job-isolation
Jul 29, 2026
Merged

fix: give log-node and archive-node real health-check margin#613
creatornader merged 1 commit into
mainfrom
fix/deploy-job-isolation

Conversation

@creatornader

Copy link
Copy Markdown
Owner

Item #5 from the graph-node incident follow-ups. The premise turned out to be wrong, and the real finding is different from what I expected.

Correction first

I previously reported that "no service has Fly health checks configured at all." That was wrong. I had grepped for checks]] — the [[http_service.checks]] array syntax — which misses the top-level [checks] form the other three services use.

Actual state: all four services have health checks and all four are passing.

atrib-log:       1 total, 1 passing   ->  GET /v1/pubkey
atrib-directory: 1 total, 1 passing   ->  GET /v6/anchor
atrib-archive:   1 total, 1 passing   ->  GET /v1/retention
atrib-graph:     1 total, 1 passing   ->  GET /v1/stats   (added in #606)

graph-node genuinely had none, so #606 was correct. But "add checks to the other three" was not a real task.

What is real

Measuring startup to justify grace periods surfaced an actual problem. From the Fly logs, "Starting init" to "listening":

service startup grace_period margin
log-node 5s 5s none
archive-node 4s 5s 1s
directory-node 3s 15s 5x

grace_period is the window before a failing check counts, and none of these bind their port until startup completes. log-node has zero margin — any slower boot marks a healthy service unhealthy.

Both numbers also grow, which is what makes this the same latent class as the graph-node outage rather than a one-off:

  • log-node replays the persisted log into its in-memory Merkle tree at startup (its own [[vm]] comment says so), so it scales with tree_size — ~115k entries replaying at roughly 23k/sec today.
  • archive-node replays archived record bodies. 13 of them right now, which is precisely why a 5s grace hasn't bitten yet.

Change

  • log-node 5s -> 45s (~9x measured boot, covers order-of-a-million entries)
  • archive-node 5s -> 30s (~7x measured boot)
  • directory-node stays at 15s, but gains the measurement in a comment so the next person re-checks instead of re-deriving

Every file now records the ceiling: Fly clamps grace_period at 60s and silently lowers anything larger. Past that the fix is a cheaper startup, not a longer grace — the same constraint P059 records for graph-node, where replay time is what bounds machine sizing.

Timeouts and intervals are deliberately untouched. I have no evidence of flapping, and changing them would be the guessing this PR is meant to replace.

Verification

flyctl config validate on all three: valid, no clamp warnings (45s and 30s are under 60s). No application code changed, so no test impact.

Scope note

This PR touches only the three fly.toml files. Items #2 (splitting log-smoke out of deploy-log-node) and #3 (deploy concurrency) are being handled in a concurrent session that is already editing deploy-services.yml; I've coordinated with it directly rather than opening a competing PR. One thing I flagged there: a single workflow-level concurrency group cancels earlier pending runs, and since change detection is per-commit, that can drop a service's only deploy — job-level groups avoid it.

All four services already had Fly health checks and all four were passing, so
this is not about adding coverage. It is about two grace periods that were set
below, or barely above, how long the service actually takes to start.

Measured 2026-07-29 from the Fly logs, "Starting init" to "listening":

  log-node        5s startup, 5s grace   no margin
  archive-node    4s startup, 5s grace   1s
  directory-node  3s grace 15s           5x, left alone

grace_period is the window before a failing check counts, and neither service
binds its port until startup finishes. At zero margin any slower boot marks a
healthy service unhealthy.

Both numbers also grow. log-node replays the persisted log into its in-memory
Merkle tree at startup, so it scales with tree_size, ~115k entries replaying at
roughly 23k/sec on the measurement date. archive-node replays archived record
bodies, currently 13, which is why its tight value has not bitten yet.

log-node goes to 45s and archive-node to 30s, roughly 9x and 7x measured boot.
directory-node keeps 15s and gains the measurement in a comment so the next
person can re-check rather than re-derive.

Every file now records the ceiling: Fly clamps grace_period at 60s and lowers
anything larger without failing, so past that the fix is a cheaper startup
rather than a longer grace. That clamp is the same constraint P059 records for
graph-node, where archive replay time is what bounds machine sizing.
@creatornader
creatornader merged commit 6b16172 into main Jul 29, 2026
14 checks passed
@creatornader
creatornader deleted the fix/deploy-job-isolation branch July 29, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant