20260815 - Give registration a timeout long enough for the server to answer - #14
Merged
Merged
Conversation
…answer Found live against production on 2026-08-15, deploying the first two nodes. Registration timed out four times against api.retina.fm at the 5s default read timeout, and the node held at `registering` with no token. The server had in fact completed all four: it minted tokens we hung up before reading, so the node was unregistered locally and registered four times remotely. Recovering from that needs an operator to reactivate the node, because a repeat attempt from a node that already holds a valid token returns the same opaque 403 as every other refusal. The cause is that registration is not bounded by our server. The spec has it query Mender while the request is open, so the call waits on a third party's API. An unknown node_id is refused in under a second — the fast path we had only ever tested — but a real enrolled node took roughly fourteen seconds on its first attempt and under half a second once warm. So the read budget goes to 30s for registration alone. Cloudflare fronts the origin with a 100s ceiling on its response, so 30s sits well inside what the edge will wait for, and being generous costs one slow start once per node lifetime. Being tight costs a node that cannot come up at all, and spends the 5/hour and 20/day allowance doing it. Detections keep their much shorter budget and are unaffected: abandoning a slow frame so the next goes out fresh is the whole transport discipline. Two tests. The first pins the ordering of all three budgets, so detections cannot quietly drift above the default or registration below it. The second asserts the Registrar actually passes the value — the default's docstring claimed the must-land endpoints overrode it, and for a long time none of them did, which is precisely how a timeout nobody had chosen ended up on the one request that cannot be retried cheaply. Both nodes registered and are streaming: ret824685c9 in 14s, ret3773656d in 0.4s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found live against production on 2026-08-15, deploying the first two nodes to
api.retina.fm.What happened
Registration timed out four times at the 5 s default read timeout, and the node held at
registeringwith no token. The server had completed all four — it minted tokens we hung up before reading. The node was unregistered locally and registered four times remotely, which needs an operator to reactivate, since a repeat attempt from a node already holding a valid token returns the same opaque403as every other refusal.Why the default was wrong
Registration is not bounded by our server. The spec has it query Mender while the request is open, so the call waits on a third party's API.
node_id(the fast path we had tested)Only the fast path was ever exercised, because every prior test registered against a mock or an unknown identity.
The change
REGISTER_TIMEOUT_S = (3.05, 30.0), for registration alone. Cloudflare fronts the origin with a 100 s ceiling on its response, so 30 s sits well inside what the edge will wait for.Being generous costs one slow start once per node lifetime. Being tight costs a node that cannot come up at all — and spends the 5/hour and 20/day allowance discovering that.
Detections are unaffected and keep their much shorter budget: abandoning a slow frame so the next goes out fresh is the whole transport discipline.
Tests
Registraractually passes it. The default's docstring claimed the must-land endpoints overrode it, and for a long time none of them did — which is exactly how a timeout nobody had chosen ended up on the one request that cannot be retried cheaply.tools/check.shandtools/check.sh --trackedboth green.Verified live
Both nodes registered on production and are streaming:
ret824685c9ndefud1o03fxwj9ret3773656dnde2dmghfu2w6qy🤖 Generated with Claude Code