Give nodes that asked for it a Cloudflare tunnel - #6
Merged
Merged
Conversation
tunnel_sync.py reads the `remote_access` inventory attribute owl-os now
reports, reconciles it against Cloudflare, and uploads the connector token back
to the device over Mender file transfer. Nodes never call us, so there is no
node-facing endpoint and nothing to authenticate: Mender's PAT proves we are
Offworld and Mender's device auth proves the node is the node.
## Separate from auto_accept.py, on purpose
auto_accept.py accepts devices and deploys OS updates for the whole fleet. If
tunnel work lived in its loop, a Cloudflare outage or an unhandled exception
would stop device acceptance, which is a far worse failure than this feature
being unavailable. Keeping them apart means auto_accept.py changes by zero
lines, so nothing here can affect how existing nodes are handled.
It also wants a slower cadence. Thirty seconds is right for catching a
newly-flashed board; nodes report tunnel intent through Mender inventory on a
600s poll, so five minutes loses nothing and costs fewer API calls.
## Backwards compatible by construction
`remote_access` is read as three states, not two. Absent is NOT false: a node
on an OS predating the feature can neither ask for a tunnel nor receive one, so
it is never touched. Collapsing absent into false would look equivalent and is
not, because an OS rollback removes the inventory script and would then tear
down a working tunnel for an owner who never asked for that.
Confirmed against the real fleet: all 18 devices classify as skip, zero
actions. Opting one node in produced exactly one create and left the other 17
alone.
## Three guards, because this deletes things
A name guard. retnode.com already carries eight hand-built tunnels for live
nodes, several serving multiple hostnames, and ensure_tunnel PUTs a tunnel's
*entire* ingress config. A miscomputed name would not fail, it would silently
replace working routing. Nothing may be touched unless it matches ^ret[0-9a-f]{8}$.
An empty-fleet guard. Mender answering with no devices is indistinguishable
from every node having opted out, and acting on it would tear down every tunnel
we own. Zero devices is treated as a failure rather than as instructions.
Orphans are reported, never deleted on a timer. --prune is opt-in and the
systemd unit runs --apply only, so orphans reach the journal for a human.
## Cost
One paginated inventory call per pass returns the whole fleet with attributes,
and reconciliation works from two bulk Cloudflare listings rather than per-node
lookups. Both are O(1) in fleet size. Cloudflare is only written to when a
node's intent differs from what was last recorded, so the steady state is zero
Cloudflare calls.
## Verified against live infrastructure
Full pipeline on ret4c844c20: create, token upload, connector attached, public
TLS, then teardown. Reconciliation caught a state file claiming a tunnel that
did not exist, and found and pruned a throwaway ret00000000 tunnel while
leaving all eight real ones intact. The empty-fleet guard refused to act with
--prune and made zero destructive calls.
Not yet done: no rate limiting anywhere, which is a recorded decision rather
than an oversight. See the docs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tion Provisioning now creates an Access application per node whose policy references the support group by id. Nothing personal appears in this repo's config or on any node: membership lives in that one group, so adding or removing someone applies across the whole fleet at once without editing an application or touching a device. Per node rather than one wildcard application over the zone. A wildcard would also cover the hand-built hostnames already there, several of which serve people outside the support team, and locking them out is not this feature's business. ## Ordering The application is created before the DNS record. The moment that record resolves the hostname serves the node's interface, so creating the policy afterwards leaves a window in which whoever finds the name is inside. The same reasoning puts access.json on the node before the token. That file names the team and this node's application, and retina-gui refuses every visitor until it has them, so landing the token first would bring the hostname up during the gap. It fails closed rather than open, but it is an outage nobody needs. ## Reconciliation A hostname that resolves with no application in front of it is reported ahead of anything else about that node, because it is a different order of problem from a stale DNS record. Orphaned applications for nodes nothing wants are reported too, and pruned only on request like the rest. ## Verified end to end on ret4c844c20 Application created with the group policy and no addresses in it; access.json and the token delivered 0600 root with the staged copies consumed; connector up with four registered connections. An unauthenticated request to the hostname was stopped at Cloudflare and redirected to the team login, carrying the same audience that had been written to the node. Teardown removed application, tunnel and DNS, leaving the three pre-existing applications, eight tunnels and twenty-four records untouched. A 400 from the upload endpoint now says what it usually means, which is that the staging directory does not exist because the node predates the owl-os role that creates it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The example file described the tunnel and DNS credentials but not the two Access settings, so following it produces a config that refuses to run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Live testing withdrawal on a real node: the tunnel came down correctly, then the same pass put it straight back up. The hostname resolved again seconds after its owner turned support access off. `repairs` is computed at the top of a pass, from the state as it stood before any teardown ran. A node being torn down therefore looks damaged, because its DNS record really is missing: we had just deleted it. The repair loop then recreated the tunnel, the DNS record and the Access application, and only failed to finish because it also unpacked ensure_tunnel's three return values into two. An agreement the next pass quietly reverses is not an agreement, so repairs are now filtered against what the owner currently wants. Two further faults the same test surfaced: Teardown bundled the Access application and the tunnel into one try, so a tunnel that refused deletion took the application with it. Cloudflare refuses while a tunnel still has connections (error 1022), which is routine rather than exceptional: a node that has just gone offline holds stale ones for a few minutes. Each resource now fails on its own terms, the entry stays for the next pass to retry, and DNS is deleted first and unconditionally because that is the step that actually revokes access, and the only one needing nothing from the node. Deleting an already-deleted tunnel now counts as gone rather than as a failure that would retry forever. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The state file sat beside the script, which meant mutable state lived inside a checkout that gets pulled and switched. StateDirectory hands it a proper home under /var/lib and systemd creates it with the right ownership, so the unit no longer needs write access to the repo at all. Losing the file does not orphan anything, since reconciliation finds tunnels by name, but it would make the next pass rebuild state it should simply have had. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Provisions a Cloudflare tunnel for any node whose owner has turned support access on, and tears it down when they turn it off. Purely additive: 4 files, 815 insertions, 0 deletions.
auto_accept.pyis untouched.What it does
tunnel_sync.pyreads each device'sremote_accessinventory attribute from Mender and reconciles Cloudflare to match: a per-node Access application, a tunnel, and a proxied CNAME. There is no node-facing endpoint anywhere in this, so there is nothing to authenticate and nothing to attack. The node states intent, we poll it.Intent is three-state. Absent is not false: it means the node's OS predates the feature, and such a node is skipped explicitly rather than treated as a request to tear anything down.
Why it is safe to merge before anything else
It is inert until a node asks. No node publishes the attribute unless it is running an OS with the matching inventory scripts, so on today's fleet every device is skipped. That has been verified repeatedly on the real droplet: 18 devices, 18 skipped, nothing to do.
Two independent guards protect the existing zone:
_guard()sits on all 9 mutating call sites and raises unless the name matches^ret[0-9a-f]{8}$. Every one of the 24 hand-built records inretnode.comfails that pattern.Live testing
Deployed to Mender-Infra and exercised end to end on two nodes, driven by the timer with nobody in the loop:
The zone went 24 -> 26 -> 24 records exactly. Teardown was also tested with a node deliberately unreachable: DNS and the Access application were removed and the hostname went NXDOMAIN while the node still held its token, with the tunnel deletion deferred and completed on the next pass. Revocation does not depend on the node cooperating.
Notes for review
--applyonly, never--prune. Pruning deletes tunnels and DNS records, so orphans are reported for a human to look at rather than removed on a schedule./var/lib/retina-tunnel-syncviaStateDirectory, not beside the script, so a pull or branch change cannot disturb it.🤖 Generated with Claude Code