Support custom (non-Tailscale) control planes end-to-end: TLS control, streaming map-poll, DERP - #22
Open
kentaro wants to merge 12 commits into
Open
Support custom (non-Tailscale) control planes end-to-end: TLS control, streaming map-poll, DERP#22kentaro wants to merge 12 commits into
kentaro wants to merge 12 commits into
Conversation
Headscale/Ionscale deployments commonly expose the control plane behind an HTTPS-only listener on port 443, which MicroLink's plain-TCP port-80 coordination transport cannot reach. This adds three Kconfig options to make custom control planes usable at build time: - CONFIG_ML_CTRL_HOST: coordination server hostname (build-time counterpart of the NVS web-UI override; useful for pre-provisioned firmware and when the config server is disabled) - CONFIG_ML_CTRL_TLS: wrap the ts2021 Noise handshake in TLS on port 443 (new src/ml_coord_tls.c, following the same mbedTLS custom-BIO pattern as the DERP connection). The server certificate is not verified; the control plane is authenticated by the pinned Noise key, the same trust model as the plain port-80 transport. Disabled by default — behavior is unchanged unless enabled. - CONFIG_ML_CTRL_NOISE_PUBKEY_HEX: pin a custom control plane Noise public key (from https://<host>/key?v=88), replacing the built-in Tailscale key. Tested against a Headscale-based control plane (443/TLS only): TLS handshake, Noise handshake and registration path build on ESP-IDF v5.4.4 / esp32s3.
Allow supplying the custom control plane hostname and Noise public key at runtime through the config struct, in addition to the build-time Kconfig options. This enables shipping a single pre-built firmware where the control plane (and its /key-fetched Noise key) is provisioned on-device instead of baked in. Priority: config struct > Kconfig > built-in Tailscale values; the NVS web-UI host override still wins. The README already documented a ctrl_host config field — this makes it real, and extends the same idea to the Noise key.
kentaro
changed the title
…false Some headscale-based control planes (e.g. Lolipop Zero Trust Link) do not respond at all to the initial Stream=false MapRequest — the client sees 60 seconds of zero bytes, an empty MapResponse, and falls into a reconnect loop, never obtaining peers or the DERPMap. The native Tailscale client instead starts a single Stream=true streaming poll whose first message already carries the full netmap (peers + DERPMap). This adds a microlink_config_t.streaming_map_fetch flag (default false = unchanged behavior). When true, COORD_FETCH_PEERS starts the long-poll with OmitPeers=false, reads the initial netmap to completion, parses Node/Peers/ DERPMap, and only then connects DERP. Self-node and DERPMap parsing are factored into shared helpers so both the Stream=false and Stream=true paths populate them.
Three fixes needed to reach a custom control plane's DERP relay, found
while bringing up an ESP32 node on a headscale-based deployment:
1. Region fallback: when the server's HomeDERP region is not present in
the DERPMap (custom control planes advertise their own region IDs, not
Tailscale's region 9), fall back to the first available region in the
DERPMap instead of the hardcoded Tailscale default host — otherwise the
client dials derp9e.tailscale.com and fails.
2. Robust TLS setup: check the return value of every mbedTLS setup call
and abort before mbedtls_ssl_handshake on failure. Previously
mbedtls_ssl_setup failing (typically -0x7F00 ALLOC_FAILED under internal
RAM pressure) was ignored; it NULLs ssl->conf, so handshake then returned
MBEDTLS_ERR_SSL_BAD_INPUT_DATA and the real cause (OOM) was masked as
"Bad input parameters". Also add derp_tls_abort() to free the mbedTLS
context (and ~20KB TLS buffers) on every failure path — the old code only
closed the socket, leaking on each retry and worsening the OOM over time.
(With CONFIG_MBEDTLS_INTERNAL_MEM_ALLOC the fix on the app side is
CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y to allocate TLS buffers from PSRAM.)
3. Pin DERP TLS to 1.2: on ESP-IDF mbedTLS the TLS 1.3 path cannot parse an
ECDSA server certificate's signature-algorithm OID ("X509 - Signature
algorithm (oid) is unsupported"), even with authmode=VERIFY_NONE, because
TLS 1.3 can't skip certificate processing. DERP relays accept TLS 1.2, so
cap the max version at 1.2 to avoid the 1.3 certificate path.
kentaro
force-pushed
the
feature/custom-control-plane-tls
branch
from
July 22, 2026 05:51
1e3e8ef to
125b529
Compare
wireguardif_process_data_message called ip_input() directly from the WireGuard rx task. When the inner packet is TCP, this ran tcp_input -> tcp_output on the rx task concurrently with app-side socket operations on the TCPIP thread, corrupting the PCB's unacked segment list and crashing in tcp_output (LoadProhibited, NULL useg) whenever a TCP response arrived over the tunnel. Deliver through netif->input instead (hosts set this to tcpip_input) so all TCP processing is serialized on the TCPIP thread. Fall back to ip_input when netif->input is unset. Free the pbuf if input() fails.
The device is an ESP32 running ESP-IDF, not Linux. Admin dashboards display this value as the node's OS, so report it honestly.
The control plane reports auth problems in-band (RegisterResponse Error / MachineAuthorized=false), but do_register never checked them: a rejected registration looked like success and the client spun in the reconnect loop with no way for the host app to tell the user. Headless devices have no UI to notice this — hosts need a hook to show their equivalent of a re-login prompt. - do_register returns ML_REG_AUTH_FAILED on in-band rejection - The coordinator raises ML_STATE_AUTH_FAILED through the state callback and jumps to maximum backoff (the loop keeps running in case the key is re-enabled server-side) - The key-expired-without-auth-key path raises the same state
The reconnect loop set ml->state = ML_STATE_RECONNECTING without invoking the state callback, unlike ML_STATE_CONNECTED and ML_STATE_AUTH_FAILED. Host apps relying on the callback for status display kept showing "connected" through a network outage (found in a live Wi-Fi outage test: the device UI stayed on the connected status while ml_coord was cycling through reconnect attempts). Fire the callback once on the transition into RECONNECTING; the loop re-enters this state on every backoff cycle, so guard on the previous state to avoid repeated callbacks.
The previous commit fired the RECONNECTING callback whenever the loop entered the reconnect state, which also fired right after a registration rejection had raised AUTH_FAILED — clobbering the host's "re-login" display with a generic "reconnecting" one (observed live: dashboard device deletion -> "authkey expired" rejection -> screen stuck on "reconnecting" instead of the re-auth prompt). AUTH_FAILED means human action is required; keep it sticky while the loop retries at maximum backoff. A successful reconnect still moves the state forward via the CONNECTED callback.
The codebase's comments are English; keep the fork consistent (and upstreamable via PR CamM2325#22).
kentaro
added a commit
to pepabo/microlink
that referenced
this pull request
Jul 28, 2026
The codebase's comments are English; keep the fork consistent (and upstreamable via PR CamM2325#22).
wireguard_tai64n_now() built the TAI64N timestamp from esp_timer_get_time(), which is microseconds since boot, not wall-clock time. A peer stores the greatest timestamp it has seen from us and rejects any handshake initiation whose timestamp is not greater (replay protection, WireGuard spec 5.1). With uptime, the counter restarts near zero on every reboot, so the peer answers with "handshake replay" and drops the initiation. A device that has been up for hours therefore cannot reconnect after a reboot until its uptime passes the previous session's — in practice, never. Observed on an ESP32-S3 talking to a userspace peer: wg: ConsumeMessageInitiation: handshake replay @ 1970-01-01 09:04:41 wg: Received invalid initiation message from 8f9b9e9a... DISCO probing succeeded the whole time, so the failure looked like a routing problem rather than a rejected handshake. Use gettimeofday() instead. The system clock is expected to be set before connecting; the ztl component already syncs it over SNTP for this reason. Also drop the periodic "[TAI64N] uptime=..." printf, which was debug scaffolding and prints to the console every five seconds.
wireguard_tai64n_now() was the only user of esp_timer in this file.
kentaro
force-pushed
the
feature/custom-control-plane-tls
branch
from
July 29, 2026 16:34
f6b454e to
a8d769e
Compare
This was referenced Aug 14, 2026
fudio101
pushed a commit
to fugo101/microlink
that referenced
this pull request
Aug 18, 2026
Extends the DERP TLS-leak fix already on this branch with the rest of what upstream shipped as one verified commit (125b529): - DERP region fallback when HomeDERP isn't in the DERPMap (falls back to the first usable non-avoid region instead of a dead-end connect to the default host) - Pin DERP TLS to 1.2 — some relays' Let's Encrypt ECDSA cert trips the ESP-IDF mbedTLS TLS 1.3 signature-algorithm OID path even under VERIFY_NONE - Check every mbedTLS setup call's return value (config_defaults, ssl_setup, set_hostname) and abort via the existing fail_tls path instead of proceeding into handshake with a half-initialized config - Rework the DERP BIO to a plain blocking f_recv bounded by SO_RCVTIMEO, dropping mbedtls_ssl_conf_read_timeout entirely — avoids a BAD_INPUT_DATA failure the recv_timeout scheme hit on some relays' TLS 1.3 path Plus the other generic (non-headscale) fixes from the same upstream PR: - Hostinfo "OS" field corrected to "esp32" (was "linux") - New ML_STATE_AUTH_FAILED state: do_register() now actually fails registration on a control-plane Error/MachineAuthorized rejection instead of silently "succeeding" into a broken state (Node absent, confusing downstream "node not found"); the expired-key-with-no- auth_key path raises the same state - state_cb now fires on the RECONNECTING transition (it never did before) and doesn't get clobbered by it while AUTH_FAILED is active Adapted from cplewes/microlink@38602ab0/@b25b1eee (already on this branch) and CamM2325#22 commits 125b529, fcdc8d9, ae3d438, 8367c1e, 9ef10bb — not a literal cherry-pick, base has diverged too far, and our mbedTLS 4.x/PSA migration already dropped the entropy/ctr_drbg fields upstream's version still frees. Deliberately excluded (see UPSTREAM_PRS.md): the 3 headscale/custom- control-plane feature commits from CamM2325#22 (this fork targets Tailscale's own SaaS control plane), and the netif->input RX fix (5c8d60c), already absorbed via CamM2325#20. Refs #14, #43 (FORK_PRS.md dj-oyu bundle's TAI64N item superseded), CamM2325#22 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3 tasks
fudio101
added a commit
to fugo101/microlink
that referenced
this pull request
Aug 18, 2026
* fix(derp): free the mbedTLS context on failed DERP connect ml_derp_connect()'s failure paths after the TLS phase begins only closed the raw socket — they never freed the mbedtls_ssl_context / mbedtls_ssl_config initialized just above. Each failed handshake leaked ~3-8 KB of internal heap, and microlink_rebind() reconnects DERP on every WiFi reconnect, so this compounds fast under captive-portal/bad-network conditions until no handshake can even allocate buffers. Extract derp_free_tls_state() (ssl/ssl_conf free + socket close) and route every post-init failure through a single `fail_tls` label via goto, reusing the same helper from ml_derp_disconnect()'s existing graceful-teardown path so the two call sites can't drift apart. Adapted from cplewes/microlink@38602ab0 and cplewes/microlink@b25b1eee. Not a literal cherry-pick: our ml_derp_conn_t already dropped the entropy/ctr_drbg fields as part of the mbedTLS 4.x / TF-PSA-Crypto migration (RNG is PSA-owned now), so derp_free_tls_state() frees only ssl/ssl_conf. Closes #14 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(derp,coord): absorb bugfix/hardening subset of upstream PR #22 Extends the DERP TLS-leak fix already on this branch with the rest of what upstream shipped as one verified commit (125b529): - DERP region fallback when HomeDERP isn't in the DERPMap (falls back to the first usable non-avoid region instead of a dead-end connect to the default host) - Pin DERP TLS to 1.2 — some relays' Let's Encrypt ECDSA cert trips the ESP-IDF mbedTLS TLS 1.3 signature-algorithm OID path even under VERIFY_NONE - Check every mbedTLS setup call's return value (config_defaults, ssl_setup, set_hostname) and abort via the existing fail_tls path instead of proceeding into handshake with a half-initialized config - Rework the DERP BIO to a plain blocking f_recv bounded by SO_RCVTIMEO, dropping mbedtls_ssl_conf_read_timeout entirely — avoids a BAD_INPUT_DATA failure the recv_timeout scheme hit on some relays' TLS 1.3 path Plus the other generic (non-headscale) fixes from the same upstream PR: - Hostinfo "OS" field corrected to "esp32" (was "linux") - New ML_STATE_AUTH_FAILED state: do_register() now actually fails registration on a control-plane Error/MachineAuthorized rejection instead of silently "succeeding" into a broken state (Node absent, confusing downstream "node not found"); the expired-key-with-no- auth_key path raises the same state - state_cb now fires on the RECONNECTING transition (it never did before) and doesn't get clobbered by it while AUTH_FAILED is active Adapted from cplewes/microlink@38602ab0/@b25b1eee (already on this branch) and CamM2325#22 commits 125b529, fcdc8d9, ae3d438, 8367c1e, 9ef10bb — not a literal cherry-pick, base has diverged too far, and our mbedTLS 4.x/PSA migration already dropped the entropy/ctr_drbg fields upstream's version still frees. Deliberately excluded (see UPSTREAM_PRS.md): the 3 headscale/custom- control-plane feature commits from CamM2325#22 (this fork targets Tailscale's own SaaS control plane), and the netif->input RX fix (5c8d60c), already absorbed via CamM2325#20. Refs #14, #43 (FORK_PRS.md dj-oyu bundle's TAI64N item superseded), CamM2325#22 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Adrian.Nguyen-Qualgo <nguyen.ndt@qualgo.net> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR makes MicroLink actually connect to a custom, non-Tailscale control plane end-to-end. It started as "build-time control plane config + TLS control transport" and grew, through bringing up a real ESP32 node on a headscale-based deployment (Lolipop Zero Trust Link), to cover the three further places where MicroLink assumed Tailscale's SaaS: the initial map poll, DERP region selection, and DERP TLS.
The README documents Headscale/Ionscale support, but several parts were hardcoded to Tailscale (control host + Noise key, the Stream=false initial fetch, DERP region 9 / derp9e.tailscale.com). Each commit is independent and defaults to current behavior.
Commits
Build-time custom control plane config with TLS transport
CONFIG_ML_CTRL_HOST, andCONFIG_ML_CTRL_TLSto wrap the ts2021 Noise handshake in TLS on port 443 (newml_coord_tls.c) for control planes exposed only over HTTPS. Cert not verified; the control plane is authenticated by the pinned Noise key, same trust model as the plain port-80 transport.ctrl_host/ctrl_noise_pubkeyinmicrolink_config_tRuntime counterparts, so the host and its
/key-fetched Noise key can be provisioned on-device rather than baked in.streaming_map_fetchfor controllers that ignore Stream=falseSome headscale control planes never respond to the initial Stream=false MapRequest (60s of zero bytes → empty MapResponse → reconnect loop). The native client starts a single Stream=true poll whose first message carries the full netmap. New flag (default false) switches to that: start long-poll with OmitPeers=false, read the initial netmap, parse Node/Peers/DERPMap, then connect DERP. Parsing is factored into shared helpers used by both paths.
DERP connection for custom control planes — three fixes:
derp9e.tailscale.com.mbedtls_ssl_handshake. A failingmbedtls_ssl_setup(typically-0x7F00ALLOC_FAILED under internal-RAM pressure) NULLsssl->conf, so handshake then returnedBAD_INPUT_DATAand the real cause (OOM) showed up as "Bad input parameters". Addedderp_tls_abort()to free the mbedTLS context (~20KB TLS buffers) on every failure path; the old code only closed the socket and leaked per retry. (App-side companion:CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=yto place TLS buffers in PSRAM.)VERIFY_NONE(TLS 1.3 can't skip cert processing). DERP relays accept TLS 1.2, so cap max version at 1.2.Testing
Brought up an ESP32-S3 node on a headscale-based control plane (443/TLS-only, DERP relays behind Let's Encrypt ECDSA certs): control TLS + Noise + registration + streaming netmap + DERP TLS + relay handshake all succeed, node shows online, and packets flow (rx/tx increasing). All commits build on ESP-IDF v5.4 / esp32s3.
With every new option off /
streaming_map_fetch=false, behavior against Tailscale's SaaS is unchanged.