feat(coord): consolidate H2 recv/JSON buffers into one dynamically-sized window

CLAUDE.md

@@ -85,7 +85,7 @@ components/microlink/
 
 **Cellular data path**: PPP is strongly preferred — gives real lwIP sockets, direct UDP, NAT traversal. AT socket bridge is automatic fallback when PPP auth fails. PPP throughput ~6.5 KB/s vs ~0.45 KB/s for AT bridge.
 
-**PSRAM usage**: H2 receive buffer (512KB) and JSON parse buffer (512KB) are allocated from PSRAM during coordination, then freed. Without PSRAM, reduce via `CONFIG_ML_H2_BUFFER_SIZE_KB=64` (supports ~30 peers max).
+**PSRAM usage**: A single H2 receive buffer (512KB default) is allocated from PSRAM during coordination, then freed. It serves double duty — HTTP/2 frames are received into it, then the extracted MapResponse JSON is compacted in place (no separate JSON buffer). At connect time the buffer is further clamped down to whatever's actually free in heap (min 64KB, never above the configured ceiling), so peak usage on a fragmented heap is often less than the configured size. Without PSRAM, reduce the ceiling via `CONFIG_ML_H2_BUFFER_SIZE_KB=64` (supports ~30 peers max).
 
 **NVS namespaces**: `"microlink"` for keys (machine key, WG key, DISCO key), `"ml_peers"` for peer cache. `microlink_factory_reset()` erases both — must call before `microlink_init()`.
 

README.md

@@ -207,16 +207,15 @@ pong from esp32-microlink (100.x.x.x) via DERP(dfw) in 150ms
 
 ### ESP32-S3 with PSRAM (Recommended)
 
-MapResponse buffers (H2 + JSON) are allocated from PSRAM only during coordination polling, then freed. Peak PSRAM usage is ~1MB (~12% of 8MB). Leaves 200KB+ SRAM free for your application.
+The MapResponse buffer (H2 receive + JSON parse, a single buffer reused in place) is allocated from PSRAM only during coordination polling, then freed. Peak PSRAM usage is ~512KB (~6% of 8MB) at defaults, and less than that whenever free heap is tight — the buffer is clamped to the largest available block (min 64KB) at connect time. Leaves 200KB+ SRAM free for your application.
 
 ### ESP32 without PSRAM
 
-Boards without PSRAM can reduce H2/JSON buffers to 64KB via menuconfig (sufficient for ~30 peers). Total SRAM usage: ~140KB. Suitable for simple sensor reporting, heartbeats, and small data payloads. Not recommended for large tailnets or memory-heavy applications.
+Boards without PSRAM can reduce the H2 buffer to 64KB via menuconfig (sufficient for ~30 peers). Total SRAM usage: ~90KB. Suitable for simple sensor reporting, heartbeats, and small data payloads. Not recommended for large tailnets or memory-heavy applications.
 
 ```ini
 # sdkconfig.defaults for ESP32 without PSRAM
 CONFIG_ML_H2_BUFFER_SIZE_KB=64
-CONFIG_ML_JSON_BUFFER_SIZE_KB=64
 CONFIG_ML_MAX_PEERS=8
 ```
 
@@ -588,8 +587,7 @@ MicroLink V2 Configuration
 | `ML_MAX_PEERS` | `16` | Maximum simultaneous active WireGuard tunnels (1-64). Each uses ~200 bytes. This is NOT the tailnet size limit — MicroLink tracks all peers (300+) but only maintains active tunnels to this many at once. Reduce to 8 for non-PSRAM. |
 | `ML_NVS_MAX_PEERS` | `64` | Peers cached in NVS flash (16-1024). Persists across reboots so DISCO probing starts immediately. Each entry: 92 bytes. LRU eviction when full. |
 | `ML_PRIORITY_PEER_IP` | Empty | Priority peer VPN IP (e.g., `100.x.y.z`). Guaranteed a WG slot even when peer table is full — LRU non-priority peer is evicted. Also settable via web UI. |
-| `ML_H2_BUFFER_SIZE_KB` | `512` | H2 receive buffer (64-2048 KB, PSRAM-backed). Size determines max tailnet: 64KB ≈ 30 peers, 512KB ≈ 300 peers, 2048KB ≈ 1200 peers. |
-| `ML_JSON_BUFFER_SIZE_KB` | `512` | JSON parse buffer (64-2048 KB, PSRAM-backed). cJSON DOM uses 2-3x raw JSON size. Match to H2 buffer. |
+| `ML_H2_BUFFER_SIZE_KB` | `512` | Ceiling for the single H2-receive-and-JSON-parse buffer (64-2048 KB, PSRAM-backed; clamped down to free heap at connect time, min 64KB). Size determines max tailnet: 64KB ≈ 30 peers, 512KB ≈ 300 peers, 2048KB ≈ 1200 peers. |
 
 #### Credentials
 
@@ -710,9 +708,10 @@ Zero-copy mode contributed by [dj-oyu](https://github.com/dj-oyu/microlink).
 
 ### "Failed to parse MapResponse JSON"
 - H2 buffer too small for your tailnet size
-- Increase `ML_H2_BUFFER_SIZE_KB` and `ML_JSON_BUFFER_SIZE_KB` in menuconfig
+- Increase `ML_H2_BUFFER_SIZE_KB` in menuconfig
 - For 300+ peers, use 512KB (default). For 600+, use 1024KB.
 - Ensure PSRAM is enabled: `CONFIG_SPIRAM=y`
+- Check the boot log for "H2 rx window" — if it's clamped well below your configured ceiling, free heap is tight at connect time, not just the ceiling itself
 
 ### PPP connection fails / falls back to AT socket
 - Check carrier APN is correct

components/microlink/Kconfig

@@ -74,8 +74,16 @@ menu "MicroLink V2 Configuration"
         default 512
         range 64 2048
         help
-            Size of the PSRAM-allocated buffer for receiving HTTP/2 frames
-            from the Tailscale control plane (MapResponse).
+            Ceiling for the single PSRAM-allocated buffer used to both
+            receive HTTP/2 frames from the Tailscale control plane
+            (MapResponse) and hold the extracted JSON for parsing --
+            the JSON is compacted in place into the same buffer, so no
+            separate JSON buffer is allocated.
+
+            At connect time this is further clamped down to whatever's
+            actually available in free heap (see ml_coord.c's
+            choose_h2_rx_window_size()), so actual usage is often less
+            than this ceiling.
 
             The initial MapResponse contains all peers in your tailnet.
             Each peer is roughly 1-2KB of JSON. After the initial response,
@@ -86,19 +94,6 @@ menu "MicroLink V2 Configuration"
             - 1024: ~600 peers (large enterprise tailnets)
             - 2048: ~1200 peers (very large tailnets, requires 8MB PSRAM)
 
-    config ML_JSON_BUFFER_SIZE_KB
-        int "JSON parse buffer size (KB)"
-        default 512
-        range 64 2048
-        help
-            Size of the PSRAM-allocated buffer for parsing JSON MapResponse
-            data. cJSON builds an in-memory DOM tree that uses 2-3x the
-            raw JSON size, so this should be at least as large as the
-            HTTP/2 buffer.
-
-            Match this to ML_H2_BUFFER_SIZE_KB unless you have a reason
-            to set them differently.
-
     menu "Cellular Modem"
 
         config ML_ENABLE_CELLULAR

components/microlink/include/microlink_internal.h

@@ -142,9 +142,11 @@ extern "C" {
  * transport health looks fine). */
 #define ML_CTRL_STREAM_STALE_MS         300000
 
-/* Large tailnet buffer sizes (PSRAM-allocated, configurable via menuconfig) */
+/* Large tailnet buffer size (PSRAM-allocated, configurable via menuconfig).
+ * Ceiling for the single H2-receive-and-JSON-parse buffer in do_fetch_peers()
+ * -- ml->h2_rx_window_size is clamped to this at connect time based on free
+ * heap (see choose_h2_rx_window_size() in ml_coord.c). */
 #define ML_H2_BUFFER_SIZE       (CONFIG_ML_H2_BUFFER_SIZE_KB * 1024)
-#define ML_JSON_BUFFER_SIZE     (CONFIG_ML_JSON_BUFFER_SIZE_KB * 1024)
 
 /* Noise protocol */
 #define ML_NOISE_KEY_LEN        32
@@ -451,6 +453,9 @@ struct microlink_s {
     /* Coordination socket (owned exclusively by coord task) */
     int coord_sock;
     uint32_t h2_next_stream_id;         /* Next H2 stream ID for endpoint updates (odd, starts at 7) */
+    /* Runtime H2 recv/JSON-parse window, chosen per connect cycle by
+     * choose_h2_rx_window_size() from free heap; never exceeds ML_H2_BUFFER_SIZE. */
+    uint32_t h2_rx_window_size;
     /* ml_get_time_ms() of the most recent DATA frame received on the
      * long-poll map stream (H2 stream 5) specifically -- real MapResponses
      * and the ~60s mapSession keepalives, nothing else. Unlike the
@@ -617,7 +622,7 @@ int ml_h2_build_headers_frame(uint8_t *out, size_t out_size,
 int ml_h2_build_data_frame(uint8_t *out, size_t out_size,
                             const uint8_t *data, size_t data_len,
                             uint32_t stream_id, bool end_stream);
-int ml_h2_build_preface(uint8_t *out, size_t out_size);
+int ml_h2_build_preface(uint8_t *out, size_t out_size, uint32_t window_size);
 int ml_h2_build_settings_ack(uint8_t *out, size_t out_size);
 int ml_h2_build_window_update(uint8_t *out, size_t out_size,
                                uint32_t stream_id, uint32_t increment);

components/microlink/src/microlink.c

@@ -393,6 +393,7 @@ microlink_t *microlink_init(const microlink_config_t *config) {
     ml->stun_sock = -1;
     ml->stun_sock6 = -1;
     ml->derp.sockfd = -1;
+    ml->h2_rx_window_size = ML_H2_BUFFER_SIZE;
 
     /* Resolve timing (0 = use defaults from #defines) */
     ml->t_disco_heartbeat_ms = ml->config.disco_heartbeat_ms ? ml->config.disco_heartbeat_ms : ML_DISCO_HEARTBEAT_MS;

components/microlink/src/ml_coord.c

@@ -655,15 +655,41 @@ static void process_proactive_frames(microlink_t *ml, ml_noise_state_t *noise) {
  * State: H2_PREFACE - Send HTTP/2 connection preface (Noise-encrypted)
  * ========================================================================== */
 
+/* Choose the H2 recv/JSON-parse window for this connect cycle: the largest
+ * contiguous free block we can find (SPIRAM preferred, internal as fallback),
+ * minus a safety margin, clamped to [64KB, ML_H2_BUFFER_SIZE] and rounded
+ * down to 1KB. do_fetch_peers() allocates exactly one buffer this size and
+ * uses it for both the raw H2 receive and the compacted JSON -- keeping this
+ * window under what's actually free avoids fragmenting the rest of PSRAM on
+ * RAM-constrained boards. Ported from djorr5/microlink's `67b230b2`. */
+static void choose_h2_rx_window_size(microlink_t *ml) {
+    size_t psram_largest = heap_caps_get_largest_free_block(MALLOC_CAP_SPIRAM);
+    size_t internal_largest = heap_caps_get_largest_free_block(MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT);
+    size_t largest_any = psram_largest > internal_largest ? psram_largest : internal_largest;
+
+    const size_t margin = 32 * 1024;
+    const size_t floor = 64 * 1024;
+    size_t window = (largest_any > margin) ? (largest_any - margin) : 0;
+    if (window < floor) window = floor;
+    if (window > ML_H2_BUFFER_SIZE) window = ML_H2_BUFFER_SIZE;
+    window &= ~(size_t)1023;  /* round down to 1KB */
+
+    ml->h2_rx_window_size = (uint32_t)window;
+    ESP_LOGI(TAG, "H2 rx window: %luKB (largest free block %luKB, ceiling %luKB)",
+             (unsigned long)(window / 1024), (unsigned long)(largest_any / 1024),
+             (unsigned long)(ML_H2_BUFFER_SIZE / 1024));
+}
+
 static int do_h2_preface(microlink_t *ml, ml_noise_state_t *noise) {
     int64_t t_h2_start = esp_timer_get_time();
+    uint32_t h2_window = ml->h2_rx_window_size ? ml->h2_rx_window_size : ML_H2_BUFFER_SIZE;
 
     /* Build H2 preface (24+6=30) + SETTINGS with INITIAL_WINDOW_SIZE (9+6=15)
      * + SETTINGS_ACK (9) + connection-level WINDOW_UPDATE (13) = 67 bytes */
     uint8_t h2_init[128];
     int pos = 0;
 
-    int preface_len = ml_h2_build_preface(h2_init, sizeof(h2_init));
+    int preface_len = ml_h2_build_preface(h2_init, sizeof(h2_init), h2_window);
     if (preface_len < 0) return -1;
     pos = preface_len;
 
@@ -675,7 +701,7 @@ static int do_h2_preface(microlink_t *ml, ml_noise_state_t *noise) {
      * beyond the 65535 default. SETTINGS INITIAL_WINDOW_SIZE only sets per-stream
      * window; the connection-level window starts at 65535 and must be explicitly
      * expanded with WINDOW_UPDATE on stream 0. */
-    uint32_t conn_window_delta = ML_H2_BUFFER_SIZE - 65535;
+    uint32_t conn_window_delta = h2_window - 65535;
     if (conn_window_delta > 0) {
         int wu_len = ml_h2_build_window_update(h2_init + pos, sizeof(h2_init) - pos,
                                                 0, conn_window_delta);
@@ -689,7 +715,7 @@ static int do_h2_preface(microlink_t *ml, ml_noise_state_t *noise) {
     }
 
     ESP_LOGI(TAG, "H2 preface sent (%d bytes, conn window=%luKB)",
-             pos, (unsigned long)(ML_H2_BUFFER_SIZE / 1024));
+             pos, (unsigned long)(h2_window / 1024));
 
     /* Read and process server's response (SETTINGS, SETTINGS_ACK, WINDOW_UPDATE, etc.) */
     uint8_t recv_buf[4096];
@@ -1495,13 +1521,22 @@ static int do_fetch_peers(microlink_t *ml, ml_noise_state_t *noise) {
     /* Read MapResponse - accumulate ALL decrypted Noise frames first, then parse H2.
      * This is critical because a single H2 frame can span multiple Noise frames
      * (v1 does the same with h2_buffer).
-     * Smart timeout: extend to 60s for large tailnets (300+ peers = 240KB+). */
-    uint8_t *h2_recv = ml_psram_malloc(ML_H2_BUFFER_SIZE);  /* 512KB for 300+ peer tailnets */
+     * Smart timeout: extend to 60s for large tailnets (300+ peers = 240KB+).
+     *
+     * Single PSRAM buffer for both the raw H2 receive and the compacted JSON --
+     * once all frames are accumulated, DATA payloads are extracted in place via
+     * memmove() (see below) instead of copying into a second buffer, halving
+     * peak footprint. Sized to the runtime window choose_h2_rx_window_size()
+     * picked for this connect cycle (<= ML_H2_BUFFER_SIZE). Ported from
+     * djorr5/microlink's `67b230b2` piece (a); adapted to keep the per-iteration
+     * frame_buf scratch read (below) instead of reading straight into h2_recv --
+     * this fork's noise_recv() returns -1 without draining the ciphertext off
+     * the socket when the frame doesn't fit the caller's buffer, which would
+     * desync the stream if the destination window shrinks near the tail end. */
+    uint32_t h2_window = ml->h2_rx_window_size ? ml->h2_rx_window_size : ML_H2_BUFFER_SIZE;
+    uint8_t *h2_recv = ml_psram_malloc(h2_window);
     if (!h2_recv) return -1;
     size_t h2_total = 0;
-
-    uint8_t *resp_buf = ml_psram_malloc(ML_JSON_BUFFER_SIZE);
-    if (!resp_buf) { free(h2_recv); return -1; }
     size_t json_total = 0;
 
     /* Set extended recv timeout for large MapResponse (60 seconds) */
@@ -1528,7 +1563,7 @@ static int do_fetch_peers(microlink_t *ml, ml_noise_state_t *noise) {
         }
 
         /* Append decrypted data to h2_recv */
-        if (h2_total + frame_len < ML_H2_BUFFER_SIZE) {
+        if (h2_total + frame_len < h2_window) {
             memcpy(h2_recv + h2_total, frame_buf, frame_len);
             h2_total += frame_len;
             window_consumed += frame_len;
@@ -1596,7 +1631,10 @@ static int do_fetch_peers(microlink_t *ml, ml_noise_state_t *noise) {
              (int)(h2_total / 1024),
              (unsigned long)(ml_get_time_ms() - recv_start_ms));
 
-    /* Now parse complete H2 frames from accumulated buffer */
+    /* Now parse complete H2 frames from accumulated buffer, compacting DATA
+     * payloads into the front of the SAME buffer (json_total <= fpos always,
+     * since fpos also advances past 9-byte frame headers and non-DATA frames
+     * that json_total never counts -- memmove handles the overlap safely). */
     int fpos = 0;
     while (fpos + 9 <= (int)h2_total) {
         uint32_t f_len = (h2_recv[fpos] << 16) | (h2_recv[fpos + 1] << 8) | h2_recv[fpos + 2];
@@ -1617,15 +1655,12 @@ static int do_fetch_peers(microlink_t *ml, ml_noise_state_t *noise) {
         }
 
         if (f_type == 0x00 && f_len > 0) {  /* DATA frame */
-            if (json_total + f_len < ML_JSON_BUFFER_SIZE) {
-                memcpy(resp_buf + json_total, h2_recv + fpos, f_len);
-                json_total += f_len;
-            }
+            memmove(h2_recv + json_total, h2_recv + fpos, f_len);
+            json_total += f_len;
         }
 
         fpos += f_len;
     }
-    free(h2_recv);
 
     /* Send connection-level WINDOW_UPDATE to replenish HTTP/2 flow control.
      * Stream 3 is already closed (END_STREAM received), so only update stream 0.
@@ -1639,7 +1674,7 @@ static int do_fetch_peers(microlink_t *ml, ml_noise_state_t *noise) {
 
     if (json_total == 0) {
         ESP_LOGW(TAG, "Empty MapResponse");
-        free(resp_buf);
+        free(h2_recv);
         return -1;
     }
 
@@ -1650,20 +1685,20 @@ static int do_fetch_peers(microlink_t *ml, ml_noise_state_t *noise) {
         int dump = json_total < 32 ? (int)json_total : 32;
         char hexbuf[97];
         for (int i = 0; i < dump; i++) {
-            sprintf(hexbuf + i * 3, "%02x ", resp_buf[i]);
+            sprintf(hexbuf + i * 3, "%02x ", h2_recv[i]);
         }
         hexbuf[dump * 3] = '\0';
         ESP_LOGI(TAG, "MapResponse first %d bytes (hex): %s", dump, hexbuf);
     }
 
     /* Check for length prefix (Tailscale binary framing: 4-byte big-endian length before JSON) */
-    char *parse_start = (char *)resp_buf;
+    char *parse_start = (char *)h2_recv;
     size_t parse_len = json_total;
 
     /* Find the start of JSON - look for '{' in first 8 bytes */
     int json_offset = -1;
     for (int i = 0; i < 8 && i < (int)json_total; i++) {
-        if (resp_buf[i] == '{') {
+        if (h2_recv[i] == '{') {
             json_offset = i;
             break;
         }
@@ -1686,7 +1721,7 @@ static int do_fetch_peers(microlink_t *ml, ml_noise_state_t *noise) {
     if (!map_json) {
         const char *err = cJSON_GetErrorPtr();
         ESP_LOGE(TAG, "MapResponse JSON parse failed near: %.50s", err ? err : "unknown");
-        free(resp_buf);
+        free(h2_recv);
         return -1;
     }
 
@@ -1884,7 +1919,7 @@ static int do_fetch_peers(microlink_t *ml, ml_noise_state_t *noise) {
     }
 
     cJSON_Delete(map_json);
-    free(resp_buf);
+    free(h2_recv);
 
     int64_t t_map_done = esp_timer_get_time();
     ESP_LOGI(TAG, "[TIMING] MapResponse recv+parse: %lld ms (total map: %lld ms, %dKB)",
@@ -2354,6 +2389,7 @@ void ml_coord_task(void *arg) {
              * before sending our H2 preface. Extracts nodeKeyChallenge
              * and adjusts rx_nonce. */
             process_proactive_frames(ml, &noise);
+            choose_h2_rx_window_size(ml);
             state = COORD_H2_PREFACE;
             break;
 

components/microlink/src/ml_h2.c

@@ -121,7 +121,7 @@ static int hpack_literal_new(uint8_t *out, const char *name, const char *value)
  *
  * Returns total bytes written, or -1 on error.
  */
-int ml_h2_build_preface(uint8_t *out, size_t out_size) {
+int ml_h2_build_preface(uint8_t *out, size_t out_size, uint32_t window_size) {
     /* Preface (24) + SETTINGS frame header (9) + INITIAL_WINDOW_SIZE setting (6) = 39 bytes */
     if (out_size < H2_PREFACE_LEN + 9 + 6) return -1;
 
@@ -131,11 +131,11 @@ int ml_h2_build_preface(uint8_t *out, size_t out_size) {
     memcpy(out, H2_CONNECTION_PREFACE, H2_PREFACE_LEN);
     pos += H2_PREFACE_LEN;
 
-    /* SETTINGS frame with INITIAL_WINDOW_SIZE = ML_H2_BUFFER_SIZE
+    /* SETTINGS frame with INITIAL_WINDOW_SIZE = window_size (the caller's
+     * runtime-chosen H2 recv window, <= ML_H2_BUFFER_SIZE)
      * Each setting is 6 bytes: 2-byte ID + 4-byte value (RFC 7540 Section 6.5.1)
      * Without this, the server uses the HTTP/2 default of 65535 bytes (64KB),
      * which is too small for large MapResponses (100KB+ on 60+ peer tailnets). */
-    uint32_t window_size = ML_H2_BUFFER_SIZE;
     pos += write_frame_header(out + pos, 6, H2_FRAME_SETTINGS, 0, 0);
 
     /* INITIAL_WINDOW_SIZE (0x04) */