Lab/scale sim

BUGSVULNS.MD

@@ -4144,7 +4144,71 @@ without, for 4 to 6 packets per second per node at rest instead of 2.4 to 4.4).
 machine: the simulator needs more cores than the 8 it has once the mesh is
 chatty, and at rest it is the first number to check again.
 
-## 109 — medium — Two builds raced to publish the base image, and the older won
+## 109 — high — A node was linked to its true nearest neighbour about one time in two, and the others could not be found
+
+> **Status — fixed** (leaf set). Tests:
+> `tests/test_neighborhood.py::TestTheLeafSet`.
+
+**Where** — `src/node.py`: `_maintain_neighbors`.
+
+**What happens** — maintenance only insisted on `_NEIGHBOR_FLOOR` = 3 links, and
+stopped searching once it held them, whichever nodes they were. A node that
+joined through two random members held a handful of links in no particular part
+of the id space, was known to a handful of nodes, and had nobody near it in id
+space that knew it. Both ways of reaching it failed: greedy forwarding ends at
+the node nearest the target and that node had no link to it, and a lookup
+cannot find a node its nearest neighbours have never heard of. The `--explain`
+output of `tests/tools/mesh_sim.py` on 1000 nodes showed it: the undelivered
+first contacts ended at nodes holding 3 to 6 links, with a routing table of 3
+to 9 entries, known to 3 to 7 nodes in the mesh. `leafcheck` measured it
+directly: of each node's true 1, 3 and 8 nearest (computed over all ids), 54 %,
+53 % and 47 % were linked.
+
+Two changes that looked like the fix were measured and changed nothing: looking
+the target up before the first forward (89 and 89 of 100 against 90 and 88, and
+slower), and a rule that looked the target up at a local minimum of the
+distance. The cause was not the way packets are carried.
+
+**Fixed** — the nodes we hold nearest in id space, `_LEAF_SIZE` = 8, are a set
+the node insists on. While they are not all linked, or are not what they were at
+the last cycle, a cycle searches: it looks up its own id (which is also how
+those nodes come to know it) and dials what is missing. One that refused
+`_LEAF_GIVE_UP` times is not waited for, and the usual back-off stretches the
+cycles while nothing changes. On 1000 nodes over twelve processes with the
+crypto stubbed (`--fast`): the true 8 nearest linked 98 % of the time (from
+47 %), first contacts 99, 100 and 100 of 100 over three seeds (from 88 to 92), one retry in a hundred
+(from eleven), the same 4 packets per second per node at rest.
+
+**Still open** — nothing larger than 2000 nodes was run (memory: about 3 MB per
+simulated node). Hostile nodes inserting near ids to hold a victim in the
+searching regime are bounded by `_LEAF_GIVE_UP` and the back-off, but were not
+tried.
+
+## 110 — medium — Every link stayed open for ever, and keeping them open was the whole traffic of an idle node
+
+> **Status — fixed.** Tests: `tests/test_link_prune.py`.
+
+**Where** — `src/node.py`: nothing closed a link that nobody used.
+
+**What happens** — a node on 1000 nodes held 28 to 36 links (the leaf set and
+two links per band of distance, plus every link a conversation or a lookup had
+ever opened) and, with nothing to say, sent 3.6 packets per second and 0.38 KiB/s
+to keep them alive: `PING` and `PONG`, nothing else. The energy and performance
+bar moved the cadence of those probes and not the number of links.
+
+**Fixed** — `_prune_idle_links` (see `routing.md`), the three numbers on the
+bar, and `LINK_RELEASE`. A first version without the message closed links the
+far end then redialled, which cycled: 0.05 handshakes per second per node and
+5 KiB/s, ten times worse than doing nothing. At rest after the settling time on
+1000 nodes: 15 links per node, 1.7 packets per second, 0.22 KiB/s, 99 % of the
+true 8 nearest linked, 100 of 100 first contacts.
+
+**Still open** — the first minutes after a start (the settling time) still cost
+more than the rest: handshakes and the lookups of every new node (29 KiB/s
+per node at 30 nodes, measured earlier). Both ends of a link still
+probe it; one probe each way would halve the rest (the quality estimates of each
+side come from its own probes, so that is not free).
+## 111 — medium — Two builds raced to publish the base image, and the older won
 
 > **Status — fixed.** Tests: `tests/test_workflows_pinned.py`
 > (`test_what_publishes_latest_does_it_one_run_at_a_time`).
@@ -4162,4 +4226,3 @@ so the only symptom was that later builds broke.
 **Fixed** — both publishers run in a concurrency group that never cancels a run
 in progress, so they finish in the order they were queued and the newest commit
 publishes last. The base was republished from `main` by hand.
-

Docs/Architecture/protocol.md

@@ -140,6 +140,7 @@ and gone once they update.
 | SPEED_PROBE | 0x31 | padding sent to measure a link by **loading** it. Answered only on a direct authenticated link, only when addressed to us, and only up to `_SPEED_CHUNK` — a stranger must never be able to make this node generate traffic, and a peer must never be able to make it answer *to somebody else* |
 | SPEED_ECHO | 0x32 | the same padding back, **one for one and byte for byte**. The handler copies the payload rather than producing one, because a reflector that answers more than it was sent is an amplifier — which is the single worst thing this pair could be. The answering side keeps its own ceiling per identity; the asking side cannot raise it |
 | STOP_RELAY | 0x33 | a destination's **signed** "stop relaying that node to me", for at most `stop_relay.MAX_TTL` — **direct**, re-stamped at each hop: a relay passes the same record back to the link that fed it the refused traffic. Honoured only for the destination that signed it (see [`security.md`](security.md#refusing-a-node-all-the-way-back-stop_relaypy)), negotiated as `stoprelay` |
+| LINK_RELEASE | 0x34 | **direct**, empty: "I am closing this link on purpose". The receiver only stops treating its end as a loss to chase (`peer.released`, read by `_note_node_lost`); a payload is charged as abuse and ignored. Sent by `_prune_idle_links` to a peer that announced `linkrelease`, never to one that did not |
 | INVITE_OFFER | 0x30 | a **rendezvous** an inviter leaves with a relay: "expect a seek for this code, and here is the proof I authorised it". Authenticated link only, and only ever about its own sender. It is what lets an invitation reach a node with no address of its own out of a string short enough for a QR code — the ML-DSA key and signature a relayed invitation needs stay with the relay, and the joiner sends 40 bytes |
 
 Groupings (constants):

Docs/Architecture/routing.md

@@ -400,13 +400,29 @@ then in a loop), but what it does depends on what we already hold:
   neighbourhood, then a directed dial of the `_NEIGHBOR_TARGET = 5` XOR-nearest
   entries it has no session with (`_connect_routing`, IPv6 then IPv4). It is a
   directed dial, not a broadcast.
-- **Quiet** — as soon as `_NEIGHBOR_FLOOR` links are held: no lookup, no dial. A
+- **Quiet** — as soon as `_NEIGHBOR_FLOOR` links are held *and the leaf set is
+  linked and unchanged* (below): no lookup, no dial, bar promotions and fingers. A
   node searching for its own id forever is nothing but traffic, and a mesh that
   never settles is a mesh an adversary can keep busy. Losing one of the three
   links (`_drop_failed_peer`, keepalive) restarts the search immediately.
 - `force=True` (join/`bootstrap`) imposes a full searching cycle whatever we
   already hold: a fresh table has to fill up.
 
+### The leaf set: a node is linked to its nearest neighbours (`_LEAF_SIZE`)
+
+Greedy forwarding ends at the node nearest the target, and a lookup finds a node
+through the nodes near it. Both rely on every node being linked to the nodes
+nearest it in id space and known to them. The maintained set used to be three
+links of whatever kind (`_NEIGHBOR_FLOOR`), which the join gave by luck, and
+measured on 1000 nodes the true nearest neighbour was linked 54 % of the time
+(BUGSVULNS 109).
+
+Each cycle takes the `_LEAF_SIZE` nearest nodes in the routing table. If they
+are not all linked, or are not the ones of the last cycle, the cycle is a
+searching one (a lookup of our own id, then a dial of whatever is missing); if
+they are, the node is quiet. An entry that refused `_LEAF_GIVE_UP` dials is not
+waited for. The cycle interval stretches while nothing changes, as before.
+
 ### Links are spread over distance, and a lookup does not buy one (`_finger_candidates`, `_UPGRADE_LINK_SOFT`)
 
 Greedy forwarding picks the neighbour nearest the target, and only makes
@@ -429,6 +445,33 @@ band is the bit length of the XOR distance to us.
 At rest on 240 nodes: about 3 packets per second per node and 25 to 30 links,
 against 20 packets per second and a full mesh before.
 
+### Links nobody needs are closed (`_prune_idle_links`, `LINK_RELEASE`)
+
+An open link costs a probe each way every twenty seconds, and nothing used to
+close one. On 1000 nodes the idle traffic of a node was 36 links and 3.6 packets
+per second, all `PING` and `PONG` (BUGSVULNS 110). What a node holds for good is
+its structure: the leaf set (`leaf_size`, the nearest nodes) and `finger_links`
+links in each band of distance, the oldest. The others are closed after
+`link_idle_minutes` without a conversation on them (`peer.last_used`, touched by
+`DATA` and the E2E handshake, sent, received or relayed; keepalives and lookups
+do not count), two per keepalive sweep.
+
+Left alone: a node we hold several links to (a bundle), relay-only and probe
+links, the last `max(_NEIGHBOR_FLOOR, leaf_size)` links, a node whose leaf set
+we probably belong to (fewer than `leaf_size` nodes in our table nearer to it
+than we are), and a node that redialled after we closed (`peer.peer_wants`,
+remembered `_PEER_WANTS_TTL` in `_idle_closed`, bounded). A closed link is not
+chased by the side that closed it, and a peer that announced `linkrelease` is
+sent `LINK_RELEASE` first so that it does not chase it either. Without that
+message the far end saw a loss, redialled, and the pair cycled: measured as
+0.05 handshakes per second per node, 5 KiB/s against 0.4 before pruning.
+
+The three numbers are on the energy and performance bar (`power.py`): from four
+near neighbours, one link per band and two idle minutes at the frugal end, to
+twelve, three and never closing at the other. Measured at rest on 1000 nodes,
+after the settling time: 15 links per node and 1.7 packets per second (0.22
+KiB/s), against 28 to 36 and 3.6 (0.38 KiB/s).
+
 ### Promoting a node seen in transit
 
 The maintained set (`_neighbor_slots`) = the `_NEIGHBOR_FLOOR` nearest live

Docs/Architecture/scale.md

@@ -17,7 +17,7 @@ test for anything that grows.
 | # | Point | State |
 |---|---|---|
 | 1 | Membership chains and the routing answers that carry them | **fixed** (compaction, lookup end); smaller chains on the wire in progress |
-| 2 | Links per node, and what probing them costs a relay | **fixed** (probe budget; links spread over distance and no link bought by a lookup, BUGSVULNS 108) |
+| 2 | Links per node, and what probing them costs a relay | **fixed** (probe budget; links spread over distance and no link bought by a lookup, BUGSVULNS 108; every node linked to its nearest neighbours, BUGSVULNS 109) |
 | 3 | Gossip planes (addresses, revocations, abuse, releases) | open |
 | 4 | Distributed storage (DHT, names, package directory) | open |
 | 5 | Per-remote-node tables | partly reviewed (below) |

Docs/WebConsole/guide

@@ -1192,7 +1192,7 @@ console cannot produce a file the node would refuse to start on.
 Above the form, one bar with ten steps from *Minimal* to *Maximum*: how much
 this node may spend to stay quick. A step is a named set of **live** settings
 (`src/power.py`: the four keepalive bounds, `mlo_always`, `dynamic_address`,
-`update_check_minutes`, `update_when_active`, `route_hold_minutes`) applied
+`update_check_minutes`, `update_when_active`, `route_hold_minutes`, and the link budget: `leaf_size`, `finger_links`, `link_idle_minutes`) applied
 together through
 `config.profile`, and remembered like any other edit. Dragging previews the
 step's name and a one-sentence description; letting go applies it. Step 6,

TEST.md

@@ -117,6 +117,21 @@ packets sent and relayed, and the **lateness of the slowest worker's event loop*
 Past a loop lag p99 of a few hundred milliseconds the figures describe the
 machine. It is a measurement tool, not a test: nothing in `pytest` runs it.
 
+Options worth knowing:
+
+* `--fast` replaces `oqs` with `tests/tools/fake_oqs.py`: the same key, signature
+  and ciphertext sizes, hashes instead of lattices. It gave the same figures as
+  the real thing at 240 nodes and reaches 2000 on twelve cores (about 3 MB per
+  simulated node, and a join phase that takes minutes: it is sequential).
+* After the settling time it prints, for every node, how many of its **true**
+  1, 3 and 8 nearest nodes (computed over all ids) it is linked to and has in its
+  routing table. A routing algorithm that depends on that is checked by it
+  directly, not by its side effects.
+* `--explain` says what both ends of each undelivered first contact looked like:
+  links, routing table size, and how many nodes know the receiver.
+* The same seed moves by five first contacts or more from run to run. Compare
+  ranges over several seeds, never one run against one run.
+
 ### Hostile requests at the console's API
 
 `tests/tools/api_fuzz.py` signs in to a **throwaway** node and sends every route
@@ -272,6 +287,12 @@ tests/
 ├── test_loop_backtrack.py                             — a routed packet that
 │     comes back round a cycle leaves by another neighbour, and what that may
 │     cost is bounded (replays, fanout, the book)
+├── test_link_prune.py — idle links closed (structure, links in use, bundles, relay and
+│     probe links, nodes whose leaf we are, redialled links kept), told with
+│     LINK_RELEASE and not chased, bounded; the link budget's bounds
+├── test_neighborhood.py — the maintained neighbourhood, including the leaf set
+│     (searching until the nearest known nodes are all linked, quiet after, bounded,
+│     not kept searching by a node nobody can reach)
 ├── test_routing_stability.py                          — routing regressions:
 │     the size of a FOUND_NODE, acquiring a route outside the receive loop,
 │     the return path learned from traffic, bounded teardown, a first

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.backends.legacy:build"
 
 [project]
 name = "nmesh"
-version = "0.4.111"
+version = "0.4.113"
 license = "GPL-3.0-only"
 
 [tool.pytest.ini_options]

src/config.py

@@ -249,6 +249,30 @@ def _as_route_hold(raw: str) -> int:
     return value
 
 
+def _as_leaf_size(raw: str) -> int:
+    """How many of the nearest nodes this one keeps a link with."""
+    value = int(str(raw).strip())
+    if not 2 <= value <= 32:
+        raise ValueError("between 2 and 32 nodes")
+    return value
+
+
+def _as_finger_links(raw: str) -> int:
+    """Links kept in each band of distance that has nodes in it."""
+    value = int(str(raw).strip())
+    if not 1 <= value <= 4:
+        raise ValueError("between 1 and 4 links")
+    return value
+
+
+def _as_link_idle(raw: str) -> int:
+    """Minutes a link nobody needs is kept. 0 keeps every link."""
+    value = int(str(raw).strip())
+    if not 0 <= value <= 1440:
+        raise ValueError("between 0 and 1440 minutes (0 = never close one)")
+    return value
+
+
 def _as_branch(raw: str) -> str:
     """A branch to follow instead of the published releases. Empty = releases.
 
@@ -316,6 +340,17 @@ def _as_path(raw: str):
                         "Minutes a node reached through a relay keeps that "
                         "relay before moving to another healthy one (0 = "
                         "spread its traffic over every relay at once)"),
+    "leaf_size":       (_as_leaf_size, 8, True,
+                        "How many of the nearest nodes (by id) this one keeps "
+                        "a permanent link with; fewer saves keepalives, more "
+                        "makes it surer to be found"),
+    "finger_links":    (_as_finger_links, 2, True,
+                        "Permanent links kept in each distance band that has "
+                        "nodes in it; they are what a packet crosses the mesh on"),
+    "link_idle_minutes": (_as_link_idle, 10, True,
+                        "Minutes a link that is not one of the permanent ones "
+                        "is kept without a conversation on it (0 = never "
+                        "close one)"),
     "mlo_always":      (_as_bool, False, True,
                         "Keep multi-link operation running even when nothing "
                         "is using this node (otherwise it wakes with the "
@@ -457,6 +492,7 @@ def _as_path(raw: str):
 LIVE = frozenset({
     "punch_keepalive", "lan_discovery", "transport_balance", "dynamic_address",
     "mlo_always", "mlo_skew_ms", "mlo_drop_percent", "route_hold_minutes",
+    "leaf_size", "finger_links", "link_idle_minutes",
     "keepalive_fast_min_ms", "keepalive_fast_max_ms",
     "keepalive_slow_min_ms", "keepalive_slow_max_ms",
     "pseudo", "no_abuse_gossip", "release_quorum", "release_auto_publish",

src/control/modules/settings.py

@@ -59,7 +59,7 @@
                ("punch_keepalive",), ("lan_discovery",),
                ("transport_balance",), ("dynamic_address",),
                ("mlo_always",), ("pseudo",), ("no_abuse_gossip",),
-               ("route_hold_minutes",))
+               ("route_hold_minutes",), ("leaf_size", "finger_links", "link_idle_minutes"))
 
 
 async def apply_live(node, values: dict, changed) -> tuple[dict, list]:
@@ -104,6 +104,10 @@ async def attempt(label, call):
     if "route_hold_minutes" in changed:
         await attempt("route_hold_minutes",
                       lambda: node.set_route_hold(values["route_hold_minutes"] * 60))
+    if changed & {"leaf_size", "finger_links", "link_idle_minutes"}:
+        await attempt("link budget", lambda: node.set_link_budget(
+            leaf_size=values["leaf_size"], finger_links=values["finger_links"],
+            idle_minutes=values["link_idle_minutes"]))
     if "dynamic_address" in changed:
         await attempt("dynamic_address",
                       lambda: node.set_dynamic_address(values["dynamic_address"]))

src/features.py

@@ -118,17 +118,22 @@
 # never sent one — and never judged for not honouring one.
 STOP_RELAY = "stoprelay"
 
+# Telling the far end that a link is being closed on purpose, so that it does
+# not take it for a loss and chase it (`MeshNode._prune_idle_links`). A node that
+# never announced it is simply closed on, as before.
+LINK_RELEASE = "linkrelease"
+
 SPOKEN = frozenset({CORE, KADEMLIA, E2E, DIRECTORY, PSEUDO, CATALOG, RELEASE,
                     PUNCH, REACH, RELAY, RENDEZVOUS, RENEW, REVOKE, ABUSE,
                     KEEPALIVE, MLO, PACKAGES, HANDOVER, SPEEDTEST,
-                    BLAKE2B_IDS, STOP_RELAY})
+                    BLAKE2B_IDS, STOP_RELAY, LINK_RELEASE})
 
 # Planes added *after* this negotiation existed. Silence about one of these is
 # not a node from before the name — it is a node that has never heard of it,
 # and sending it the new thing is exactly what rule 2 forbids. See
 # ``MeshNode.peer_announces``, which is the predicate these are asked through.
 SINCE_NEGOTIATION = frozenset({KEEPALIVE, MLO, PACKAGES, HANDOVER, RENDEZVOUS,
-                               SPEEDTEST, BLAKE2B_IDS, STOP_RELAY})
+                               SPEEDTEST, BLAKE2B_IDS, STOP_RELAY, LINK_RELEASE})
 
 
 class FeatureError(Exception):