testbed/simnet: discv5 testbed via simnet#66
Draft
srene wants to merge 16 commits into
Draft
Conversation
srene
changed the title
srene
force-pushed
the
feat/simnet-testbed-v2
branch
from
May 12, 2026 13:03
266fa32 to
144ce70
Compare
srene
marked this pull request as draft
May 12, 2026 13:09
In-process discv5 testbed via marcopolo/simnet with TopDisc incremental-deployment validation, scaling, and search/registration instrumentation. ENR-flag handling is type-agnostic (deletes by key), so it builds against topdisc's TopicDiscovery. go.mod uses a local replace (../..) so the testbed exercises this branch's protocol code.
…drains IsDone gated on queriesWithoutNewNodes >= 4 after verifying the unasked set was empty. That counter only advances when AddNodes runs without finding novel nodes, but once every bucket's `new` set is empty QueryTarget returns nil, so no query is ever dispatched, no response arrives, AddNodes is never called, and the counter cannot reach 4. The search loop then sits with only s.quit progressable, leaking the iterator until Close. Treat "no unasked nodes and no buffered results" as the terminating condition. Drop queriesWithoutNewNodes — nothing else read it. Closes #27.
The previous QueryTarget implementation tracked a warm-up frontier and randomized the next pick across buckets that had received at least one query. When the farthest bucket was empty and numRequests==0, the frontier could not advance and the search stalled even when closer buckets held unasked nodes (issue #65). Replace with a deterministic linear walk: scan buckets far -> close and return the first unasked node. Each call re-walks from the farthest bucket so AddNodes responses that land in earlier buckets are picked up before the search descends to closer ones. Empty buckets are skipped by construction. Drop the numRequests counter (no longer read) and the math/rand import. Closes #65.
The previous commit's strict far→close drain removed cross-bucket query spreading. Once a closer bucket became active, farther buckets that refilled via AddNodes would take over completely, pausing all closer work until they drained again. In a session where aux responses keep seeding far buckets faster than they drain, the close-to-topic buckets (where actual registrants live) could be starved indefinitely. Restore the cross-bucket random selection across visited buckets, with the warm-up frontier (numRequests gate) intact. Apply the issue #65 fix narrowly: move the break inside the `if len(new) > 0` block, so the frontier only halts on a populated unqueried bucket. Empty unqueried buckets are now invisible to the gate and the scan continues past them. Net change vs pre-PR upstream is one line moved (the break) and one test added (TestSearchQueryTargetWidensFrontierAfterResponse). Closes #65.
The test was skipped under #30 (in-flight RPC cancellation latency), which is fixed by #58. The remaining failure was a data race in the test itself: it seeded node1's topic table by calling topicTable.Add directly from the test goroutine, racing with node1's dispatch loop reading the table via NextExpiryTime. Seed the table through onDispatchCh instead, so the writes run on the goroutine that owns the table. The search now converges given the IsDone/QueryTarget fixes on this branch; passes 10x under -race.
…cket Prevent a single registrar from dominating a registration bucket by returning many nodes for it. Track the buckets a registrar has filled on its own RegAttempt (filledBuckets), enforcing one entry per source per bucket within and across responses. The set is freed when the registrar is evicted, so the accounting is tied to its lifetime rather than accumulating in the buckets. Replaces the per-call bucketCheck with this single mechanism.
…ndby pool is full On ad expiry, demote the registrar back to Standby for renewal so the bucket's candidate pool isn't drained (which caps per-registrant fan-out). When the standby pool is already full, drop the expired attempt instead: retaining it gives no anti-drain benefit and blocks newly discovered nodes from entering the bucket (AddNodes rejects while standby is full), ossifying registration around already-used registrars. Adds TestRegistrationExpiryDropsWhenStandbyFull.
Topic registration and search kept handing work to nodes that had gone away, and the local ad cache held ads pointing at dead advertisers. There was no liveness gate beyond the DHT routing table's own findnode-failure eviction, which the topic system doesn't observe. Observe the outcome of every discv5 call (PING, FINDNODE, TOPICQUERY, REGTOPIC — both DHT-layer and topic-layer) at the dispatch chokepoint: a call that receives a response is a success, one that times out without any response is a failure, and a caller-cancelled call produces no sample. The samples are delivered to the topic system over a buffered channel. The topic system reuses the existing per-peer discv5 failure counter (enode.DB.FindFailsV5, previously unused) to count consecutive failures. After MaxNodeFailures in a row a node is added to a TTL-bounded blacklist (BlacklistTTL) and evicted from all registration tables, search tables and the local ad cache; a success resets the counter. Registration and search also skip blacklisted nodes on add, so a banned node cannot re-enter while the ban holds. Both knobs are configurable and independent of the DHT's own maxFindnodeFailures eviction, which is left untouched. Defaults: MaxNodeFailures=3, BlacklistTTL=15m. Also adds a goroutine-safe nodeCount() accessor on the registration loop and switches TestTopicRegNodeTableUpdates to poll it, fixing a latent test data race (the test read loop-owned registration state directly).
A node selected for the next REGTOPIC (a Waiting attempt still in the heap) could be evicted via RemoveNode before the send fired. RemoveNode's in-flight guard only protects already-sent attempts (index == -2), so the selected-but-not-yet-sent attempt was removed and the subsequent StartRequest panicked with 'bad attempt index -1'. Cancel the pending send when the evicted node is the one selected.
Completes the topic-discovery liveness/eviction design (#21 folded in): - Rename enode.DB.FindFailsV5 -> TopDiscLivenessFails (and the setter) — it counts consecutive failed topic-discovery RPCs, not findnode failures. - Add a routing-table eviction feed (Table.subscribeRemovedNodes, fired in nodeRemoved). The topic system subscribes and evicts the node from the reg tables, search tables and ad cache when the DHT drops it (failed reval), without blacklisting it (it's gone, not proven malicious). - Refactor banAndEvict into evictEverywhere (used by both signals) + the ban. So ad-cache / reg / search eviction is now driven by BOTH the failure-counter blacklist AND DHT routing-table eviction. The latter is the main lever for ad-cache cleanup, since we never send topic RPCs to our own advertisers. Adds TestTopicDHTEvictionEvictsAd. Closes #21.
… panicking
When a call's caller cancels (quit fires) while the call is still parked in callQueue behind another active call to the same node, callDone reports a call that is not the active one. dispatch used to panic("BUG: callDone for inactive call"). Remove it from the queue and continue instead; keep panicking only when the call is neither active nor queued, which is a genuine bookkeeping bug. Adds TestUDPv5_callDoneQueued.
srene
force-pushed
the
feat/simnet-testbed-v2
branch
from
June 15, 2026 13:29
aec78d3 to
712ffc5
Compare
Adds a count accessor on the topic failure blacklist (#71) and a UDPv5.BlacklistLen() wrapper, so the churn testbed can sample how many nodes are blacklisted over time as a failure-eviction metric.
Adds the churn workload (-churn-interval/-churn-frac/-churn-mode: steady-state 50/50 leave-join or killonly) with membership tracking and dead-result accounting, plus the mixed-binary interop workload (-vanilla-frac: a fraction of nodes run stock upstream geth v1.17.3 as a real separate discv5 stack alongside TopDisc). Build notes: the interop stack pulls in github.com/ethereum/go-ethereum-vanilla (a module-path-renamed fork of upstream v1.17.3). It is currently sourced via a local-path go.mod replace (/home/sergi/geth-vanilla on the eval host); to be replaced with a published branch later. Because two secp256k1 cgo copies collide at link time, the testbed must be built with CGO_ENABLED=0.
Sets discover.Config.Topic.AdLifetime on every TopDisc node (initial population, churn joiners, and interop forks all go through spawnNode). 0 keeps the discv5 default of 15m.
Churn runs tear down a partially-stuck network and the search wg.Wait can overrun the nominal deadline; the 8m grace was too tight (churn-repl22 hit the watchdog before writing metrics on the 10k sweep).
Match PR #72: a call reported done must be in exactly one of {active, queued}; panic on both/neither rather than silently continuing. Preserves emitCallOutcome on the active path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an in-process discv5 testbed under
testbed/simnet/for protocol-correctness experiments on the DISC-NG topic registration/search pipeline. Usesgithub.com/marcopolo/simnetfor the UDP layer (SimConn / variable-latency packet router) with bounded per-pair latency and bandwidth.What's included
main.go— workload driver:-topics=1): every node registers and searches a fixed test topic.-topics=N -zipf-s=…): each node draws a topic from a deterministic Zipf distribution and both registers and searches that topic.searchResult.uniqueFoundAtMs[]— wall-clock timestamps of each distinct registrant the iterator yielded, per searcher (drives the unique-found-over-time figure).registrationTimingNs[topicHex][regId]— per-registrant time-to-first-remote-admission, recorded by arunRegistrationProbegoroutine pollingLocalTopicNodesevery-reg-probe-period(default 500 ms).registrationCoverage.byTopic[t]— per-registrant fan-out and per-host load at the end of the register-wait.figures.py— renders a 7-figure Markdown report (parameters + topic distribution; aggregate; post-register-wait coverage; per-searcher unique recall; time-to-first CDF; unique-found-over-time; ID-space registrants; ID-space discovery coverage; fan-out two views; registration latency). Figures 3 and 7 cleanly skip when their data is absent so the script stays backward-compatible with older runs.adapter.go— thin shim wrappingsimnet.SimConnto satisfynet.PacketConnfordiscover.ListenV5..gitignorefor local artifacts (compiled binary, metrics JSON/log, figure outputs).Network setup details
33.N1.N2.1, one node per /24) so go-ethereum'snetutil.IsLANdoes not bypass the per-/24 bucket cap and the IP-similarity term in §6 eq. 1. The simulation exercises the full DISC-NG IP-defence pathway.maxBootnodes=20(random from the already-spawned set) to avoid the O(N²) bootstrap PING storm that stalls indefinitely above ~2000 nodes.How to run
Dependency
The testbed uses a small fork of
marcopolo/simnet(adds router sharding, per-link buffer/AQM controls, andSimConn.Stats()for thousands-of-node runs), pinned via ago.modreplace togithub.com/srene/simnet@b1b838a.go buildfetches it automatically — no manual setup.Running at scale
The example above targets a quick ~500-node run. For thousands of nodes, use the scaling knobs so the in-process router/links don't become the bottleneck:
Scaling flags (all have safe defaults; values above are illustrative):
-router-shards/-router-buf— VariableLatencyRouter shard count and per-shard buffer (router throughput).-link-buf— per-direction Simlink buffer depth.-link-no-aqm— disable fq_codel + rate limiting (~5–10× faster packet drain; AQM modeling is noise at high node counts).-spawn-delay/-max-bootnodes— spread the bootstrap PING burst that otherwise stalls above ~2000 nodes.Run
/tmp/simnet -helpfor the full flag set (staggering, paced consumption, checkpointing, routing-table refresh, etc.).Closes #57 (Task 3.1.2: simnet-based in-process testbed). Part of #10 (Task 3.1).