testbed/simnet: in-process discv5 testbed with simulated UDP via simnet#52
Closed
srene wants to merge 4 commits into
Closed
testbed/simnet: in-process discv5 testbed with simulated UDP via simnet#52srene wants to merge 4 commits into
srene wants to merge 4 commits into
Conversation
…arcopolo/simnet
Adds a separate Go module under testbed/simnet/ that spawns N discv5
nodes inside a single Go process, wired through github.com/marcopolo/simnet
as the simulated UDP transport. simnet provides per-link bandwidth, MTU,
and per-pair latency via a LatencyFunc; this scaffolding uses a static
per-pair latency for now.
Layout:
testbed/simnet/go.mod separate module to keep simnet + (future)
ethereum-network-latencies deps out of the
root go.mod. Mirrors the cmd/keeper precedent.
testbed/simnet/adapter.go simUDPConn shim adapting *simnet.SimConn
(net.PacketConn / net.Addr) to discv5's
UDPConn interface (netip.AddrPort).
testbed/simnet/main.go CLI: -nodes, -latency, -bandwidth-mibps,
-duration. Generates keys, spawns LocalNode
+ ListenV5 per slot, bootstraps each new
node from previously-spawned ones.
A 4-byte IP form is used for endpoint addresses so the simnet router's
strict address keying matches what discv5 + the adapter produce on
WriteTo (net.UDPAddrFromAddrPort yields 4-byte IPs for IPv4); 16-byte
IPv4-in-IPv6 forms surface as 'unknown destination' packet drops.
Smoke run with 10 nodes bootstraps cleanly through simnet with no
dropped packets and tears down without errors.
This commit is scaffolding only. Follow-ups:
* topic registration / search workload
* structured per-node JSON logging compatible with testbed/analyse.py
* realistic per-pair latencies via the
github.com/marcopolo/ethereum-network-latencies dataset
Without a cap, every newly-spawned node bootstrapped from all previously-spawned nodes (cfg.Bootnodes appended unconditionally). For N nodes that produces ~N^2/2 bootstrap PINGs across the run; at N=2000 the startup stalls indefinitely on the M3-class host I'm using. Cap at 20 bootnodes per node, mirroring the Python testbed's select_bootnodes helper. Always include the first node admitted so every later node has a guaranteed reachable peer; the rest are uniformly random. Effect at N=1000: before: peak memory 1.85 GB, user CPU 13.2 s after: peak memory 324 MB, user CPU 1.34 s The change unblocks scale tests at 2000+ nodes.
Adds a minimal end-to-end workload to the in-process testbed:
- Half the spawned nodes (configurable via -register-frac) call
RegisterTopic on a fixed test topic; the rest call TopicSearch and
consume the iterator until either the configured -search-timeout
fires or all registrants have been found.
- Per-search metrics are aggregated in-process:
NodeIdx, NodeID, Found, FoundRegistrant (in expected set),
FoundExtra (returned but not registrants), TimeToFirstNs,
TimeToCompletionNs, HitTimeout, FoundIDs.
- Summary prints recall (full / mean), timeout count, time-to-first
and time-to-completion percentiles. With -metrics-out, the per-search
detail is written as JSON for offline analysis.
The metrics are gathered via direct in-process introspection of the
TopicSearch iterator (timestamping yields, counting matches against
the registrant set) rather than via go-ethereum's metrics package or
log scraping. In a single-process testbed this is cleaner than scoping
metric handles per-node, and avoids slog handler multiplexing.
Smoke runs:
-nodes 10 -search-timeout 10s
mean recall 92%, full recall 3/5, t-to-completion median 438ms.
-nodes 20 -search-timeout 30s
mean recall 65%, t-to-completion median 30s — most searches hit
the timeout, consistent with Search.IsDone not natively
terminating when the bucket new-set drains (Issue #27).
Mirror the existing registration tracking: topicSystem.search holds every *topicSearch returned by newSearchIterator, removed when the iterator's Close() fires. topicSystem.stop() now stops searches alongside registrations, so UDPv5.Close() deterministically tears down both instead of leaking search goroutines past transport teardown. topicSearch.stop() is made idempotent (sync.Once around close(quit)) so iter.Close() and topicSystem.stop() can safely race. Closes #28.
srene
force-pushed
the
feat/simnet-testbed
branch
from
May 11, 2026 14:05
c5042f1 to
2f2bd36
Compare
Member
Author
|
Closing until Current ceiling is empirically ~500–700 nodes — see #64 for the bottleneck analysis (single global dispatcher, real-time latency, small channel buffers) and the proposed improvements. The fork is at https://github.com/srene/simnet. The functional simnet code on |
Member
Author
Known limitations of
|
| nodes | outcome | notes |
|---|---|---|
| 50 | works cleanly | DISC-NG mixed-deployment validation (issue #6) passes |
| 500 | works cleanly | wall ≈ 1.5–2 × simulated time; full coverage + recall matching Mac baseline |
| 1 000 | stalls | 11 min wall to advance ~3 min simulated; 0 % CPU, scheduler-blocked |
| 10 000 | stalls | spawn OK, ~3.5 GB RSS, but never reaches the coverage snapshot phase |
SIGQUIT goroutine dump at 1k: ~30 000 goroutines parked in simnet.linkDriver.processQueue → VariableLatencyRouter.RecvPacket for ≥12 min. Not CPU-bound, not memory-bound — serialized through the simulator's event loop.
Root causes (also tracked in #64)
- Single global dispatcher.
VariableLatencyRouter.Start(router.go:166-206) runs one goroutine reading from a singlechan *Packetof buffer 128. Every packet from every node serializes through it. - Real wall-clock latency.
deliveryTime := time.Now().Add(latency)+time.NewTimer— simulation cannot run faster than wall time, and beyond a threshold goes slower than wall time because per-packet overhead exceeds packet inter-arrival. - Small channel buffers. Router 128, linkDriver 32. At 1k nodes these fill within microseconds during a burst and start backpressuring senders all the way into the discv5 stack.
fq_codelalways on. Non-trivial per-packet cost from a fair-queue AQM that we don't need for static-topology simulation.- Mutex-protected
addrMap. Every packet dispatch doesLock/Unlockon the global node map.
Status of this PR
The testbed code in this PR is functional and useful at the 50–500-node range — it's how we validated the DISC-NG ENR flag (issue #6), exercised the IP-bucket caps, and stressed the registration/search pipelines under PR #58 / #60 / #62. Keeping the PR open so the merge target remains live; we'll revisit the upper bound once the simnet improvements proposed in #64 (mirror at https://github.com/srene/simnet) are landed.
Member
Author
|
Closing until the simnet scaling issues tracked in #64 are addressed. The testbed code is functional at 50-500 nodes but cannot reliably reach the 1k+ scale we need for protocol experiments — see the limitations comment above. Will reopen once the proposed improvements (synctest wrapping, sharded router, bigger buffers) land on the fork at https://github.com/srene/simnet. |
This was referenced May 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an in-process discv5 / DISC-NG testbed under
testbed/simnet/that spawns N nodes inside a single Go program, wired throughgithub.com/marcopolo/simnetas the simulated UDP transport.This is a parallel testbed to the Docker-based mode in #51. Both have a place:
LatencyFuncsynctestintegration)The simnet testbed is well-suited for protocol-correctness testing, scale tests beyond what Docker can boot, and reproducible CI runs. The Docker testbed remains useful for final integration smoke tests against real OS sockets.
Status: scaffolding
This PR brings up N discv5 nodes through simnet with a static per-pair latency and exits. The DHT bootstraps cleanly between them; verified with smoke runs of 3 and 10 nodes (zero dropped packets).
Workload (registration / search) and dataset-driven realistic latencies are deliberately follow-up commits.
Layout
Smoke run
No errors, no dropped packets.
One nuance worth flagging
The simnet router keys addresses strictly.
net.IPv4(...)produces 16-byte IPv4-in-IPv6 forms;net.UDPAddrFromAddrPortproduces 4-byte forms. If the endpoint is registered with one form and packets arrive with the other, simnet drops them as "unknown destination". This PR uses 4-byte IP form (net.IP{10, 100, ...}) for endpoint registration to match what the adapter'sWriteToproduces. Documented inline.Follow-up commits planned for this branch
testbed/analyse.pyexpects, so the existing analysis pipeline works.github.com/marcopolo/ethereum-network-latencies(predicted RTT matrix between ~7000 real Ethereum nodes, generated by an ML model trained on RIPE Atlas + WonderNetworks data). Plumbed throughSimnet.LatencyFuncfor true pairwise behaviour.Each will be a focused commit on this branch (or a separate stacked PR if review velocity requires). Marking as draft until at least the workload step lands.
Closes #57.