[WIP] prototype Overlay V2 in Rust#5147
Draft
marta-lokhova wants to merge 8 commits intostellar:masterfrom
Draft
[WIP] prototype Overlay V2 in Rust#5147marta-lokhova wants to merge 8 commits intostellar:masterfrom
marta-lokhova wants to merge 8 commits intostellar:masterfrom
Conversation
Add the Rust overlay crate to the workspace and update the build system to compile and link the Rust overlay library alongside stellar-core. - Add workspace Cargo.toml and overlay/Cargo.toml with libp2p, tokio, and IPC dependencies - Update configure.ac to detect Rust toolchain and set RUSTFLAGS - Update Makefile.am and src/Makefile.am to build the Rust overlay library and link it into stellar-core Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement the Rust overlay's core modules: IPC transport (overlay/src/ipc/): - Unix domain socket transport for C++ <-> Rust communication - Binary message framing with length-prefix encoding - Async message handling with tokio - Message types for SCP envelopes, transactions, tx sets, and peer management commands Flood control (overlay/src/flood/): - Mempool for transaction storage and deduplication - INV/GETDATA protocol for pull-based transaction dissemination - Inventory batcher for efficient batched advertisement - Inventory tracker to avoid redundant sends - Pending request tracking with timeout support - Transaction set assembly and caching - Transaction buffer for outbound flow management Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement the peer-to-peer networking layer using libp2p: - config.rs: Network configuration (listen addresses, bootstrap peers, Kademlia DHT settings, connection limits) - libp2p_overlay.rs: Main libp2p swarm implementation with Kademlia DHT for peer discovery, gossipsub for message dissemination, and connection management with configurable limits - integrated.rs: Integration layer that bridges IPC transport with the libp2p overlay, routing messages between C++ core and the P2P network - main.rs: Standalone binary entry point for running the overlay as a separate process - http/mod.rs: HTTP server for overlay status, metrics, and admin endpoints (peer info, connection stats) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add comprehensive test suites for the Rust overlay: - e2e_binary.rs: End-to-end tests that launch overlay processes and verify IPC communication, message routing, and peer connectivity - kademlia_test.rs: Tests for Kademlia DHT peer discovery including multi-node topologies, peer table convergence, and bootstrap scenarios Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the legacy C++ overlay networking layer with a thin IPC bridge that communicates with the new Rust overlay process. New files: - IPC.cpp/h: Low-level IPC transport (Unix domain sockets, message framing, async read/write) - OverlayIPC.cpp/h: High-level IPC protocol handling (message serialization, request/response routing) - RustOverlayManager.cpp/h: OverlayManager implementation that delegates networking to the Rust overlay via IPC - NetworkConstants.h: Shared constants between C++ and Rust Removed legacy components: - Peer.cpp/h, TCPPeer.cpp/h: Direct TCP peer connections (now handled by Rust libp2p) - FlowControl.cpp/h, FlowControlCapacity.cpp/h: Per-peer flow control (now handled by Rust flood control) - Floodgate.cpp/h: Transaction flooding (replaced by Rust INV/GETDATA) - PeerAuth.cpp/h, Hmac.cpp/h: Authentication (now handled by libp2p noise protocol) - PeerManager.cpp/h, PeerDoor.cpp/h: Peer lifecycle management - ItemFetcher.cpp/h, Tracker.cpp/h: SCP message fetching - TxAdverts.cpp/h, TxDemandsManager.cpp/h: Pull-mode tx dissemination - Survey*.cpp/h: Network survey (to be reimplemented) - OverlayManagerImpl.cpp/h: Legacy overlay manager Updated tests: - IPCTests.cpp: IPC transport unit tests - OverlayIPCTests.cpp: IPC protocol integration tests - OverlayIPCBenchmark.cpp: IPC throughput benchmarks Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Update the C++ core to work with the Rust overlay via IPC instead of the legacy in-process overlay. Herder changes: - Remove TransactionQueue and TxQueueLimiter (transaction queuing now handled by Rust mempool) - Adapt HerderImpl to submit transactions via IPC - Update PendingEnvelopes to fetch SCP messages via IPC - Simplify TxSetFrame to work with externally-assembled tx sets Application changes: - Add RustOverlayManager instantiation to ApplicationImpl - Update Config with Rust overlay settings (IPC socket path, overlay binary path) - Update AppConnector to expose IPC interface - Adapt CommandHandler for new overlay endpoints Simulation changes: - Rewrite Simulation and Topologies to launch Rust overlay processes - Update LoadGenerator and TxGenerator for IPC-based tx submission - Adapt CoreTests for new topology model Other adaptations: - Database: Remove overlay-specific tables (peers, peer preferences) - LedgerManager: Remove direct overlay interactions - Fuzz tests: Adapt for new overlay interface Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Update Dockerfiles to install Rust toolchain and build the overlay crate as part of the stellar-core build pipeline. - Dockerfile: Add rustup installation, cargo build step - Dockerfile.testing: Include Rust build for test images Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add comprehensive design document for the Rust overlay, cross-checked against the actual implementation: - Architecture overview with ASCII diagrams - QUIC transport and stream protocol specification - IPC protocol with all message types and wire formats - INV/GETDATA pull-based TX flooding protocol (new) - Mempool, TX set building, and shared state documentation - Complete test coverage summary (178 Rust + C++ integration tests) - Known issues and unimplemented features Excludes session notes, debugging logs, and agent instructions from dev/ai/ — this document is the single source of truth for the overlay design. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is still a half-baked prototype of Rust overlay, but pushing as a draft for visibility and initial feedback.
The last commit adds a couple of files to
docsthat give an overview of what this prototype is doing, but tl;dr is that is implements overlay as a separate process in Rust.The good news is that we can use many things out of the box via libp2p: Kademlia for peer discovery, peer auth, GossipSub for topology forming, QUIC for multiplexing. We also benefit from using Rust's tokio async scheduler. Of course, these things need tuning and probably a bunch of debugging to hash out all the edge cases, but it's quite encouraging to see something that kinda works in just a few days of "vibe coding" (as confirmed by a good amount of tests on the Rust side and in OverlayIPCTests). It would be interesting to see if we can actually get a reliable initial implmenetation of Rust overlay, such that we can optimize individual areas as needed (for example, we know GossipSub isn't what we want, so we can implement a custom solution in Rust, rather than poking at C++ which has a bunch of weird legacy stuff like PeerManager etc).
Another interesting observation in this experiment was finding the clean cut point between overlay <> SCP. By moving most of peer and flooding components to Rust, this actually leaves a single digit number of interaction points with C++, which is quite a bit easier to reason about and encourages good separation of responsibilities.
Next steps would be to actually get it to run in supercluster (right now there are some annoying issues I'm debugging like DNS resolution and containers being able to properly talk to each other), and see what kind of SCP latency we can get from removal of HOL blocking via multiplexing plus the natural parallelism we get from separation of processes.