Merge upstream - #3
Open
OmicronTau wants to merge 2525 commits into
Open
OmicronTau wants to merge 2525 commits into
OmicronTau wants to merge 2525 commits into
Conversation
The goals of this refactor are (a) to break up the monstrosity `ValidatorManagerImpl::update_shards` was and (b) to stop tracking destroyed validator sessions. (a) is achieved by moving almost all logic from `update_shards` into a new NetworkState struct living inside `validator-group.cpp`. On a conceptual level, manager stops caring about consensus groups at all and just gives NetworkState::update the new state which it needs to update groups to match. For (b), the only problem was always restart of an already torn-down groups (mostly because of self-inflicted reasons such as non-atomic clean-up of consensus groups and restart from a rotated-all-shards block). To fix both, manager starter now iterates the DB forward to find last mc block with inited proof. This block was either never seen by update_shards or was the last block that was seen by it. Either way, if we start groups that it mandates us to, it is easy to see that we will never restart a torn-down group. Unfortunately, we still make manager rewind from rotated-all-shards block since otherwise there is no easy way to figure out what were genesis blocks for the groups. Notable functional changes are: - We no longer wait for initial sync before starting the groups. This makes start from the zerostate near instant. - We don't try to compute apply_blocks for IValidatorGroup::update_options. It is unused on Simplex side regardless. - Patch changes session ID computation for groups that were "unsafely rotated" for the sake of simplicity of the implementation. Since we do not have (or anticipate) any -F flags in the public networks during the update, this is harmless. - We no longer call `cleanup_external_messages`. The pool will clean up on its own. - We no longer thread (unused) CollationManager. It will be removed in the future regardless. - With this commit, the proper way to rotate keys is to keep the old validator/ADNL keys until the next validator epoch ends. Incidentally, same changes that allow key rotation make us half-support multi-identity validators. Such use is discouraged by LOG(ERRORR). Note that downstream (simplex) will still crash if someone decides to run validator-engine with multiple active validator keys.
They are unused.
This prepares us for DB support on observers.
Otherwise we would either get an explosion of flags or start lying about what each flag enables.
Note that we do not yet have private overlay interface implementation that would support pool on an observer.
The commit builds upon observer support in pool and adds support for observers in private overlay. The end result is that both block and votes are synced in real-time on all validators which will be necessary for a future plumtree broadcast in fast-sync and public overlays. The functionality is gated by simplex_config_v2::protocol_version >= 2.
Bus runtime natively supports asynchronous DI and it has support for
synchronous DI via Actor(Bus&) constructor which we use for
simplex::Bus::bootstrap_{votes,certificates} inside simplex::DbImpl.
However, in some cases, it is not so convenient to create a separate
actor for each injected synchronous dependency. A particular example of
this is consensus::Bus::db which we *can* inject inside simplex::DbImpl
but that would require making DbImpl template over underlying DB type.
Thus the commit introduces a concept of "provider" which is basically a
lightweight version of an empty bus listening actor that has a
constructor performing a transform on the bus.
The if-spaghetti in bridge.cpp was otherwise getting hard to read. The change will also help with future testing of bridge.cpp + validator-engine.cpp.
Traditionally, LLM reported tricky corner cases that are impractical, but better be fixed.
Having methods like > fun (T | Box<T>).method(self) and a receiver `Box<int> | int` correctly infer T=int. For unions, we try greedy permutations.
Toncoin is being renamed to GRAM, that's why:
- `ton("0.05")` becomes `grams("0.05")`
- `ton("0.05")` still supported and will be
`@deprecated` in Tolk v1.5
- stdlib and comments updated to reflect GRAM naming
Observers in private overlay
Tolk v1.4.2: rename TON to GRAM
Merge Tolk-1.4.2 to testnet branch
Otherwise it will register PrecheckCandidateBroadcast which would conflict with BlockSyncObserver's handler.
Removing options crashes validators that used them.
The old behavior was just broken, it made shared PTY fd non-blocking for all programs sharing the fd which in turn led to random crashes (e. g. Python exits with exitcode=120 when it cannot flush to stderr). As an aside, the whole terminal/ seems utterly broken but that's none of my business.
* Throw away old collator code
It was never supported with simplex and we will be building a proper
validator/collator split in next commits.
* Remove almost unused validator-session-types.h
* Move validator-session/candidate-serializer to validator/consensus/
The commit foreshadows the future name of validator-session's candidates
("payloads") that will be introduced in the next commit.
* Move around methods in validator/consensus/payload.cpp
This will make next diff easier to read.
* Simplify consensus/payload API
The commit finally embraces the new naming for payloads.
* Remove unused IValidatorGroup::create_session
* Stop supporting reading simplex_config_v1
In case we see it, we fall back to the default parameters.
* Make MasterchainState::get_new_consensus_config always return config
* Initial plumtree broadcast commit. All interfaces and TL structures are properly designed and implemented. Also it includes implementation of multi-point broadcast mode and signatureless broadcasts. * Move parts schedule to a separate helper class. Initialize encoder only after ensuring that node needs to send parts. Other small refactoring * Remove memory overhead by storing each part payload only once and reconstructing TL object when needed. * Refactor timeouts and TL parsing. * Implemented sending parts schedule among current shard validator set participants. Exported local validator index from consensus for deterministic schedule. * Pass options to plumtree constructor. * Don't send protoviolation on requests to GC'ed parts. * Tighten eager and full_sent_payloads limit for corner cases. * Added separate larger overlay neighbours list for plumtree. * Increase default quic MTU * Add local parts normally to decoder. Fix eager edges limiting. * Added test simulation that runs real overlay, plumtree, TL objects, signatures over the simulated network class. * Implemented single-source plumtree broadcast mode for masterchain blocks. * Added single-source mode to simulator for masterchain blocks * Fix plumtree call sites to match recent testnet changes * Added validator-engine options to switch between fec/plumtree/dual public broadcast mode * Add Overlay_traffic log. * Redesign plumtree public broadcast to all validators sources mode. Removed shard validator group mode, single & multisource modes, schedule. * Wire plumtree broadcast sending to CandidateReceived event from any validator * Implemented simple broadcast mode for plumtree. Added new mode to broadcast different sets of signatures from multiple sources w/o duplicate traffic. Added separate BlockFinality broadcast. Integrated plumtree into fast-sync overlay. * Adapt simulator to new all-validators-sources plumtree approach. * Use normal certificates for plumtree senders in public overlay. * Set separate eager limit for validators. * Send repair request immediately when no eager neighbours. * Change default plumtree parameters * Use default plumtree options in simulator * Fix compile errors * Support new block finality broadcasts in BlockReceiveStats. * Don't receive broadcasts on validators in fast-sync. Still allow plumtree control messages. * Increase random peers selection attempts to better work with many permanent peers. * Replace plumtree repair messages with queries. Setup larger MTU for eager peers. * Redesigned missing parts repair logic. Replaced TTL-based cache with size-based. Optimized repairs alarms and cache to optimal complexity. * Keep max repair targets limit for cold start immediate repairs too. * Add hard cap on max active rapir queries * Add warning when max repair requests limit reached * Free local decoder and don't try again in case of decoding with incorrect hash. * Turn on plumtree by protocol version. * Forward candidates from observers too. * Send candidates and finality to fast-sync and public via plumtree from consensus. * Fully support candidate+finality approach in custom overlays * Store block receive stats for MC blocks too. * Turn on new plumtree broadcasts by config. For old config all broadcasts stay as before. * Don't grant eager quic MTU to anyone on DoNotReceiveBroadcasts nodes. * Properly pass is_original_sender to plumtree in public overlay. Reject IHAVE and PAYLOAD on validators. * Always send telemetry via twostep in fast-sync. * Wrap finality broadcast id in TL object. * Style fix * Adapt plumtree tests to the new repair queries. * Fix tests for windows * Added strict check that USEFUL/PRUNE correspond to previous payload send. * Remove inactive eager peers
#2453) * Don't send shard block description in new version * Fix CO_TRY macro * Expose methods generate_block_proof_root and ProofLink::get_root_cell * Remove utime from get_validator_set (it's unused) * Implement generate_shard_block_description * Generate shard block description after block broadcast * Check finality signatures and generate shard block desc from finality * Check sig_set cc_seqno and hash in ValidateBroadcast * Support next validator set in check_finality_signatures * Don't generate shard block desc when out of sync * Process final sig_set if cached is not final
Grafana dashboads and more stats in prometheus
…t in consensus overlays
This also requires adding all collators to private-overlay, not just collators of the group's validators
Add all collators to all private overlays (including masterchain) because we want collators to reliably get blocks
Collator improvements
drop early return in ShowCollatorsListQuery::receive
Fix timeouts and cancelling in BlockProducerImpl::prepare_delegation
Improve adnl packet size calculation
TON 2026.08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
18 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.