Implement the RPCBlockHeaderSubscriber for indexing finalized results#728
Implement the RPCBlockHeaderSubscriber for indexing finalized results#728m-Peter wants to merge 131 commits into
RPCBlockHeaderSubscriber for indexing finalized results#728Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds experimental soft-finality ingestion (flags/config) and event-hash storage; introduces batching-only tx pool behavior (always enqueue), updates tests for batching semantics, and small storage/db batch helpers and keys. ChangesSoft Finality
Batching & Tests
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 3❌ Failed checks (3 warnings)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
m-Peter
force-pushed
the
mpeter/poc-index-finalized-block-results
branch
from
10a8bdd to
4fd4a5d
Compare
peterargue
left a comment
peterargue
left a comment
There was a problem hiding this comment.
added a couple small comments, but otherwise this looks good. let's get it running against one of the live networks.
|
@peterargue Thanks for the review 🙌 I've addressed all the comments. |
m-Peter
requested review from
janezpodhostnik and
zhangchiqing
as code owners
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…ersFromStartHeight
…or function resuse
m-Peter
force-pushed
the
mpeter/poc-index-finalized-block-results
branch
from
314e7cc to
bfe6188
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…lized-block-results
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
bootstrap/bootstrap.go (1)
630-647: Force cadence height enhancements are well-implemented.The logic correctly handles resetting event hash state when forcing a start height:
- Conditionally resets processed sealed height only when it's ahead of the forced height (line 640-644)
- Always removes event hash entries above the forced height (line 645-647)
- Proper error handling with descriptive messages and wrapping
- Batch operations ensure atomicity
Consider clarifying the error message on line 642 to better reflect the operation:
- return nil, nil, fmt.Errorf("failed to set latest verified sealed height: %w", err) + return nil, nil, fmt.Errorf("failed to set processed sealed height to forced start height: %w", err)The current message says "latest verified sealed height" but we're actually setting it to the forced start height, which may not be the latest.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
bootstrap/bootstrap.go(5 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-10-06T10:14:49.706Z
Learnt from: m-Peter
Repo: onflow/flow-evm-gateway PR: 890
File: services/ingestion/event_subscriber.go:235-239
Timestamp: 2025-10-06T10:14:49.706Z
Learning: In services/ingestion/event_subscriber.go, when reconnecting after disconnect errors (DeadlineExceeded, Internal, Unavailable), the subscription should reconnect at lastReceivedHeight rather than lastReceivedHeight+1. This avoids errors when the next height doesn't exist yet, and duplicate event processing is safe because the ingestion engine is explicitly designed to be idempotent (storage uses batch.Set() which overwrites existing entries).
Applied to files:
bootstrap/bootstrap.go
🧬 Code graph analysis (1)
bootstrap/bootstrap.go (5)
storage/pebble/events_hash.go (2)
EventsHash(14-16)NewEventsHash(18-22)services/ingestion/event_subscriber.go (2)
EventSubscriber(27-33)NewRPCEventSubscriber(49-65)services/ingestion/sealing_verifier.go (2)
SealingVerifier(30-54)NewSealingVerifier(57-82)services/ingestion/block_tracking_subscriber.go (1)
NewRPCBlockTrackingSubscriber(48-66)models/errors/errors.go (1)
ErrStorageNotInitialized(37-37)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Lint
- GitHub Check: Test
🔇 Additional comments (4)
bootstrap/bootstrap.go (4)
60-60: LGTM! EventsHash storage added to Storages.The new
EventsHashfield is properly typed and consistent with the other storage fields in the struct.
618-618: LGTM! EventsHash initialized correctly.The initialization follows the expected pattern and properly uses the store parameter.
711-711: LGTM! EventsHash properly wired into returned Storages.The field is correctly added to the return struct, completing the integration of EventsHash into the storage initialization flow.
163-193: Based on the verification results, the original review comment makes valid architectural observations but the execution verification was incomplete. The script output confirms the structure but doesn't show how theverifierfield is actually used in methods, which is necessary to fully verify nil-safety.Code design is sound; verify verifier nil-handling in implementation.
The conditional subscriber creation follows good architectural patterns—the two-level feature flag (outer for soft finality, inner for sealing verification) is clean, and both paths create fresh subscriber instances starting from
nextCadenceHeight.However, confirm that
NewRPCBlockTrackingSubscriberproperly handles the optionalnilverifier parameter in all method calls. The verifier field can be nil whenExperimentalSealingVerificationEnabledis false, and Go requires explicit nil checks before dereferencing. Verify that all methods using the verifier guard against nil or document that the parameter must never be nil.Additionally, since both subscribers are instantiated fresh from the current height on each bootstrap, toggling
ExperimentalSoftFinalityEnabledshould not cause state inconsistencies—confirm no persistent subscriber-type mapping exists in storage.
…lized-block-results
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
cmd/run/cmd.go(3 hunks)config/config.go(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- cmd/run/cmd.go
| // ExperimentalSoftFinalityEnabled enables the experimental soft finality feature which syncs | ||
| // EVM block and transaction data from the upstream Access node before the block is sealed. | ||
| // CAUTION: This feature is experimental and may return incorrect data in certain circumstances. | ||
| ExperimentalSoftFinalityEnabled bool | ||
| // ExperimentalSealingVerificationEnabled enables the experimental sealing verification feature | ||
| // which verifies the hash of the EVM events ingested by the requester engine match the hash | ||
| // of the events from the sealed block in the Flow network. | ||
| // CAUTION: This feature is experimental and will cause the node to halt if the events don't match. | ||
| ExperimentalSealingVerificationEnabled bool |
There was a problem hiding this comment.
🧩 Analysis chain
LGTM! The experimental configuration fields are well-documented.
The addition of these experimental feature flags is clear and includes appropriate CAUTION warnings about potential data issues and node halting behavior. The naming follows Go conventions and the documentation is explicit about the risks.
However, given the PR context noting that integration tests cannot currently run (Emulator limitation) and the known limitations with execution forks requiring manual operator intervention, please ensure comprehensive operator-facing documentation exists.
Run the following script to verify operator documentation exists for these experimental features:
🏁 Script executed:
#!/bin/bash
# Description: Verify that operator documentation exists for the new experimental features
# Search for documentation files that might cover these features
echo "=== Searching for documentation files ==="
fd -e md -e txt -e rst . | head -20
# Search for references to the experimental flags in documentation
echo -e "\n=== Searching for soft finality documentation ==="
rg -n -C3 --type=md "soft.finality|ExperimentalSoftFinalityEnabled"
echo -e "\n=== Searching for sealing verification documentation ==="
rg -n -C3 --type=md "sealing.verification|ExperimentalSealingVerificationEnabled"
# Search for any operator guides or deployment docs
echo -e "\n=== Searching for operator/deployment guides ==="
rg -n -l --type=md "operator|deployment|configuration|experimental" | head -10Length of output: 355
🏁 Script executed:
#!/bin/bash
# Search for any references to the experimental features in the entire codebase
echo "=== Searching for ExperimentalSoftFinalityEnabled references ==="
rg -n "ExperimentalSoftFinalityEnabled" . | head -20
echo -e "\n=== Searching for ExperimentalSealingVerificationEnabled references ==="
rg -n "ExperimentalSealingVerificationEnabled" . | head -20
echo -e "\n=== Checking README.md for experimental feature documentation ==="
cat README.md | head -100
echo -e "\n=== Checking if there's any EXPERIMENTAL documentation pattern ==="
rg -n -C2 "EXPERIMENTAL|experimental.*feature" --type=md | head -40Length of output: 6591
🏁 Script executed:
#!/bin/bash
# Check how the experimental flags are used in bootstrap.go
echo "=== bootstrap.go lines 165-175 (ExperimentalSoftFinalityEnabled usage) ==="
sed -n '160,180p' bootstrap/bootstrap.go
echo -e "\n=== cmd/run/cmd.go lines 225-240 (Flag definition and assignment) ==="
sed -n '225,240p' cmd/run/cmd.go
echo -e "\n=== Check if there's any configuration documentation ==="
find . -name "*config*" -type f | grep -E "\.(md|txt|rst)$"
echo -e "\n=== Search for any mention of flags/configuration in docs ==="
rg -n "flag|--experimental" README.md | head -20Length of output: 2161
🏁 Script executed:
#!/bin/bash
# Find where the experimental flags are defined as command-line flags
echo "=== Searching for experimentalSoftFinalityEnabled flag definition ==="
rg -n "experimentalSoftFinalityEnabled" . -B2 -A2 | grep -E "(flag|Flag|Define|cmd|command)" -A2 -B2
echo -e "\n=== Searching for experimentalSealingVerificationEnabled flag definition ==="
rg -n "experimentalSealingVerificationEnabled" . -B2 -A2 | grep -E "(flag|Flag|Define|cmd|command)" -A2 -B2
echo -e "\n=== Check cmd/run/cmd.go for flag.Bool definitions ==="
rg -n "flag.Bool|flag.String" cmd/run/cmd.go | tail -20
echo -e "\n=== Extract README.md configuration table (lines 231-280) ==="
sed -n '231,280p' README.mdLength of output: 9751
Add experimental feature flags to README.md configuration table.
The code changes are sound and the experimental flags have appropriate runtime warnings in cmd/run/cmd.go (lines 299-300). However, these flags are missing from the README.md configuration documentation table (lines 231-280), where operators are directed to find all available configuration options.
Add entries for experimental-soft-finality-enabled and experimental-sealing-verification-enabled to the configuration table so operators have complete visibility of all available flags and their purposes.
🤖 Prompt for AI Agents
In config/config.go around lines 123 to 131, two experimental boolean flags
(ExperimentalSoftFinalityEnabled and ExperimentalSealingVerificationEnabled)
were added but the README.md configuration table (around lines 231-280) is
missing entries for them; update the README.md config table to include rows for
--experimental-soft-finality-enabled and
--experimental-sealing-verification-enabled with concise descriptions matching
the comments/warnings in cmd/run/cmd.go (mentioning that they are experimental
and include the CAUTION text about possible incorrect data for soft finality and
node halt for sealing verification), and ensure flag names, default values, and
descriptions align with existing table formatting and style.
…lized-block-results
…lized-block-results
…lized-block-results
…lized-block-results
BatchTxPool previously submitted the first transaction from any EOA burst immediately, bypassing the pool. If that first transaction carried a future nonce (common when many goroutines dispatch in parallel), it failed on-chain and left a nonce gap that caused EVM.batchRun to drop all subsequent nonces. Confirmed on Flow EVM testnet (2026-06-03, chain 545): - 10 txs sent in parallel, shuffled order [8 7 3 9 2 1 6 0 4 5] - eth_sendRawTransaction returned success for all 10 - Only 6/10 landed on-chain; 4 were silently dropped - Nonce 6 arrived first → submitted immediately → failed (on-chain nonce=0) - Batch [0 1 2 3 4 5 7 8 9] executed 0–5 then stopped at the gap → drop Fix: Add() now always enqueues every transaction into the per-EOA pool. The flush timer (TxBatchInterval) is the sole submission path. This ensures the full burst accumulates before any of it is sorted and submitted, giving EVM.batchRun a complete, gap-free nonce sequence. Trade-off: all transactions incur up to TxBatchInterval of additional latency (default 2.5 s on mainnet). EOAActivityCacheTTL is now a no-op and the flag is marked deprecated for backward-compatible config parsing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Update soft-finality branch with latest changes from main
fix(batch): always pool transactions to eliminate first-tx nonce race (soft-finality)
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@services/requester/batch_tx_pool.go`:
- Line 115: The goroutine started with go
batchPool.processPooledTransactions(ctx) lacks panic recovery; wrap the
goroutine invocation or add a top-level defer/recover inside
processPooledTransactions to catch panics, log the error and stack trace, and
continue (so a panic won't crash the whole process). Specifically, add a defer
func() { if r := recover(); r != nil { /* log r and runtime/debug.Stack() via
the package's logger (e.g., batchPool.logger or existing logger) */ } }() at the
start of processPooledTransactions (or wrap the go call in an inline goroutine
that defers recover) so panics are captured and surfaced without terminating the
gateway.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 78abda1c-325a-41d2-a7df-83cb0a0fdf64
📒 Files selected for processing (5)
cmd/run/cmd.goconfig/config.gogo.modservices/requester/batch_tx_pool.gotests/tx_batching_test.go
🚧 Files skipped from review as they are similar to previous changes (2)
- config/config.go
- cmd/run/cmd.go
| eoaActivity: eoaActivity, | ||
| } | ||
|
|
||
| go batchPool.processPooledTransactions(ctx) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Search for panic recovery in batch_tx_pool.go and verify goroutine resilience
rg -n 'defer.*recover|recover\(\)' services/requester/batch_tx_pool.go
# Also check if processPooledTransactions has internal recovery
rg -A 20 'func.*processPooledTransactions' services/requester/batch_tx_pool.go | rg 'defer|recover'Repository: onflow/flow-evm-gateway
Length of output: 88
Add recover around the batch processor goroutine to prevent gateway crashes on panic.
In services/requester/batch_tx_pool.go, go batchPool.processPooledTransactions(ctx) is launched without any panic recovery, and processPooledTransactions only defers ticker.Stop()—no recover is present. If anything inside panics, the whole gateway process will terminate (not just the goroutine), breaking batch processing until restart. Wrap the goroutine (or add defer recover() inside processPooledTransactions) to log/alert and keep the service running.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@services/requester/batch_tx_pool.go` at line 115, The goroutine started with
go batchPool.processPooledTransactions(ctx) lacks panic recovery; wrap the
goroutine invocation or add a top-level defer/recover inside
processPooledTransactions to catch panics, log the error and stack trace, and
continue (so a panic won't crash the whole process). Specifically, add a defer
func() { if r := recover(); r != nil { /* log r and runtime/debug.Stack() via
the package's logger (e.g., batchPool.logger or existing logger) */ } }() at the
start of processPooledTransactions (or wrap the go call in an inline goroutine
that defers recover) so panics are captured and surfaced without terminating the
gateway.
Co-authored-by: Leo Zhang <zhangchiqing@gmail.com>
Co-authored-by: Leo Zhang <zhangchiqing@gmail.com>
…dations Batch tx submission mode with minimal in-memory tx pool
…lized-block-results
5 participants
Closes: #727
Description
For contributor use:
masterbranchFiles changedin the Github PR explorerSummary by CodeRabbit