Add nonce-aware transaction pool

[vishalchangrani]

Summary

Implements the Nonce-Aware Transaction Pool — the third TxPool strategy designed in the #965 discussion (revised plan) and follow-up. It replaces the blanket batch-window delay with nonce-aware behavior so the fix can serve all production traffic, not just the dedicated DFNS gateway. See also the animated scenario walkthroughs.

Behavior

• Fast path (zero added latency): a tx carrying the expected next nonce (per the local state index), with an empty queue and nothing in flight, is submitted immediately.
• Burst collection: out-of-order arrivals queue per-EOA under a sliding collection window (--tx-collection-window, default 300ms, reset on each arrival). On flush, the longest consecutive nonce prefix is submitted as one EVM.batchRun, capped at --tx-max-batch-size (default 5).
• Submission spacing: consecutive Cadence submissions for the same EOA are spaced ≥ --tx-submission-spacing apart (default 1200ms ≈ 1.5x block production rate, per @m-Peter's suggestion) so they land in different blocks and cannot be reordered by Collection Nodes. The spacing also serves as the flush deadline anchored at first enqueue — there is deliberately no separate hard-cap knob (see fix(batch): eliminate silent tx drops from first-tx nonce race in BatchTxPool #965 discussion).
• Gap handling: txs behind a nonce gap are held until the gap fills, the local index advances past them (stale → pruned with metric + log), or --tx-pool-ttl (default 30s) expires — on expiry they are submitted anyway so the failure is observable on-chain rather than a silent drop (@janezpodhostnik's concern).
• In-flight dedup: a tx whose nonce is already submitted-and-unconfirmed is rejected with a proper RPC error (transaction with the same nonce already submitted), since letting it through burns Flow fees on a guaranteed nonce-mismatch failure (@m-Peter's cost concern). A same-nonce resubmission of a queued (not yet sent) tx replaces it, matching mempool semantics.

Configuration

--tx-nonce-aware-mode          enable the pool (default false)
--tx-collection-window         per-EOA sliding window (default 300ms)
--tx-submission-spacing        min gap between submissions per EOA; also the flush deadline (default 1200ms)
--tx-pool-ttl                  how long to hold gap-stranded txs (default 30s)
--tx-max-batch-size            max txs per EVM.batchRun (default 5)

Validated at startup: window ≤ spacing; requires --tx-state-validation=local-index; mutually exclusive with --tx-batch-mode. SingleTxPool and BatchTxPool are untouched.

Known tradeoff

The fast path submits while holding the pool-wide mutex, so one EOA's fast-path submission briefly serializes other EOAs' Add calls behind one Flow round-trip. This is documented in the type comment; per-EOA locking is the known upgrade path if contention shows up under multi-EOA load.

Testing

• Unit (services/requester): 13 tests covering prefix/expiry selection, fast-path gating, in-flight rejection, and a regression test for a failed-flush reconcile path (a failed submission previously could have permanently wedged an EOA's queue; submitWork/rollbackInFlight fix that and the test exercises the real path).
• E2E (tests/, emulator): out-of-order 10-tx burst lands exactly 10 executions (the DFNS regression scenario), fast-path single tx submits via the EVM.run path, gap hold-and-fill, TTL eviction (duplicate-check lifecycle proves it), and in-flight same-nonce rejection. Existing batch-mode e2e tests still pass.

Deployment notes

• Rollout target: the Quicknode EVM gateways (sticky sessions confirmed, which this design assumes) so DFNS/Hifi can move off the dedicated gateway.
• All timing knobs are configurable; 1200ms spacing should be validated on testnet under load before mainnet rollout.

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

• New Features

  • Added an optional nonce-aware transaction mempool mode with per-account queueing and timed batching.
  • Introduced CLI flags for mempool mode, collection window, submission spacing, pool TTL, max batch size, and nonce-gap behavior.
  • Added Prometheus metrics for txpool queue depth and queued transaction count.

• Bug Fixes

  • Improved rejection of duplicate nonces while a nonce is in-flight and added stricter mempool parameter validation.

• Tests

  • Expanded unit and integration test coverage for queueing, gaps, TTL expiry, batching caps, duplicates, and in-flight reconciliation.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a nonce-aware transaction mempool mode, local nonce reading from indexed state, queueing and batching logic with TTL and spacing controls, startup wiring, metrics, and tests for the new flow.

Changes

Nonce-Aware Transaction Mempool

Layer / File(s)	Summary
Config fields, CLI flags/validation, error type, and metrics instrumentation config/config.go, cmd/run/cmd.go, models/errors/errors.go, metrics/collector.go, metrics/nop.go	Adds the TxMemPool mode config fields and CLI flags, validates mempool constraints when enabled, adds ErrInFlightNonce, and wires new txpool size gauges into the collector interfaces and implementations.
NonceProvider interface and LocalNonceProvider services/requester/nonce_provider.go	Defines the nonce-provider contracts and implements local nonce reads from indexed EVM state with cached block-view lookup.
TxMemPool helpers and nonce tracking services/requester/tx_mempool.go	Adds nonce normalization, consecutive-prefix and TTL-expiry selection helpers, queue state, and the nonceTracker state machine used by the mempool.
TxMemPool construction and Add path services/requester/tx_mempool.go	Builds the TxMemPool, starts background processing, and routes incoming transactions through fast-path submission or per-EOA enqueueing with duplicate and nonce checks.
Bootstrap TxMemPoolMode pool selection bootstrap/bootstrap.go	Adds the startup branch that creates a LocalNonceProvider and initializes the new mempool mode.
Unit tests for mempool internals services/requester/tx_mempool_test.go	Covers the selection helpers, nonce tracking, Add-path behavior, flushing, pruning, metrics, logging, and clock-driven timing cases.
Emulator integration tests for TxMemPool behavior tests/tx_mempool_test.go	Exercises the mempool mode end to end with gateway-emulator transactions, including ordering, batching, TTL expiry, and duplicate rejection.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant TxMemPool
  participant LocalNonceProvider
  participant eoaQueue
  participant CrossSporkClient

  Client->>TxMemPool: Add(tx)
  TxMemPool->>LocalNonceProvider: GetNextNonce(sender)
  LocalNonceProvider-->>TxMemPool: nonce
  alt immediate submission
    TxMemPool->>CrossSporkClient: submitBatch(tx)
  else enqueue
    TxMemPool->>eoaQueue: store tx and timers
  end
  loop background tick
    TxMemPool->>eoaQueue: collectDueBatches()
    eoaQueue-->>TxMemPool: flushWork
    TxMemPool->>CrossSporkClient: send batch
  end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related issues

• Improve nonce-rejection error reporting (include tx nonce + next/pending nonce) #975: The added nonce-aware mempool logic and ErrInFlightNonce directly overlap with the issue’s nonce rejection and error-handling work.

Possibly related PRs

• onflow/flow-evm-gateway#374: Both touch bootstrap transaction-pool initialization wiring in bootstrap/bootstrap.go.
• onflow/flow-evm-gateway#831: Both extend transaction-pool mode selection and related CLI/config plumbing.
• onflow/flow-evm-gateway#974: Both modify the nonce provider and tx mempool implementation for nonce-aware behavior.

Suggested labels

Feature, Improvement

Suggested reviewers

• janezpodhostnik
• peterargue
• zhangchiqing
• m-Peter

Poem

🐇 I hop through nonces, neat and small,
Queues line up in EOA hall.
One tx flies, or waits its turn,
Till gaps are filled and batches churn.
Cadence hums, the pool stays bright—
A rabbit’s mempool feels just right.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 57.53% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding a nonce-aware transaction pool.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch vishal/nonce-aware-mini-pool

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@services/requester/nonce_aware_tx_pool.go`:
- Around line 183-197: The pending-tx event is being published too early by
t.txPublisher.Publish(tx); update Add so it only publishes after the transaction
is successfully accepted into the pool/queue (i.e., after the fast-path
acceptance or after enqueue returns success), moving the
t.txPublisher.Publish(tx) call from its current position to immediately after
the success branch that uses models.DeriveTxSender/tx.MarshalBinary and the
queue acceptance logic; apply the same fix for the duplicate publish site around
the later block (the one referenced at 243-251) so subscribers see only
transactions that were actually enqueued/submitted.
- Around line 230-235: The code sets q.lastSubmittedAt before calling
t.submitBatch (e.g., the q.lastSubmittedAt assignment around submitBatch and
related blocks), which causes spacingElapsed/TxSubmissionSpacing to rate-limit
EOAs even when submitBatch fails; move the q.lastSubmittedAt assignment to
immediately after a successful return from t.submitBatch (i.e., only set
q.lastSubmittedAt and q.lastSentNonce and q.hasInFlight = true after submitBatch
returns nil), and ensure failure paths do not touch lastSubmittedAt (also update
the other occurrences referenced in the 397–420 region to follow the same
pattern), keeping submitBatch error handling as-is and adding any necessary
tests to cover transient submit failures.
- Around line 412-427: selectExpired can return arbitrarily many expired txs but
the current code appends them as a single flushWork (flushWork{from: from, txs:
expired}), bypassing the configured batch size and risking an unbounded Flow
transaction; change the logic to split the expired slice into bounded chunks of
at most t.config.TxMaxBatchSize (or the equivalent TxMaxBatchSize config) and
append one flushWork per chunk (each flushWork{from: from, txs: chunk}) instead
of a single large flushWork; keep the existing deletions and q.lastSubmittedAt
update but ensure the work queue receives multiple bounded batches so build/send
failures only affect a single chunk rather than the entire expired set.
- Around line 353-358: Empty queues removed by pruneStaleTxs never age out
because the deletion only checks q.lastSubmittedAt and that stays zero for EOAs
that never submitted; to fix, track a last-activity timestamp that pruneStaleTxs
updates (e.g., q.lastPrunedAt or a generic q.lastActivityAt) whenever it removes
txs and then change the idle deletion condition in the empty-queue branch (the
block that checks q.txs, q.hasInFlight, q.lastSubmittedAt and
idleQueueRetention) to consider either lastSubmittedAt OR the new last-activity
timestamp when deciding to delete stale empty queues; make the same change in
the other identical block (around lines 437-453) so empty queues pruned by
pruneStaleTxs can be removed after idleQueueRetention.
- Around line 254-260: Detect whether the nonce already existed and whether the
queue was empty before overwriting q.txs: capture wasEmpty := len(q.txs) == 0
and existed := (q.txs[tx.Nonce()] present) before assigning q.txs[tx.Nonce()] =
userTx; then only set q.windowDeadline and q.flushDeadline when wasEmpty is true
(i.e., on the first enqueue), so replacements of an existing nonce do not reset
the original deadlines; keep using t.config.TxCollectionWindow and
t.config.TxSubmissionSpacing for the deadline values.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 136b3b65-6469-4c8a-8a81-6d6ab310954a

📥 Commits

Reviewing files that changed from the base of the PR and between f79fb1a and 2127a6f.

📒 Files selected for processing (8)

• bootstrap/bootstrap.go
• cmd/run/cmd.go
• config/config.go
• models/errors/errors.go
• services/requester/nonce_aware_tx_pool.go
• services/requester/nonce_aware_tx_pool_test.go
• services/requester/nonce_provider.go
• tests/nonce_aware_tx_pool_test.go

[m-Peter]

That's a good remark. Maybe it would be worthwhile to introduce a dedicated DB index nonce, just like we have for the other models, such as blocks/transactions/receipts etc. It would be a lot more lightweight, compared to using the entire EVM state.

[m-Peter]

The previous condition of if q.hasInFlight || (len(q.txs) == 0 && t.spacingElapsed(q, now)) doesn't always guarantee that the q.txs is empty, so it would be more efficient to do the check for duplicates much earlier, and return early whenever possible.

[m-Peter]

Regardless of whether q.hasInFlight is true or false, whenever tx.Nonce() <= q.lastSentNonce, doesn't this imply that tx.Nonce() is inherently inflight? In which case, we wouldn't want to re-submit a transaction with tx.Nonce(). Also, I think that this check could be done before the condition of if q.hasInFlight || (len(q.txs) == 0 && t.spacingElapsed(q, now)) . What do you think?

[m-Peter]

That's correct, this should be possible, as long as both transactions are still in the pool, and tx.Nonce() > q.lastSentNonce.

[m-Peter]

We shouldn't be emitting that metric here, as it is meant to track the error rate during the build/submission of the Cadence transaction to the Flow network. We use it to track service/network errors, it wouldn't make sense to use it for state EVM txs that were pruned.

[m-Peter]

We seem to use quite frequently the len(q.txs) outcome, maybe we could add some helper methods on q, to view the txs state, whether it is Empty or has a single tx etc.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

services/requester/tx_mempool_test.go (1)

123-128: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Let the fake simulate GetBlockView() failures too.

GetBlockView() always returns nil, so the updated nonce-provider/view path can only exercise NonceView.GetNonce() failures. That leaves the separate view-acquisition error branch untestable from this suite.

♻️ Suggested test-double split

 type fakeNonceProvider struct {
-	nonce uint64
-	err   error
+	nonce    uint64
+	viewErr  error
+	nonceErr error
 }

 func (f *fakeNonceProvider) GetNextNonce(_ gethCommon.Address) (uint64, error) {
-	return f.nonce, f.err
+	return f.nonce, f.nonceErr
 }

 func (f *fakeNonceProvider) GetNonce(_ gethCommon.Address) (uint64, error) {
-	return f.nonce, f.err
+	return f.nonce, f.nonceErr
 }

 func (f *fakeNonceProvider) GetBlockView() (NonceView, error) {
+	if f.viewErr != nil {
+		return nil, f.viewErr
+	}
 	return f, nil
 }

Then add a small pool.Add(...) regression that injects viewErr and asserts the error is returned directly.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@services/requester/tx_mempool_test.go` around lines 123 - 128, The fake nonce
provider only simulates GetNonce failures, so the GetBlockView error branch in
the nonce-provider path is still untested. Update fakeNonceProvider.GetBlockView
in tx_mempool_test.go to return a configurable view error (for example via a new
field like viewErr), then add a regression test around pool.Add that injects
that error and asserts the error is returned directly from the view-acquisition
path.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@services/requester/nonce_provider.go`:
- Around line 80-90: The cache check in LocalNonceProvider.GetBlockView is using
a stale height snapshot taken before mu is acquired, so it can return cachedView
for an outdated block height under contention. Re-read or validate the current
height after locking in GetBlockView, and only return cachedView when the locked
height still matches the cachedHeight; otherwise rebuild or loop until the built
NonceView matches the latest EVM height. Keep the fix localized to GetBlockView
and the cachedHeight/cachedView path so Add and flush see a consistent nonce
state.

---

Nitpick comments:
In `@services/requester/tx_mempool_test.go`:
- Around line 123-128: The fake nonce provider only simulates GetNonce failures,
so the GetBlockView error branch in the nonce-provider path is still untested.
Update fakeNonceProvider.GetBlockView in tx_mempool_test.go to return a
configurable view error (for example via a new field like viewErr), then add a
regression test around pool.Add that injects that error and asserts the error is
returned directly from the view-acquisition path.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ccaadba1-89a8-4a96-8090-748f55f7a698

📥 Commits

Reviewing files that changed from the base of the PR and between 8f50460 and 7e9f185.

📒 Files selected for processing (6)

• cmd/run/cmd.go
• config/config.go
• models/errors/errors.go
• services/requester/nonce_provider.go
• services/requester/tx_mempool.go
• services/requester/tx_mempool_test.go

✅ Files skipped from review due to trivial changes (1)

• models/errors/errors.go

🚧 Files skipped from review as they are similar to previous changes (3)

• cmd/run/cmd.go
• services/requester/tx_mempool.go
• config/config.go

[zhangchiqing]

We don't have to lock the nonce provider when querying the view. We only need to lock when reading and updating the cached view.

[vishalchangrani]

k done - I also added a few additional metrics to track the EOA based cache (hits vs misses etc.) to help debug on testnet and mainnet.

[m-Peter]

LGTM!

[vishalchangrani]

Testnet testing summary

This change was exercised end-to-end on a live Flow EVM testnet gateway (chain id 545). A build off this branch was deployed with the mempool enabled (--tx-mempool-mode --tx-state-validation=local-index) using the default knobs (300ms collection window, 1200ms submission spacing, 30s pool TTL, max batch size 5, max nonce gap 500).

Testing was driven by a Go harness that signs and fires eth_sendRawTransaction bursts and verifies the resulting on-chain receipts. Observed behavior was cross-checked against the gateway's debug logs and the new Prometheus metrics (txpool_submissions_total, txpool_nonce_view_cache_total, txpool_queues, txpool_queued_transactions, transactions_dropped_total).

Round 1 — single-EOA functional and sustained load

• Fast path: a single expected-nonce tx was submitted immediately.
• In-order bursts: 50 and then a sustained 200 sequential-nonce txs fired concurrently were all mined. The pool coalesced them into consecutive-nonce batches of 5 and paced submissions ~1.2s apart (measured ~1.23s/batch across 40 back-to-back batches in the 200-tx run), so batching plus submission-spacing held precisely under sustained load and consecutive Flow txs landed in separate blocks (ordering protection).
• Out-of-order gap: a tx sent ahead of a nonce gap was correctly held, then flushed once the gap was filled.
• Exact duplicate: resending an already-queued tx was rejected ("transaction already in pool").
• TTL submit-anyway: a tx parked behind a gap that never filled was submitted anyway ~30s later (per the TTL) rather than silently dropped — an observable on-chain outcome.
• Every accepted tx was mined in order, with zero silent drops.

Round 2 — multi-EOA concurrency

• 3 funded EOAs each burst 50 txs concurrently (150 total).
• The pool maintained an independent per-EOA queue for each (queue-count metric peaked at 3) and processed them in parallel, applying the 1.2s submission spacing per-EOA. Ordering is protected within each EOA without throttling across EOAs — the 3-EOA run completed in roughly the time a single EOA's 50-tx burst would.
• All 150 mined; zero drops.

Conclusion

All mempool behaviors (fast path, burst batching, submission spacing, out-of-order hold-and-fill, duplicate rejection, TTL submit-anyway, and per-EOA queueing) behaved as designed on live testnet — confirmed via both on-chain receipts and gateway logs/metrics — with no silent drops in any scenario.

[vishalchangrani]

Testnet testing summary

This change was exercised end-to-end on a live Flow EVM testnet gateway (chain id 545). A build off this branch was deployed with the mempool enabled (--tx-mempool-mode --tx-state-validation=local-index) using the default knobs (300ms collection window, 1200ms submission spacing, 30s pool TTL, max batch size 5, max nonce gap 500).

Testing was driven by a Go harness that signs and fires eth_sendRawTransaction bursts and verifies the resulting on-chain receipts. Observed behavior was cross-checked against the gateway's debug logs and the new Prometheus metrics (txpool_submissions_total, txpool_nonce_view_cache_total, txpool_queues, txpool_queued_transactions, transactions_dropped_total).

Round 1 — single-EOA functional and sustained load

• Fast path: a single expected-nonce tx was submitted immediately.
• In-order bursts: 50 and then a sustained 200 sequential-nonce txs fired concurrently were all mined. The pool coalesced them into consecutive-nonce batches of 5 and paced submissions ~1.2s apart (measured ~1.23s/batch across 40 back-to-back batches in the 200-tx run), so batching plus submission-spacing held precisely under sustained load and consecutive Flow txs landed in separate blocks (ordering protection).
• Out-of-order gap: a tx sent ahead of a nonce gap was correctly held, then flushed once the gap was filled.
• Exact duplicate: resending an already-queued tx was rejected ("transaction already in pool").
• TTL submit-anyway: a tx parked behind a gap that never filled was submitted anyway ~30s later (per the TTL) rather than silently dropped — an observable on-chain outcome.
• Every accepted tx was mined in order, with zero silent drops.

Round 2 — multi-EOA concurrency

• 3 funded EOAs each burst 50 txs concurrently (150 total).
• The pool maintained an independent per-EOA queue for each (queue-count metric peaked at 3) and processed them in parallel, applying the 1.2s submission spacing per-EOA. Ordering is protected within each EOA without throttling across EOAs — the 3-EOA run completed in roughly the time a single EOA's 50-tx burst would.
• All 150 mined; zero drops.

Conclusion

All mempool behaviors (fast path, burst batching, submission spacing, out-of-order hold-and-fill, duplicate rejection, TTL submit-anyway, and per-EOA queueing) behaved as designed on live testnet — confirmed via both on-chain receipts and gateway logs/metrics — with no silent drops in any scenario.

Here is how the new metrics looked on Grafana during the test:

[vishalchangrani]

hey @zhangchiqing @m-Peter - thanks for the review on this PR! I have this another PR that port this change over to the soft-finality branch #977. I tested that one as well on testnet.

(I will create an issue to merge the soft-finality branch into main behind a feature flag after all these changes)