QA: `RPC Integration Tests Latest` flaky — testing node can't converge to reference tip after restart (nodes not synced)

[yperbasis]

Summary

QA - RPC Integration Tests Latest intermittently fails before any RPC test runs, at the "Run Erigon and wait for sync" gate, with:

Error: nodes not synced, testingNode=<X> referenceNode=<Y>

It's a non-deterministic tip-convergence race, not a code regression — on the same commit the parallel job frequently passes while the serial job fails.

Symptom / evidence

The test builds a local Erigon, starts it from a freshly-mirrored datadir, and waits for it to reach the same latest block as the remote reference node (REFERENCE_SYSTEM_HOST_ADDRESS, always at tip) before comparing RPC responses. The freshly-started local node lags 2–3 blocks behind the moving tip and the gate gives up after ~3 retries.

Example — run 28143986802 (release/3.5, commit 420c5ef6), serial job:

03:22:12  testing node starts from mirrored datadir
03:22:22 … 03:23:34  currentBlock=highestBlock=25391905   ← ~80s not advancing
                                                            (cold cache + Caplin checkpoint-sync / forkchoice handshake)
03:23:46  "node in sync! height = 25391925"               ← catches up in a burst
03:24:19  nodes not synced  testingNode=25391927  referenceNode=25391929
03:24:28  nodes not synced  testingNode=25391927  referenceNode=25391930
03:24:37  nodes not synced  testingNode=25391927  referenceNode=25391930   → FAIL

The parallel job of the same commit (warm datadir, already near tip) converged (node in sync! height = 25391959) and ran 194 tests.

Why it's a flake, not a regression

• Same commit: serial fails, parallel passes.
• Recurs on main (failed 2026-06-20 677d3aca, 06-21 9e2e4d09; passed 06-22…06-25) and across unrelated commits on release/3.5 (e.g. it also failed on the commit before 420c5ef6 with the identical signature).
• The gate fails before RPC comparison; the node is healthy, just a few blocks behind a moving tip.

Root cause (harness)

The convergence gate requires a freshly-restarted node to match the remote reference's current latest block within a short retry window (~3 attempts / ~20s). A cold-started node needs ~80s to resume following the tip (cold page cache + CL handshake), by which point the reference has advanced — so it can't converge on the first (cold) attempt.

Suggested fixes (any one)

• Widen / make adaptive the convergence retry window or timeout.
• Warm the node up to the tip before arming the sync gate.
• Compare both nodes at a fixed agreed block height instead of "current latest".

Open secondary question

Is ~80s to resume following the tip after a restart expected, or itself worth a look?

[VBulikov]

yperbasis added Imp1 as Importance

[mriccobene]

Main branch outcomes

On main branch the serial version of the test has been failing for a few days: the Erigon version under test is unable to reach the tip (step Run Erigon and wait for sync); in the logs I see repated failures like this one:

[INFO] [07-06|01:02:05.455] [4/6 Execution] serial starting          from=25469977 to=25470072 initialTxNum=3629467323 initialBlockTxOffset=0 lastFrozenStep=9289 initialCycle=true isForkValidation=false
[DBUG] [07-06|01:02:05.498] [commitment] processed                   block=25469977 txNum=3629467818 keys=1.99k keys/s=333.17k mode=hex-patricia-hashed bufmode=direct spent=5.981916ms rootHash=
[INFO] [07-06|01:02:05.499] [4/6 Execution] serial done              in=43.130445ms buf=252.8KB/512.0MB blk=25469977 blks=1 blk/s=23 txs=0 tx/s=0 gas/s=729.49M stepsInDB=1.44 step=9291.4 alloc=8.7GB sys=12.3GB isForkValidation=false isApplyingBlocks=true
[DBUG] [07-06|01:02:05.499] [4/6 Execution] error while executing stage err="hash sort failed: followAndUpdate: GetAccount for key cb53a818042db647cb01672751b0e4efca29cf55 failed: code hash mismatch: account 'c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470' != codeHash 'fc042a3345c0f2bcd299bb051be7dbb37f5d7b5664cbaced140fbbbf546a4bbd'"
[WARN] [07-06|01:02:05.499] Cannot update chain head                 hash=0xc9a3230d57aa0d863e933b36a0646564f5a5b8f611cd5dd92bfada7923553c1e err="updateForkChoice: [4/6 Execution] hash sort failed: followAndUpdate: GetAccount for key cb53a818042db647cb01672751b0e4efca29cf55 failed: code hash mismatch: account 'c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470' != codeHash 'fc042a3345c0f2bcd299bb051be7dbb37f5d7b5664cbaced140fbbbf546a4bbd'"
[DBUG] [07-06|01:02:05.499] [dispatcher] PublishOverlay: publishing  sdNil=true
[DBUG] [07-06|01:02:05.499] updateforkchoice failed                  err="updateForkChoice: [4/6 Execution] hash sort failed: followAndUpdate: GetAccount for key cb53a818042db647cb01672751b0e4efca29cf55 failed: code hash mismatch: account 'c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470' != codeHash 'fc042a3345c0f2bcd299bb051be7dbb37f5d7b5664cbaced140fbbbf546a4bbd'"
[WARN] [07-06|01:02:05.499] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of new fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execution] hash sort failed: followAndUpdate: GetAccount for key cb53a818042db647cb01672751b0e4efca29cf55 failed: code hash mismatch: account 'c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470' != codeHash 'fc042a3345c0f2bcd299bb051be7dbb37f5d7b5664cbaced140fbbbf546a4bbd'"
[DBUG] [07-06|01:02:05.499] clstage finish                           app=caplin stage=ForkChoice in=164.254865ms next=CleanupAndPruning
[DBUG] [07-06|01:02:05.503] pruning data column sidecars             cutoff_slot=14570000
[DBUG] [07-06|01:02:05.507] clstage finish                           app=caplin stage=CleanupAndPruning in=8.241895ms next=SleepForSlot

Note that this failure pattern is not present in the parallel version log.
So the test indicates there may be a regression in the serial code.
You can inspect logs, for example, here https://github.com/erigontech/erigon/actions/runs/28760645917.

Release/3.5 branch outcomes

Of the last 9 executions since July 1st, 3 have failed:

• 2 due to mismatches in RPC responses (first, second)
• 1 due to the timing issue reported here (logs)

To resolve this timing issue, please, @lupin012, change the rpc-test suite to allow more time for a sync window.

[yperbasis]

Split: main-branch serial regression → #22321

The main-branch failure in @mriccobene's comment above is a distinct regression, not this timing/convergence flake: on main the serial variant fails deterministically with a code hash mismatch in the [4/6 Execution] commitment hash-sort, so updateForkChoice never lets the node reach tip. That pattern is absent from the parallel variant.

I've split it into #22321 so it can be bisected/fixed on its own track.

This issue (#22036) now tracks only the release/3.5 convergence-timing flake — a cold-started node lagging a moving reference tip at the sync gate. For context, the release/3.5 status from the comment: of the last 9 runs since July 1st, 3 failed — 2 RPC-response mismatches (a separate matter) and 1 this timing issue. The fix here remains the sync-window change requested of @lupin012.

[lupin012]

@yperbasis @mriccobene:
The default convergenct time is changed from 10 to 300
[main]: (contains also possibility to change default as config param )

• integration-tool: Make sync convergence retries configurable (--sync-retries) default 300 rpc-tests#584 (v2.22.0)
• QA: increase timer sync verify with reference node #22279

[3.5]:

• [v2.13.1] (branch release/3.5)
• [3.5] QA: Modified default wait sync latest from 10 to 300 #22280

09/07: Since #22280 was merged, 7 consecutive release/3.5 runs have completed without any recurrence of the convergence-timing flake.