fix(telos): restore replay parity and add opt-in transaction retry

[TheJudii]

Fixes canonical chain-ID-3 withdrawal burn handling so replayed state matches the authoritative Telos account delta, and adds opt-in native transaction retry through nodeos send_transaction2 while retaining push_transaction as the default.

Also enables canonical WebSocket subscriptions and refreshes the controlled dependency-security policy. The corrected build has been qualified side by side on all three mainnet beta origins and the testnet beta path without modifying or replacing incumbent RPC services.

[TheJudii]

Testnet beta qualification update for d20afde36b053b01bea7f8a8e89da51c661c87d7:

• Reproducible maxperf release SHA-256: d97ea0b453afd6bea70c849f09d0cedc316f5e3cdf5cc065d28953cb15f2d29f.
• The identical binary is active side-by-side on rpc9, rpc10, and rpc11; every candidate, companion, and beta router service is healthy. Legacy testnet Reth and mainnet were not changed.
• --telos.transaction-retry is enabled on all three candidates. A real transaction submitted through the public beta endpoint finalized successfully with receipt status 1: 0xeeb8bdc09db87fe155ecfbd846979909fa320a577668cf9a5ad8d986833f75b9. Candidate/router/legacy receipt parity passed, and nodeos observed send_transaction2 with retry enabled.
• Canonical WebSocket newHeads subscriptions delivered live blocks directly on all three candidates and through wss://rpc-beta.testnet.telos.net. Pending-transaction and syncing subscriptions remain rejected because nodeos, not Reth's local pool, is authoritative.
• Public three-origin qualification passed for https://rpc-beta.testnet.telos.net: chain ID 41, rpc9/rpc10/rpc11 all observed, head spread within the qualification limit, and a live WebSocket block notification received.
• Failover passed with rpc9's new stack intentionally offline: 60 HTTP sample rounds were served only by rpc10/rpc11 and 20/20 WebSocket sessions returned chain ID 41. rpc9 was restored and caught up afterward.
• Full Telos CI passed: https://github.com/telosnetwork/telos-reth-2/actions/runs/31753697829
• Reproducible Ubuntu 22.04/24.04 build comparison passed byte-identically: https://github.com/telosnetwork/telos-reth-2/actions/runs/31753811477

Sealed operational evidence is stored on rpc9 under /telos/telos-reth-2-testnet-beta-v1/evidence/, including the live retry transaction, public qualification, and failover records.

[TheJudii]

Mainnet beta rollout update for d20afde36b053b01bea7f8a8e89da51c661c87d7:

• Transaction retry is enabled end to end on rpc9, rpc10, and rpc11 behind https://rpc-beta.telos.net. Each Reth candidate uses --telos.transaction-retry; each local mainnet nodeos HTTP service has the same 1 GB / 20 second / 120 second retry configuration qualified on testnet.
• The identical reproducible binary is active on all three origins: SHA-256 d97ea0b453afd6bea70c849f09d0cedc316f5e3cdf5cc065d28953cb15f2d29f.
• Rollout was one withdrawn beta origin at a time. During each nodeos restart, both rpc.telos.net and the surviving beta pool passed every continuous read check (160/160 for rpc9, 120/120 for rpc10, and 120/120 for rpc11). Legacy Reth services were not restarted or replaced.
• Direct qualification passed independently on every origin: chain ID 40, advancing heads, zero-spread comparison with the colocated incumbent, canonical hash parity at a recent block and block 483297607, and successful receipt parity for 0x04fd605da353a771a85a0536832587cf7861d1d9815f1fefd47d3f4a7548f7b8 after excluding the incumbent's known effectiveGasPrice presentation difference.
• A previously signed raw transaction was resubmitted on each origin and through the public beta. It reached nodeos through the retry-enabled forwarding path and returned the expected already-used nonce error; no new economic mainnet transaction was created.
• Direct and public WebSocket newHeads subscriptions passed. Unsupported pending-transaction subscriptions remain rejected intentionally because nodeos is authoritative.
• Public three-origin qualification passed with rpc9/rpc10/rpc11 observed and a three-block maximum head spread at the time of capture.
• Failover passed with rpc9 withdrawn: 60 HTTP rounds and 20 WebSocket sessions succeeded through rpc10/rpc11. rpc9 was restored afterward; the final audit showed all services active and all candidate heads within one block.

Sealed evidence is stored on rpc9 under /telos/telos-reth-2-mainnet-v1/evidence/:

• mainnet-beta-transaction-retry-public-20260813.json — SHA-256 8abdcbbab1b86540565b6933889a8e6ba0ee573e13ff24c7b25d9648aa9fa7ac
• mainnet-beta-public-failover-20260813.json — SHA-256 f6cad6d5a0cae5c4af9eb561ebbcd1f3bc24ccc32b3136272e4bcd140b354849

This changes only the mainnet beta pool. rpc.telos.net remains on the incumbent stack.

[TheJudii]

Fresh transaction-retry validation passed on 2026-08-14 UTC against build d20afde36b053b01bea7f8a8e89da51c661c87d7 / binary SHA-256 d97ea0b453afd6bea70c849f09d0cedc316f5e3cdf5cc065d28953cb15f2d29f.

Testnet:

• Real candidate submission: 0xef89cf0d8943d6048473b4027379ec799ad7a75850fe360de89bc0f1528d6d2c, included at block 439561996.
• Real beta-router submission: 0xa0089fba733eebcf263e8160615d3d79b71815dc5ac676482ac887ab9b98833f, included at block 439561998.
• Real public-beta submission via rpc10: 0x3e9dc2b98385c324bcc7c5b54388be1be1b47974564850a6f0396b47250c47c0, status 1 at block 439562034.
• Instrumented nodeos path observation recorded 3 /v1/chain/send_transaction2 calls with HTTP 202 and 0 /v1/chain/push_transaction calls. Request bodies were not recorded.
• Duplicate and wrong-chain submissions were rejected safely.
• Inclusion, finality, canonical head/hash parity, and identical receipt fields were verified across the rpc9/rpc10/rpc11 new candidates and legacy.
• rpc9 was restored to its normal direct nodeos topology after the temporary path observer; incumbent services were not modified.

Mainnet beta:

• A previously mined raw transaction was resubmitted as a non-economic forwarding probe via rpc10. It reached nodeos and returned the expected safe nonce rejection; the retry-disabled error was absent. No raw transaction was disclosed.

Sealed evidence on rpc9:

• /telos/telos-reth-2-testnet-beta-v1/evidence/transaction-retry-live-20260814.json — 402daf3f434b5a4940730d44d532cd41746fd0232999cadfd29b13c4ac8e1662
• /telos/telos-reth-2-testnet-beta-v1/evidence/testnet-beta-public-transaction-retry-20260814.json — 8b637d363901bf1a7f0c53e354a317f7012cad352a324dfa005fb599262685de
• /telos/telos-reth-2-testnet-beta-v1/evidence/transaction-retry-validation-20260814.json — 1e44181feda9e292d358b408ef880ea4a22a3d99678c2a82aefc23bd5ac1cb26
• Path-only capture: /telos/telos-reth-2-testnet-beta-v1/evidence/transaction-retry-paths-20260814.jsonl — 150765776e601d5066b0104d1e21e63a29125726e22103c05fa8746f4bfcc839

[TheJudii]

Follow-up boundary test: a nonce-gapped EVM transaction is not a supported transaction-retry case.

I submitted nonce 25 while the native contract expected nonce 24, then mined nonce 24 without resubmitting nonce 25 afterward. The higher-nonce transaction remained absent after 150 seconds. This is expected: Antelope defines retry_trx as retry for a validated transaction. The future nonce fails eosio.evm validation with incorrect nonce, so nodeos does not admit it to the retry queue.

This does not invalidate the successful supported-path qualification above:

• retry-enabled requests use /v1/chain/send_transaction2 with retry_trx: true;
• valid real transactions were accepted, mined, finalized, and matched across all new candidates and legacy;
• invalid/duplicate submissions fail safely;
• the mainnet beta forwarding path no longer returns Transaction retry not enabled on node.

The remaining retry-specific chaos test is to withhold P2P propagation from an otherwise valid native transaction for more than the configured 20-second interval and prove retransmission. That needs an isolated nodeos instance; it was not forced on the shared testnet nodeos because doing so could disrupt the incumbent testnet RPC.

Boundary evidence:

• /telos/telos-reth-2-testnet-beta-v1/evidence/transaction-retry-nonce-gap-boundary-20260814.json
• SHA-256 717171cfc35c5c893837e1453b75de37cd99b17d48ecc45ae5878f58f15442e0

Reference: https://github.com/AntelopeIO/leap/releases/tag/v3.1.0#new-transaction-submission-api

[TheJudii]

Isolated P2P-loss retry qualification — PASS

A real Telos EVM testnet transaction was submitted through the PR build while its dedicated, loopback-only nodeos instance had zero P2P connections.

• P2P withheld: 25.439952 s
• Receipts observed on the direct new and legacy RPCs during the hold: 0
• Client submissions after P2P reconnect: 0
• Reconnect requested: 2026-08-14T15:24:32.260164Z
• nodeos logged the peer connection: 2026-08-14T15:24:32.388Z
• Receipt block timestamp: 2026-08-14T15:24:33Z
• Receipt observed: 1.225669 s after reconnect
• New/legacy receipt parity: exact
• Receipt status: success (0x1)
• Finality: confirmed on both
• Public testnet beta: same successful receipt

Transaction: 0x25cc3f2ea3e0ebb6e76fa6197ea540003cb9a73396743b7844928be968f525f4

This proves the retry queue retained and propagated a validated transaction across a controlled loss of P2P connectivity. The forwarding client had already exhausted its expected blocking timeout before reconnect; it did not resubmit afterward.

Safety/cleanup:

• incumbent services and routing were not modified
• recovery snapshot was taken online before the test
• transient nodeos was stopped after the run and its private listeners verified absent
• temporary signing material was removed
• testnet beta, mainnet beta, and production mainnet passed post-test health checks

Sealed evidence on rpc10:

• transaction-retry-isolated-p2p-chaos-20260814-v2.json — SHA-256 1a238b175e5c62665aa132316ee34bfff8b89ffb1f6b50afbc72ed7c7b857f7d
• transaction-retry-isolated-p2p-chaos-20260814-manifest.json — SHA-256 c1c0dd4c1ea8d1e1aa9a48a69c74fe2f61e29372abdbd77a5c618903dfe9d903

Note: the native speculative head can be numerically ahead of the canonical EVM receipt block. The ordering assertion therefore uses the peer-connection log and canonical receipt block timestamp, not a cross-layer height comparison.

[TheJudii]

rpc11 testnet retry configuration rollout — PASS

rpc11's testnet nodeos HTTP instance now has the same transaction-retry configuration as rpc9 and rpc10:

• transaction-retry-max-storage-size-gb = 1
• transaction-retry-interval-sec = 20
• transaction-retry-max-expiration-sec = 120

Rollout safety:

• backed up and checksum-verified the previous config
• atomically drained rpc11 from rpc-beta.testnet.telos.net
• served 356/356 successful requests with zero failures through rpc9/rpc10 during a 150.329-second drain; WebSockets also passed
• restarted only /telos-testnet/nodeos-http under an automatic config rollback guard
• rpc11 caught up to the reference SHiP node at 0-block lag
• restored the exact prior Cloudflare Worker version (7f337871-7965-4d1c-962a-6d5f28245a52)
• legacy Reth was not restarted or modified

Retry-path validation:

• startup logs enumerate all three active retry settings
• a safe duplicate transaction submitted directly through rpc11's new Reth reached native execution and returned the expected incorrect nonce classification, not Transaction retry not enabled on node
• no state changed

Post-restore full qualification:

• 120 public reads: rpc9 37, rpc10 39, rpc11 44
• 60 safe signed duplicate submissions: rpc9 20, rpc10 19, rpc11 21
• seven canonical-history parity points passed
• deployment/state-change/revert/transfer and contract-state matrix passed
• 12 WebSocket heads received and unsubscribe passed
• every relevant service was active/running; historical restart counters were recorded, not reset

Sealed evidence on rpc11:

• rpc11-transaction-retry-rollout-20260814.complete.json — SHA-256 fc2869feaea5d5990d11d27c194d5502306783d7d65b93e6585be67e389fd380
• testnet-beta-post-rpc11-retry.complete.json — SHA-256 dbc13662073c9e8e861b59b852a455659cedf8cc4daaf0f5d1c4982c3694711f
• rpc11-nodeos-http-retry-restart-20260814.sh — SHA-256 74140f282dcf51bdef77fd29d065764012aa97d6574bb030d0628950cf36185d

Result: transaction retry is now configured consistently across rpc9, rpc10, and rpc11 testnet beta origins.

[TheJudii]

Final qualification evidence for head 5d40d86:

• All required GitHub checks pass, including Telos tests, security, clippy, docs, state tests, RocksDB e2e, and all three crate-check shards.
• Runtime code commit 13ca85c is deployed on mainnet beta rpc9/rpc10/rpc11; later head changes are documentation/security policy only. Binary SHA-256: 414640c49faa416748b9cf0452ceff8801b15a3c185c2b79bde88510b50d1e3f.
• Replayed the former failure at block 483427410 and matched the incumbent canonical hash/state-visible balances, then caught all three origins up with zero lag and current finalized parity.
• Public mainnet beta passed 36 allowed RPC methods, filter lifecycles, 150 concurrent requests across all three origins, historical receipt/storage parity, 64-call batch acceptance and 65-call rejection, CORS, and 12 linked WebSocket heads.
• Transaction retry passed 60 safe canonical duplicate submissions distributed across all three origins; every response was the expected stale-nonce rejection and the original receipt/nonce remained unchanged.
• Controlled rpc9 drain passed: 60 HTTP requests continued on rpc10/rpc11 with 18 explicit failovers, WebSocket delivered eight linked heads, and all three origins passed again after restoration.
• Exact-binary clean restart and forced-crash recovery passed on rpc9; current-binary restore/open/restart parity passed against an isolated database copy.
• Testnet signed-write, duplicate, nonce-boundary, and isolated P2P disconnect/reconnect retry tests passed. The public three-origin HTTP/WebSocket gate passed again after the canary diagnostic.
• The prior isolated full-stack Engine API reorg result is reused because this diff changes Telos EVM/RPC/storage policy but no engine or forkchoice implementation. Evidence SHA-256: efab2477295de0bcbb1994f5309a70635fff268bc998888a43a7799de3f5cd35.
• Incumbent mainnet and testnet services were not modified or replaced.

Sealed runtime evidence is stored on rpc11 under /telos/telos-reth-v2-bootstrap/retry-replay-13ca85cfa-v1/evidence/ with matching local/remote checksums.

[TheJudii]

Final side-by-side qualification is complete for merge commit cd30fb5b39d2ef495b851101d5c11f560b8f9348.

• Exact maxperf artifact SHA-256: 0338d4f9e1e274df605056b9905e2c3231e52103fea7b588ace61b4a3e5e43c4
• The artifact is running on rpc9, rpc10, and rpc11 for both mainnet beta and testnet beta.
• Every rolling gate passed live-head, canonical-hash, and finalized-hash parity against an independent or incumbent reference.
• Mainnet public beta passed the 36-method RPC matrix, filter lifecycles, 150 concurrent calls across all three origins, 64/65 batch enforcement, historical receipt/storage parity, 12 linked WebSocket heads, and 60 duplicate signed submissions across all origins with unchanged canonical state.
• Testnet public beta passed all three origins, full-history witnesses, transaction/contract reads, duplicate signed submissions, batch/CORS, and WebSockets.
• A drained rpc9 passed clean restart and forced-SIGKILL automatic recovery with 20 post-recovery parity samples.
• The optional future-nonce test confirmed the documented Spring boundary: transaction retry applies after validation; a nonce-gapped transaction is rejected before admission and must be resubmitted by the client after the nonce gap closes.
• Complete early history remains served by the independently copied, digest-pinned retained-history archive described in docs/telos/history-routing.md; it is not the incumbent's live database. This qualification does not claim that sparse Telos Reth 2 alone is a full archive or authorize incumbent retirement.

The upgraded stack is qualified for controlled side-by-side production canary admission. The incumbent was not modified or replaced.