Skip to content

fix(relayer): recover Walrus register destroy_zero from stale WAL price (#351)#352

Open
ducnmm wants to merge 2 commits into
devfrom
fix/walrus-register-destroy-zero-stale-price
Open

fix(relayer): recover Walrus register destroy_zero from stale WAL price (#351)#352
ducnmm wants to merge 2 commits into
devfrom
fix/walrus-register-destroy-zero-stale-price

Conversation

@ducnmm

@ducnmm ducnmm commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes #351 — mainnet writes were failing 100% at the register_sponsor phase with an Enoki dry_run_failedMoveAbort in 0x2::coin::destroy_zero (ENonZero).

Not an Enoki fault. @mysten/walrus's #withWal helper pre-funds an exact WAL payment (storageUnits × price × epochs) from the client's cached systemState price, then asserts the coin is empty via coin::destroy_zero. When mainnet storage/write price drifts down between the cached read and execution, the contract deducts less WAL than we split off and the leftover trips destroy_zero. Enoki only surfaces it during its budget dry-run.

Evidence (prod, 2026-07-02)

  • On-chain storage_price_per_unit_size moved 71464 → 70922 within ~15 min.
  • After a relayer restart, 6 writes succeeded then the same sidecar client flipped to 100% destroy_zero at the next price move — a time/price transition, not per-user/blob/wallet.
  • classification=permanent retryable=false, so every write was Dead-marked with no retry.

Why it didn't self-heal (two gaps)

  1. The sidecar auto-refresh only fired on isMoveAbortBalanceSplit (matches balance+split); this message says destroy_zero, so refreshWalrusClient() never ran.
  2. The Rust worker swept it into the MoveAbort → Permanent catch → Apalis never retried.

A newer SDK does not fix it: @mysten/walrus 1.2.x has no cost/#withWal/destroy_zero change (installed 1.1.7, latest 1.2.3).

Fix

  • enoki.ts — add isMoveAbortWalDestroyZero detector.
  • walrus-upload.tsrefreshWalrusClient() on the abort so the retry rebuilds against the live price.
  • jobs.rs — classify the register destroy_zero abort as Transient (retryable), disjoint from the balance::split gas-budget path.
  • config.ts — drop WALRUS_CLIENT_MAX_AGE_MS default 30m → 60s so the cached price tracks a live-drifting mainnet price (still env-overridable).

Testing

  • New TS detector suite (7 cases) + Rust classification test using the verbatim prod error.
  • Full suites green: 85 TS, 40 jobs Rust tests.

Deployment

Already hot-deployed to the prod relayer (Railway, deployment 26742fd0). Verified live: clean sidecar boot, writes succeeding, and refreshed reason=max_age lines confirm the new 60s max-age is active. This PR lands the same change on dev.

Follow-up worth filing upstream to Mysten: the exact-change #withWal + coin::destroy_zero pattern is inherently racy against a moving on-chain price — it should tolerate leftover (return change) or add a small buffer.

…ce (#351)

Mainnet writes were failing 100% at the register_sponsor phase with an Enoki
dry_run_failed -> MoveAbort in 0x2::coin::destroy_zero (ENonZero). Root cause is
not Enoki: @mysten/walrus '#withWal' pre-funds an *exact* WAL payment computed
from the client's cached systemState price, then asserts the coin is empty via
coin::destroy_zero. When mainnet storage/write price drifts down between the
cached read and execution, the contract deducts less WAL than we split off and
the leftover trips destroy_zero. Verified on prod: on-chain price moved
71464->70922 within ~15 min, and a fresh relayer boot served 6 writes before the
same client flipped to 100% destroy_zero at the next price move.

Two gaps let this fail hard instead of self-healing:
- The sidecar's auto-refresh only fired on isMoveAbortBalanceSplit ('balance'
  + 'split'); this message says destroy_zero, so refreshWalrusClient never ran.
- The Rust worker swept it into the MoveAbort -> Permanent catch, so Apalis
  Dead-marked every write with no retry.

Fix:
- enoki.ts: add isMoveAbortWalDestroyZero detector.
- walrus-upload.ts: refreshWalrusClient() on the destroy_zero abort so the retry
  rebuilds against the live price.
- jobs.rs: classify the register destroy_zero abort as Transient (retryable),
  disjoint from the balance::split gas-budget path.
- config.ts: drop WALRUS_CLIENT_MAX_AGE_MS default 30m -> 60s so the cached
  price tracks a live-drifting mainnet price (still env-overridable).

Tests: new TS detector suite (7) + Rust classification test using the verbatim
prod error. Full suites green (85 TS, 40 jobs).
The sidecar hardcoded getJsonRpcFullnodeUrl(mainnet) and ignored the SUI_RPC_URL
env, so it could not be pointed at a dedicated RPC when the public fullnode pool
degrades. Observed today: the public pool served stale reads (a blob certified
at 00:49:14 read back as 'does not exist' at 00:49:49, 35s later), failing
uploads at get_blob/certify/metadata. Honor an explicit SUI_RPC_URL override so
ops can switch to a healthy RPC via env; fall back to the network default.
@DrVelvetFog

Copy link
Copy Markdown

Reviewed this against the SDK internals — diagnosis and fix both look right. The two-gaps writeup nails why it didn't self-heal; I'd independently landed on the same destroy_zerobalance::split distinction, and keeping the two detectors disjoint is the load-bearing call — a split ENotEnough is genuinely ambiguous with gas-pool exhaustion, so you don't want a starved gas wallet reclassified as recoverable stale-price. Register destroy_zero → Transient plus the 30m → 60s max-age both read as correct for tracking a live-drifting price.

On the upstream follow-up you flagged — it's already in flight, and it's exactly what you described:

So once that lands and you bump the SDK, the recovery here becomes the defense-in-depth layer: the buffer means the down-drift destroy_zero mostly stops firing in the first place, and your refresh-and-retry still catches anything that drifts past it. Good to have both sides covered.

LGTM.

@harrymove-ctrl harrymove-ctrl self-requested a review July 9, 2026 05:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mainnet writes fail on both prod relayers: Enoki dry_run_failed (MoveAbort balance::destroy_zero in sponsored upload)

3 participants