Skip to content

fix: pay and finalize multi-batch merkle uploads (antd >= 0.12.0) - #162

Open
Nic-dorman wants to merge 2 commits into
masterfrom
bug/multi-batch-merkle-uploads
Open

fix: pay and finalize multi-batch merkle uploads (antd >= 0.12.0)#162
Nic-dorman wants to merge 2 commits into
masterfrom
bug/multi-batch-merkle-uploads

Conversation

@Nic-dorman

Copy link
Copy Markdown
Member

Fixes multi-batch merkle uploads against antd >= 0.12.0 (V2-1056), plus a latent allowance-formula bug the e2e exposed.

Linear: https://linear.app/autonominetwork/issue/V2-1056/multi-batch-merkle-uploads-fail-merklebatches

What

antd >= 0.12.0 splits uploads larger than one merkle tree (256 fresh chunks ≈ 1 GiB) into multiple MerkleBatches and leaves the legacy mirror fields empty, so the worker previously submitted an empty commitments list and failed opaquely.

  • Batch planning (merkleBatchPlan): MerkleBatches when present (antd >= 0.12.0 always populates it for merkle, len >= 1), legacy single-batch synthesis as the pre-0.12 fallback, and a clear "needs antd >= 0.12.0" error when both are empty. All batches are validated (depth 1-8, 16 candidates/pool, parseable amounts) before any money moves.
  • Payment loop (payMerkleBatches): one payForMerkleTree tx per batch (no ABI change — payForMerkleTree2() in antd docs is a naming slip for PaymentVaultV2's function, confirmed against evmlib + ADR-0003). Each confirmed batch is recorded before the next is paid (V2-426): one transactions row per batch, tx_hash = that batch's winner pool hash.
  • Single approve tx (Signer.EnsureMerkleAllowance): pre-approves the whole plan's worst case, so the per-batch approval short-circuits (1 approve instead of N).
  • Finalize: FinalizeMerkleUploadMulti with the index-aligned winner list when > 1 batch; the single-batch path keeps the legacy FinalizeMerkleUpload call unchanged.
  • Salvage on mid-plan failure: a definitive payment failure (revert/RPC error) after >= 1 paid batch best-effort-finalizes with "" for unpaid batches — the paid batches' chunks store (surfaced as PARTIAL_UPLOAD), then the upload fails preserve-paid (errPaidNoRetry). A confirmation timeout never salvages: the tx may still mine, and skipping keeps the manual full finalize possible (the error carries the pending tx hash + batch index). If salvage unexpectedly fully succeeds (unpaid batches deduped), the upload completes normally.
  • Allowance/cost formula fix (second commit, found live): the old bound — "sum over pools of max candidate" — is V1-era. PaymentVaultV2 charges median16(winner pool quotes) * 2^depth and picks the winner at execution time (PaymentVaultV2.sol:140), so batch 1 of the e2e reverted with ERC20InsufficientAllowance. Both the allowance bound and the max_gas_fee estimate now use the contract-accurate worst case (highest pool median << depth, summed across batches). The single-batch path had the same latent under-approval bug.

E2E proof (local devnet, 2026-08-24)

24-node devnet (merkle pools need 16 candidate peers; the 10-node preset can't quote) + anvil + antd v0.12.0, this branch's indelible:

  1. Happy path: 1.15 GB random file → plan batches=2 (depth 8 + depth 5) → one approve tx → batch 1/2 paid 3.0 ANT, batch 2/2 paid 0.75 ANT (distinct winner hashes) → FinalizeMerkleUploadMulticompleted, 291 chunks, downloaded byte-identical (cmp clean). DB: two transactions rows with the exact per-batch winner hashes/amounts; actual_cost = 3.75 ANT = sum.
  2. Failure drill: fresh wallet funded with 3.2 ANT (batch 1 needs 3.0) → batch 1 paid, batch 2 reverted (insufficient balance, definitive) → salvage finalize stored chunks_stored=256, chunks_failed=35, total_chunks=291 → upload failed/paid_unfinalized, temp source preserved, exactly one transaction row.
  3. Retry pays only the remainder: re-upload of the same content with a funded wallet dedups the 256 salvaged chunks and pays only the ~35-chunk remainder.

Unit: go test ./... green (batch planning/fallback/version-gap error, validation, pad/salvage decision, contract-formula payout bounds, classifyFailure rows for the new wrapping shapes).

Notes for review

  • merkleAllowancePayer is an optional interface seam so the uncommitted hosted-payment work (payer interface) merges without changes — payers lacking the method fall back to per-batch self-approval inside PayForMerkleTree.
  • Pre-existing, out of scope: RequeueStuck(60) headroom vs N×5-min confirmation waits on very large files — follow-up ticket suggested.

🤖 Generated with Claude Code

Nic-dorman and others added 2 commits August 24, 2026 10:22
Uploads larger than one merkle tree (256 fresh chunks ~= 1 GiB) arrive
from antd >= 0.12.0 as multiple MerkleBatches with the legacy mirror
fields empty, so the worker submitted an empty commitments list and the
upload failed opaquely. The merkle branch now:

- plans batches from MerkleBatches (legacy single-batch fields remain as
  the pre-0.12 fallback; both-empty errors with a clear version-gap
  message) and validates every batch before any money moves
- pre-approves the summed per-batch max payout in one approve tx
  (Signer.EnsureMerkleAllowance), then pays one payForMerkleTree tx per
  batch, recording each confirmed spend before the next batch (V2-426;
  one transactions row per batch, tx_hash = that batch's winner hash)
- finalizes via FinalizeMerkleUploadMulti with the index-aligned winner
  list (single-batch keeps the legacy FinalizeMerkleUpload call)
- on a definitive mid-plan payment failure, salvage-finalizes the paid
  batches ("" for unpaid) so their chunks store and a retry pays only
  the remainder, then preserves the source (errPaidNoRetry); a
  confirmation timeout skips salvage so the manual full finalize stays
  possible
- sums estimatedUploadCost across all batches for the max_gas_fee check

V2-1056

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found live on the multi-batch devnet e2e: batch 1's payForMerkleTree
reverted with ERC20InsufficientAllowance because the allowance bound
still used the V1-era "sum over pools of max candidate" formula.
PaymentVaultV2 charges median16(winner pool quotes) * 2^depth and picks
the winner pool at execution time (PaymentVaultV2.sol:140), so the
correct worst case is the highest pool median shifted by depth, summed
across batches. The single-batch path had the same latent under-approval
bug; both bounds now share the contract-accurate formula, and the worker
validates batch depth (1-8) before paying.

V2-1056

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@dirvine dirvine left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at exact head 85228779cc6701945fa57388636ff053e8334a2c.

The multi-batch flow is consistent with antd-go v0.12.0 and PaymentVaultV2: the allowance bound uses the upper median quote multiplied by 2^depth, payments are nonce-serialised, and the timeout/salvage branches avoid automatically re-paying an unconfirmed transaction. The single-batch compatibility path and multi-finalize ordering also match the SDK contract.

Verified locally:

  • go test ./... -count=1
  • go test -race ./... -count=1
  • go vet ./...
  • frontend npm ci && npm run build
  • git diff --check

Required exact-head CI is green (Docker, frontend, lint, smoke, PostgreSQL and SQLite).

Non-blocking follow-up: validateMerkleBatches does not yet fully enforce its “reject malformed data before any money moves” comment: it could also validate pool-hash syntax, the contract-required pool count, and non-negative uint256 candidate amounts. The later signer/contract checks plus partial-finalize path keep this from being a merge blocker, but validating those fields up front would make the stated invariant complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants