Skip to content

fix: keep inventory-limited pools in water-fill splits via get_limits - #405

Closed
kayibal wants to merge 3 commits into
mainfrom
ah/waterfill-limit-capped-ranking
Closed

fix: keep inventory-limited pools in water-fill splits via get_limits#405
kayibal wants to merge 3 commits into
mainfrom
ah/waterfill-limit-capped-ranking

Conversation

@kayibal

@kayibal kayibal commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Problem

An inventory-limited pool — e.g. a propAMM whose output-side balance is below the order — lost its entire allocation the moment the order exceeded its inventory, instead of contributing up to its capacity in a split. Measured live (USDC→WETH, 1-hop water_fill pool, fermiswap): 93% route share at a $300k order, 0% from $310k upward, cliff moving with the pool's per-block inventory.

Cause

Three compounding defects, found by bisecting live quotes and instrumenting the chunk allocation:

  1. Ranking: a path failing simulation at the full order amount was ranked with gross 0, sorting it below every split candidate window (top-4 disjoint, top-8 full-amount / top-32 marginal fill-and-spill probes).
  2. Window selection: even ranked by its real capped output, an inventory-limited path sinks below every order-absorbing pool once the order is a small multiple of its inventory, and falls out of the marginal probe window again.
  3. Allocation: propAMM balance overrides reject only a single swap larger than the balance — EVMPoolState::get_amount_out re-derives token-balance overwrites on every call, so per-chunk simulations never see cumulative depletion. The allocator over-filled the pool (observed: $707k of an $800k order into a pool with $280k inventory, zero chunk failures), the rebuilt single leg reverted, and the whole split candidate was discarded.

Fix

  1. Rank a path that fails at the full amount by its output at the get_limits-capped input (per hop, back-converted to path-input units through the preceding edges' spot prices; one retry at 99% for marginally stale limits; unexplained failures keep gross 0). get_limits was verified against live states: 0 errors across 1,868 pool directions on 15 protocols; get_amount_out succeeds at exactly max_in on all but one broken pool.
  2. Always include the best limit-capped paths in the fill-and-spill marginal probe (CAPPED_PROBE_PATHS), where the first-chunk probe judges them by price.
  3. Bound each path's chunk allocation at its get_limits cap in both chunk loops (disjoint_waterfill, fillspill_waterfill).

Adds get_limits_guarded to GuardedProtocolSim (same panic containment as get_amount_out_guarded).

Extra get_limits/simulation calls run only for paths that fail at the full amount, plus the small allocation subsets. No bisection loops.

Result (live, USDC→WETH, fermiswap inventory ~$278k)

Order fermi share fermi absolute
$100k–$300k 70–86% grows with order
$400k 69.5% $278k
$800k 34.8% $278k
$2M 13.7% $273k

The pool's absolute allocation pins at its inventory from $400k to $2M; the remainder spills to on-chain pools.

Tests

  • test_inventory_limited_component_kept_in_split — 38 healthy competitors push a full-amount-failing path below every candidate window; asserts it still receives a substantial share.
  • test_per_swap_limited_component_capped_at_its_limit — a pool that accepts every chunk but rejects the rebuilt leg (MockProtocolSim has exactly the propAMM per-swap semantics); asserts allocation respects the sell limit.
  • test_max_fillable_input_single_hop_uses_exact_limit, test_max_fillable_input_two_hop_converts_via_spot — cap computation.

Each test was verified to fail with its mechanism reverted. Full workspace suite passes (one pre-existing macOS-local issue: panic-containment tests abort under rustc 1.95/macOS 26 because unwinding aborts; they pass on Linux).

An upstream issue about the non-depleting balance overwrites in EVMPoolState::get_amount_out will be filed on propeller-heads/tycho; if fixed there, mechanism 3 here becomes an optimization rather than a correctness requirement.

🤖 Generated with Claude Code

An inventory-limited pool — e.g. a propAMM whose output balance is
below the order — lost its entire allocation to worse-priced pools the
moment the order exceeded its inventory, through three compounding
defects:

1. A path failing simulation at the full order amount was ranked with
   gross 0, sorting it below every split candidate window (top-4
   disjoint, top-8 full-amount / top-32 marginal fill-and-spill
   probes). Rank such paths by their output at the get_limits-capped
   input instead: each hop's sell limit, back-converted to path-input
   units through the preceding edges' spot prices (pessimistic under
   AMM concavity, so the capped simulation succeeds). One retry at 99%
   covers marginally stale limits; a failure the limits don't explain
   keeps gross 0.

2. Even ranked honestly, a capped path's output sinks below every
   order-absorbing pool once the order is a small multiple of its
   inventory, dropping out of the probe window again. The fill-and-
   spill marginal probe now always includes the best limit-capped
   paths (CAPPED_PROBE_PATHS) wherever they rank; the first-chunk
   probe then judges them by price.

3. Chunked allocation over-filled such pools: propAMM balance
   overrides reject only a single swap larger than the balance, so
   per-chunk simulations all succeed past the real capacity, and the
   rebuilt single leg at the summed amount reverts — discarding the
   whole split candidate. Both chunk loops now stop allocating to a
   path at its get_limits cap.

The extra get_limits/simulation calls run only for paths that fail at
the full amount plus the small allocation subsets. get_limits was
verified against live states: 0 errors across 1,868 pool directions on
15 protocols, and get_amount_out succeeds at exactly max_in on all but
one broken pool. Live result (USDC->WETH, 1-hop water_fill, fermiswap
inventory ~\$278k): the pool's absolute allocation now pins at its
inventory from \$400k to \$2M order size instead of dropping to zero,
with the remainder spilling to on-chain pools.

Adds get_limits_guarded to GuardedProtocolSim for panic containment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The capped ranking pass simulated every full-amount-failing path at its
limit cap with uncached per-hop get_limits calls. At three hops most of
the ~4.5k candidate paths fail once the order is large, which added
1-2s to setup on large orders. Now sell limits are memoized per solve
(LimitsCache), failing paths are deduplicated by bottleneck component
and ranked by spot price, and only the top CAPPED_PROBE_PATHS get the
honest capped simulation; the rest keep gross 0 as before the capped
ranking existed.

The allocation passes replace per-path caps with a shared per-component
budget (ComponentBudget) fed by exact hop inputs from simulate_step:
fill-and-spill paths share components, and their combined allocation
must stay a valid single swap or the rebuilt merged leg reverts and the
whole candidate is discarded. The gated coarse pass also anchors the
top-ranked candidate in the active set, so inventory-limited probe
winners cannot claim every max_paths slot, saturate together, and leave
the order unfillable.

Regression test: 20 two-hop paths bottlenecked by one limited pool must
not crowd a distinct best-price limited pool out of the split
(test_capped_probe_slots_deduped_by_bottleneck, fails on the previous
commit).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@kayibal

kayibal commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up commit 658bc43 after benchmarking against main (300 real aggregator trades x2 seeds + an A/A control, water_fill max_hops=3 / timeout 5000, all_onchain + vm:fermiswap, min_tvl 100):

Problem found: the capped ranking pass simulated every full-amount-failing path at its limit cap with uncached per-hop get_limits calls. At 3 hops most of the ~4.5k candidate paths fail once the order is large, adding 1–2 s to setup ($600k–2M orders: 2.2–4.2 s vs baseline 0.7–2 s).

Fixes in this commit:

  • LimitsCache: get_limits memoized per solve — one call per (component, direction) instead of per path per hop.
  • Failing paths are deduplicated by bottleneck component, ranked by spot price, and only the top CAPPED_PROBE_PATHS get the honest capped simulation; the rest keep gross 0.
  • ComponentBudget: the chunk loops replace per-path caps with a shared per-component budget fed by exact hop inputs — fill-and-spill paths share components, and their combined allocation must stay a valid single swap or the rebuilt merged leg reverts and the whole candidate is discarded (this, not ranking, turned out to be the mechanism dropping fermiswap at ~2× inventory).
  • The gated coarse pass anchors the top-ranked candidate in the active set so inventory-limited probe winners cannot claim every max_paths slot and leave the order unfillable (the deficit was ratio-scaled onto over-committed legs at rebuild, then discarded).

After (live, paired same-block probes): setup 0.64–0.71 s at $600k–2M (capped_paths 6 instead of saturating 16), solve times at baseline parity (694 vs 692 ms at $600k, 744 vs 713 at $1M, 830 vs 780 at $2M). Quality on the 300-trade head-to-heads was tie-dominated with the avg diff inside the A/A noise floor.

New regression test test_capped_probe_slots_deduped_by_bottleneck (20 same-bottleneck two-hop paths must not crowd a distinct best-price limited pool out of the split); fails on the parent commit.

One live check is still pending: fermi retention at $600k–2M with the anchor change was validated in the unit scenario but not yet live — Tycho beta RPC and the node endpoint started refusing new connections mid-validation. Will re-run the paired probe when the endpoints recover.

🤖 Generated with Claude Code

Live validation showed the capped-probe slots being crowded out on
blocks with many failing paths: broken pools advertise fantasy spot
prices (multiples of the market rate) that outrank honest inventory,
and VM components can lack derived edge data entirely, which excluded
their paths from selection. Probes are now deduplicated keeping the
largest cap per bottleneck and ranked by input cap — the path's real
fillable notional from the same get_limits oracle the allocation
budgets trust, comparable across paths because every cap is in
path-input token units. A failed capped simulation no longer consumes
one of the CAPPED_PROBE_PATHS slots; attempts continue down the ranked
list, bounded by CAPPED_PROBE_ATTEMPTS.

Adds debug-level logs for capped-probe selection, fill-and-spill
candidate selection, and per-candidate nets with legs.

Live result (paired same-block probes, USDC->WETH, water_fill 3-hop):
fermiswap retained at ~its inventory in 8/9 probes across $600k-$2M
orders (previously 0/9 at these sizes), +0.9 to +6.9 bps over main.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@kayibal

kayibal commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Live validation is complete (the earlier network issue was Little Snitch blocking the rebuilt binaries). Two more defects surfaced and are fixed in 3af5651:

  1. Fantasy-spot crowding. The capped-probe slots were ranked by path spot price. On blocks with many failing paths (~1,400 at $600k), broken pools advertising spot prices at 2–5× the market rate claimed all 16 slots — their capped simulations then failed anyway, wasting the slots while honest inventory (fermiswap) was excluded. VM components lacking derived edge data were also silently skipped.
  2. Probes are now deduplicated by bottleneck keeping the largest cap, and ranked by input cap — the path's real fillable notional from the same get_limits oracle the allocation budgets trust (unit-comparable since every cap is in path-input tokens). A failed capped simulation no longer consumes a slot; attempts continue down the ranked list, bounded by CAPPED_PROBE_ATTEMPTS = 48.

Final live result (paired same-block probes vs main, USDC→WETH, 3-hop water_fill): fermiswap retained at ~its inventory in 8/9 probes across $600k–$2M orders — previously 0/9 at these sizes on this branch and structurally impossible on main — winning +0.9 to +6.9 bps. Solve times ~1–1.7 s on fermi-heavy solves vs 0.7–1.3 s baseline, versus 2.2–4.2 s before the perf commit.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants