Skip to content

fix(rpc)!: bound getaddressutxos by the requested height range and limit - #11317

Merged
mergify[bot] merged 2 commits into
ZcashFoundation:mainfrom
LarryRuane:fix/getaddressutxos-pushdown
Aug 25, 2026
Merged

fix(rpc)!: bound getaddressutxos by the requested height range and limit#11317
mergify[bot] merged 2 commits into
ZcashFoundation:mainfrom
LarryRuane:fix/getaddressutxos-pushdown

Conversation

@LarryRuane

@LarryRuane LarryRuane commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Motivation

Zebra serves GetAddressUtxos over its built-in CompactTxStreamer gRPC server, added in #10953. That method takes a startHeight and a maxEntries, and it honors both, but only by filtering the finished reply: the node reads every unspent output the named addresses hold, and then discards whatever the caller did not ask for. The answers are correct, but the work is not bounded by the request. A caller asking for ten entries from an address holding thousands pays for all of them.

The getaddressutxos JSON-RPC underneath has no such arguments, which is why the gRPC layer had to filter afterwards, and why no other caller can ask for less work at all. Adding the two arguments there lets the bounds reach the index scan, which is the only place they can actually save anything.

The same shape appears one layer further out. lightwalletd, a server that serves Zcash chain data to light wallets, offers its clients the same method and answers it with this RPC; as of zcash/lightwalletd#592 it forwards both arguments, and a node without this change silently drops them as unknown JSON keys.

Bounding the scan is worth doing on its own, but it is also a fix: this is the unremediated part of GHSA-x4m7-3gpp-xc36, and item 1 of #11239. The worst shape is a single high-UTXO address with a start height at the chain tip, which is a tiny request and an all-but-empty reply on top of a full index scan, leaving no reply bandwidth with which to throttle the caller.

Measured on the built-in gRPC server against a synced mainnet node, on t1cArVf7BgN3zPx6UHGy87ZAurTA4AQejj7, holding ~6,704 UTXOs:

query before after
unbounded 220ms, 6,705 entries 182ms, 6,704 entries
maxEntries=10 106ms, 10 entries 17ms, 10 entries
startHeight=3450000 105ms, 8 entries 18ms, 7 entries

The address is in active use, and the "after" column was measured first, so one output arrived at it between the two runs. That single output is the whole of the difference between 6,704 and 6,705, and between 7 entries and 8: it landed above height 3,450,000, so it is counted in both rows. Nothing is dropped by the change.

The entries returned are otherwise the same before and after; only the work behind them changes. The improvement is understated here, because a trivial GetLightdInfo call through the same grpcurl harness costs 15ms, so the 17ms row is about 2ms of server time against roughly 91ms before.

Solution

ReadRequest::UtxosByAddresses becomes a struct variant carrying a height_range and an optional max_entries, and both are pushed down to the finalized index scan. The RocksDB address index is keyed by (AddressLocation, OutputLocation), so a start height is a seek and a limit is a take.

Two parts are subtle and are what the new tests cover.

The non-finalized chain can spend UTXOs that the finalized query returned, so a limited finalized scan over-fetches by the number of UTXOs the chain spends for those addresses. Without that, a full page of results can come back short, and a client paging by count reads a short page as the end of the address' UTXOs.

Truncation happens after the finalized and non-finalized sets are merged, not while they are chained. Chaining two sorted maps is not globally sorted, and entries in the overlap window would otherwise be counted twice.

A start height above the chain tip is clamped rather than rejected, because a start height above the chain already selects nothing, which is what the request means.

Tests

Unit tests in zebra-state cover the height-range filter on non-finalized UTXOs, that a limited result keeps the lowest entries in chain order, and that over-fetching keeps a limited page full when the chain spends an entry the finalized query returned. zebra-rpc gains rpc_getaddressutxos_limits, covering each argument alone, both together, and clamping at and above u32::MAX.

Mutation testing drove the shape of those tests: truncating inside the merge, dropping the over-fetch, and an off-by-one in the limit all initially survived. Two mutants still escape and are called out under Follow-up Work.

Manually tested on both interfaces, on mainnet and on regtest. Every comparison is against main built from source with the same release profile, takes the median after three warm-up passes, and reports entry counts so any drift is visible. JSON-RPC rows use curl over 15 runs; gRPC rows use grpcurl over 11, and carry that client's ~15ms startup floor.

JSON-RPC getaddressutxos, mainnet, same address as above, 6,704 entries in both runs. Here the returned entries change, because the arguments did not previously exist:

query before after
unbounded 120ms, 6,704 entries 94ms, 6,704 entries
maxEntries=10 121ms, 6,704 entries 10ms, 10 entries
maxEntries=100 120ms, 6,704 entries 11ms, 100 entries
startHeight=3450000 121ms, 6,704 entries 10ms, 7 entries

Results were verified, not just timed: maxEntries=N returns exactly the first N entries of the unbounded result, startHeight=H returns exactly the unbounded result filtered to height >= H, and results stay ascending by height.

Regtest, reproducible. A regtest node mines to a fixed transparent address, so generate 10000 accrues exactly one coinbase UTXO per block to tmJymvcUCn1ctbghvTJpXBwHiMEB8P6wxNV, and nothing ever spends them:

[network]
network = "Regtest"
[rpc]
lightwalletd_listen_addr = "127.0.0.1:18067"
[mining]
miner_address = "tmJymvcUCn1ctbghvTJpXBwHiMEB8P6wxNV"
query JSON-RPC before JSON-RPC after gRPC before gRPC after
unbounded 87ms, 10,000 73ms, 10,000 159ms, 10,000 146ms, 10,000
maxEntries=10 87ms, 10,000 10ms, 10 58ms, 10 15ms, 10
startHeight=9000 87ms, 10,000 14ms, 1,001 66ms, 1,001 26ms, 1,001

The unbounded rows improve because the scan replaces a per-entry zs_next_key_value_from seek with a single bounded range iterator. That gain is smaller on regtest than on mainnet, because a 41MB database has a shallow, fully cached LSM tree, which is the case most favorable to the old per-entry seek.

Also tested end to end through lightwalletd. Against a released 6.3.0 node every query returned the address' full UTXO set, confirming a node without this change ignores the fields rather than erroring; against this branch the same requests came back bounded.

Specifications & References

Follow-up Work

Two mutants survive, both needing fixtures this PR does not add: the finalized_scan_limit call site needs a populated non-finalized chain, and the range end bound is never exercised because the RPC always passes Height::MAX.

A backend must implement both arguments or neither. Honoring maxEntries while ignoring startHeight would truncate before the range filter runs and silently return fewer entries than exist, which a caller has no way to detect.

Paging by startHeight cannot express a resume point when one height holds more entries than the page size. That is a protocol-level gap, tracked in zcash/lightwallet-protocol#33, and is independent of this change.

AI Disclosure

  • AI tools were used: Claude Code, for the implementation, tests, mutation testing, and the benchmarking described above.

PR Checklist

`ReadRequest::UtxosByAddresses` becomes a struct variant carrying a
`height_range` and an optional `max_entries`, and both are pushed down to the
finalized address index scan. The index is keyed by `(AddressLocation,
OutputLocation)`, so a start height is a seek and a limit is a `take`.

Two parts are easy to get wrong, and the new tests cover them.

The non-finalized chain can spend UTXOs that the finalized query returned, so a
limited finalized scan over-fetches by the number of UTXOs the chain spends for
those addresses. Without that, a full page of results can come back short, and
a client paging by count reads a short page as the end of the address' UTXOs.

Truncation happens after the finalized and non-finalized sets are merged, not
while they are chained. Chaining two sorted maps is not globally sorted, and
entries in the overlap window would otherwise be counted twice.

No caller sets either bound yet: `get_address_utxos` passes the full height
range and no limit, so this commit changes no behavior.
@LarryRuane
LarryRuane force-pushed the fix/getaddressutxos-pushdown branch from 4e72794 to 4229b64 Compare August 21, 2026 06:26
The `GetAddressUtxos` gRPC method accepted both arguments and applied them,
but only to the finished reply: the node read every UTXO held by the named
addresses first, then discarded whatever was not asked for. The
`getaddressutxos` JSON-RPC underneath had no such arguments at all, so no
caller could ask for less work.

Pass both bounds into the state query, and drop the reply-side filter in the
gRPC method. The cost is now set by what the caller asked for rather than by
the size of the addresses' UTXO sets. That is the unremediated part of
GHSA-x4m7-3gpp-xc36, and item 1 of ZcashFoundation#11239.

A start height above the chain tip is clamped rather than rejected, because a
start height above the chain already selects nothing, which is what the
request means.

Because the JSON-RPC previously ignored both fields, a client that already
sends them now receives fewer entries than before for the same request.
@LarryRuane
LarryRuane force-pushed the fix/getaddressutxos-pushdown branch from 4229b64 to b772175 Compare August 21, 2026 15:25
@conradoplg
conradoplg self-requested a review August 24, 2026 21:47
@mpguerra

Copy link
Copy Markdown
Contributor

Thank you for this PR, we'll review shortly

@mergify mergify Bot added the queued label Aug 25, 2026
@mergify

mergify Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Merge Queue Status

  • Entered queue2026-08-25 17:23 UTC · Rule: batched · triggered by rule move to any queue if GitHub Rulesets are satisfied
  • Checks passed · on draft merge queue: checking #11317 on main (5687be2) #11328
  • Merged2026-08-25 18:00 UTC · at b7721750c69d0bc16abe087ac0199d7e6f4dcb76 · merge

This pull request spent 36 minutes 49 seconds in the queue, including 35 minutes 40 seconds running CI.

Required conditions to merge

@mergify
mergify Bot merged commit b2cc75b into ZcashFoundation:main Aug 25, 2026
137 of 139 checks passed
@mergify mergify Bot removed the queued label Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants