Proposal
The trustless-gateway broker fetches one ?format=raw request per block. For a many-small-block DAG (a sharded HAMT directory is hundreds of tiny blocks) that is hundreds of round trips. The graph walker's lookahead overlaps them, but the per-request latency still dominates.
A session that issues one ?format=car&dag-scope=all request per root and serves the walker's per-block retrieve() calls from the streamed CAR collapses that to a single request. Blocks stay hash-verified through the session's validateFn, and any block the CAR omits or delivers corrupt falls back to a single ?format=raw fetch.
Impact
One streaming request instead of N for any CAR-endpoint-backed retrieval: @helia/verified-fetch CAR responses and @helia/car export over a trustless gateway. In a local bench (249-block sharded DAG, 25ms/request) the single-CAR path ran ~24x faster than per-block, with byte-identical output.
The design question
NetworkedStorage.createSession(root, options) (@helia/utils 2.5.2) passes only options to broker.createSession(options), so the broker session never receives the root. A CAR-stream session needs it to open the stream. Options, in order of preference:
- Pass
root to broker.createSession(root, options). Small signature change, makes root-scoped streaming brokers first-class.
- Infer the root from the first
retrieve(cid) (the walker's first get is the root). Works today, but it is implicit coupling.
- Leave it, and I keep this in my own project.
It would not subclass AbstractSession (that is per-block, per-provider, racing); it implements SessionBlockBroker directly with a no-op addPeer. Want to confirm that shape is acceptable before I open a PR.
Working implementation (as a BlockSource in my project, same pump/index/verified-gap-fill logic): https://github.com/SgtPooki/ipfs2foc/blob/main/packages/core/src/car-stream-source.ts . Happy to share the bench too.
Proposal
The trustless-gateway broker fetches one
?format=rawrequest per block. For a many-small-block DAG (a sharded HAMT directory is hundreds of tiny blocks) that is hundreds of round trips. The graph walker's lookahead overlaps them, but the per-request latency still dominates.A session that issues one
?format=car&dag-scope=allrequest per root and serves the walker's per-blockretrieve()calls from the streamed CAR collapses that to a single request. Blocks stay hash-verified through the session'svalidateFn, and any block the CAR omits or delivers corrupt falls back to a single?format=rawfetch.Impact
One streaming request instead of N for any CAR-endpoint-backed retrieval:
@helia/verified-fetchCAR responses and@helia/carexport over a trustless gateway. In a local bench (249-block sharded DAG, 25ms/request) the single-CAR path ran ~24x faster than per-block, with byte-identical output.The design question
NetworkedStorage.createSession(root, options)(@helia/utils2.5.2) passes onlyoptionstobroker.createSession(options), so the broker session never receives the root. A CAR-stream session needs it to open the stream. Options, in order of preference:roottobroker.createSession(root, options). Small signature change, makes root-scoped streaming brokers first-class.retrieve(cid)(the walker's first get is the root). Works today, but it is implicit coupling.It would not subclass
AbstractSession(that is per-block, per-provider, racing); it implementsSessionBlockBrokerdirectly with a no-opaddPeer. Want to confirm that shape is acceptable before I open a PR.Working implementation (as a
BlockSourcein my project, same pump/index/verified-gap-fill logic): https://github.com/SgtPooki/ipfs2foc/blob/main/packages/core/src/car-stream-source.ts . Happy to share the bench too.