Skip to content

feat(download): harden same-URL dedupe (resolved-URL keys + ref-counted cancel) #455

Description

@rejifald

The v1 opt-in dedupe in @stitchapi/download (#451) has two documented limitations:

  1. Keys off item.url / explicit id, not the resolved request URL. Two { path: '/x' } items sharing a baseUrl are not deduped (their keys differ) — only items given a full url (or the same explicit id) collapse.
  2. Coarse cancel-under-dedupe. A follower reuses the leader's in-flight promise; cancelling a follower only detaches it (the shared fetch keeps running), and cancelling the leader fails the followers.

Proposal

  • Key dedupe off the resolved request target (baseUrl + path + query), so logically-identical items collapse regardless of spelling.
  • Ref-count the sharers of an in-flight fetch: cancelling a sharer decrements; the shared request aborts only when the last sharer cancels.

Also: dedupe collapses only items in-flight together today — decide whether a just-finished result is reusable within a short window (probably not, since blobs are uncacheable) and document it.

Introduced alongside #451; P18 in the rig spec.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions