v1 redesign — tracking issue

[thorwhalen]

Tracking issue for the v1 redesign. Design docs live in misc/docs/ — architecture.md is the entry point, decisions/ holds ten ADRs.

What v1 is

s3dol becomes a dol-based adapter for S3 and S3-compatible object storage (MinIO, R2, Scaleway, Hetzner, Backblaze, Wasabi, Ceph, Supabase, GCS-interop, Tigris, …), with AWS as the reference semantics. Same shape as azuredol, so one adapter in the family reads like the next.

Three layers: connection (credential + endpoint SSOT, lazy, picklable, redacting) → base (close-to-metal Collection→Reader→Store triads, owns the prefix, one error seam, ObjectHandle) → recipes (factories + codec stacks, by composition only).

Scope: core + large-object I/O. Deferred to v1.x, tracked separately: versions/tags/bucket-config/in-flight-uploads as Mappings, async, fsspec adapter, obstore engine.

Why now — what's actually broken

Verified against the current release:

explicit endpoint_url silently dropped when env credentials exist (base.py:82)	store aims at AWS instead of the configured provider
explicit credentials overridden by env (base.py:71)	wrong identity, silently
list(store) returns [] on any error (base.py:109)	wrong bucket, expired token, missing permission → empty list, no error
writing to a missing bucket creates it — even with make_bucket=False	a typo mints a bucket
url_for presigns with SigV2	#10 — rejected by modern S3 and every major alternative
stores are unpicklable	unusable with ProcessPoolExecutor or Dask
del store[k] on a missing key silently succeeds; del buckets[name] cascades unpaginated	partial, non-idempotent destruction past 1000 objects
anonymous access to public buckets is impossible	the entire open-data use case
Supabase support is a per-vendor subclass that hand-parses HTTP chunked framing out of object bodies	read-side workaround for a write-side client misconfiguration; corrupts payloads >1 chunk

Most of these share one root cause: credential/endpoint resolution is spread across three functions instead of living in one connection object.

Phases

• P0 — safety net first. Tier-1 (no-I/O) tests: error classification over synthesized ClientErrors, key-codec property tests, preset merging. Plus s3dol.diagnose() shipped in a 0.1.x patch so dependents can record what their environment currently resolves to before the resolution order changes. (ADR-0008, ADR-0007 §5)
• P1 — url_for produces SigV2 presigned URLs, which modern S3 and every major S3-compatible provider reject #10 (SigV2) as its own 0.1.x fix. Independent of everything else, strictly a fix.
• P2 — upstream dol fixes: Prefix relativization silently corrupts non-matching keys (boundary violation) dol#82 (prefix corruption), Key-transform wrappers delegate capability methods with the unmapped key (url_for etc. silently address the wrong object) dol#83 (delegation with unmapped key). Raise the dol floor.
• P3 — connection.py + presets.py: the credential/endpoint SSOT, the provider registry, capabilities. Fixes the first two rows above.
• P4 — errors.py: the (operation, code, status) translation seam. Fixes the third row. (discussion An extendable KeyError for s3dol and more #6)
• P5 — base.py: the triads, prefix in the leaf, ObjectHandle.
• P6 — values.py/writes.py/reads.py: large-object I/O. (Appendable s3dol #5)
• P7 — testing.py: in-memory fake + exported conformance suite, so downstream users can test without S3.
• P8 — compat + release: s3dol.store.S3Store deprecated shim; pin dependents; release notes lead with behaviour changes.

Compatibility

s3dol.store.S3Store keeps working with its current signature and is removed in v2. It is also the fix delivery mechanism — dependents get corrected endpoint/credential resolution without changing a line.

s3dol/store.py must survive as a module (dependents import the fully-qualified path), bucket_name stays accepted both positionally and by keyword, path= is not renamed, and s3dol/tests/util.py keeps its two functions (py2store imports them under suppress(ImportError), so breaking them fails silently).

Release ordering matters because merging auto-publishes to PyPI and versions burn permanently — see ADR-0007 §5. In particular, the endpoint/credential fix ships separately from the naming/API change so a dependent whose data target moves can bisect it.

Three landmines for anyone picking this up

1. dol's prefix machinery corrupts non-matching keys — the "obvious" refactor reproduces the bug it was meant to fix. dol#82.
2. A dol key-wrapper delegates methods with the outer key, so url_for/sub/handle/info/delete_many silently address the wrong object. This is why the prefix lives in the leaf. dol#83.
3. Never pass EncodingType to a list call — botocore sets it and decodes it, but only when it set it itself.

[thorwhalen]

Design docs merged in #13 — misc/docs/architecture.md plus ten ADRs.

Next actionable steps, in dependency order:

1. url_for produces SigV2 presigned URLs, which modern S3 and every major S3-compatible provider reject #10 (SigV2) — independent of everything else, strictly a fix, worth a 0.1.x patch.
2. s3dol.diagnose() in the same 0.1.x line, so dependents can record what their environment currently resolves to before the resolution order changes (ADR-0007 §5, step 0).
3. Prefix relativization silently corrupts non-matching keys (boundary violation) dol#82 and #83 — both block the implementation and both benefit every *dol adapter.
4. P0 tier-1 test scaffolding, then the module-by-module build.

Not started: any code.

[thorwhalen]

Design question 1 resolved — ADR-0011, merged in #17

Discussion #14 (layered transformations / the unmapped-key problem) is settled:
ADR-0011. Full reasoning in the
resolution comment.

Layer B carries no key-taking public methods. Keyed capabilities become sibling stores
you index — s3dol.handles(store)[k], .urls, .info — because __getitem__ is the one thing
dol maps correctly at every wrapper depth, so they are correct by construction with no
key-resolution primitive. Non-keyed operations (sub, prefixes, delete_many,
delete_bucket) are free functions. url_for survives as one guarded method for
dol.SupportsUrlFor.

Phase impact

• P5 (base.py) — the triads gain three sibling readers (BucketHandles, BucketUrls,
  BucketInfo) sharing one private base with BucketReader, plus a new capabilities.py. Layer
  B's method table loses url_for/info/handle/sub/prefixes/delete_many.
• P0 — add the reflective keyed-surface conformance test (ADR-0011 §D5). Cheap, and it is
  what keeps the decision from decaying. Note a first-argument-name heuristic is not enough:
  it misses delete(name, ...), delete_many(keys), batch(operations).
• P2 — the upstream list grew and is partly done:
  • fix(dig): resolution failures raise instead of returning None; export inner_most_key dol#84 — export inner_most_key; resolution failures raise instead of returning
    None. Green, awaiting review.
  • fix: two delegation bugs in dol's own code (content_url, filesys key validation) dol#85 — fix dol.content_url (it resolved with the outer key, which blocked the
    dol.content integration dol/content.py names s3dol for) and
    mk_relative_path_store key validation. Green, awaiting review.
  • dol#82 and dol#83 remain.
• P8 — endpoint.delete(name, force=True) becomes s3dol.delete_bucket(...); the compat
  shim absorbs the difference for legacy S3Store users.

Also worth knowing

Two design drafts were refuted by testing rather than argument — recorded in ADR-0011 §D1a so
nobody re-proposes them:

• inner_most_key(wrapped_self(self), k) — the escape ADR-0006 §2 prescribed — is silently
  wrong when nothing holds a reference to the wrapper, and because the prefix lives in the
  leaf the wrong answer is a plausible str. Reported upstream as wrap_kvs will wrap the instance but self of instance is not wrapped dol#18.
• Free functions are not categorically safe either: they break on a non-Store layer in the
  chain, where the method form is right.

The census behind all this is indexed from i2mint/dol#83 (13 packages verified, 13 issues filed,
12 survey claims refuted); design rationale for the family is in i2mint/dol#86.

Next: discussion #15 — credential and endpoint resolution. That is the last thing gating code.

[thorwhalen]

Both design questions are closed — implementation is unblocked

Discussion #15 (credential and endpoint resolution) is resolved:
ADR-0012, merged in #18.
Full reasoning in the resolution comment.
That was the last thing gating code.

Phase impact

• P0 — unchanged and now more important: diagnose() ships first, in a non-breaking
  0.1.x patch, and lacing/reelee/http_cosmo_prep run it in their real environments before
  anything about resolution changes. Its output is the input to two decisions that were
  deliberately left to telemetry. Capture the v0 golden table before any source change —
  git diff 0.1.9 HEAD -- s3dol/ is empty, so this is the last moment that evidence exists.
• P1 (url_for produces SigV2 presigned URLs, which modern S3 and every major S3-compatible provider reject #10, SigV2) — the fix is unchanged, the diagnosis was wrong twice. It is region-gated
  (12 v2-capable region strings), not universal, and not caused by a custom endpoint. The
  conformance test must cover both a v2-legacy region and a v4-only region, and must assert
  'X-Amz-Algorithm' in the URL — client.meta.config.signature_version reports 's3v4' while
  emitting SigV2.
• P2 (dol upstreams) — the two blocking items landed in dol 0.3.59 (fix(dig): resolution failures raise instead of returning None; export inner_most_key dol#84, #85).
  dol#82 and dol#83 remain, neither blocking.
• P3 (connection.py/presets.py) — this is what ADR-0012 specifies. Note the work got
  smaller: there is no precedence ladder to vendor, because botocore already ranks explicit
  above env and v0's bug is a dropped argument.
• P8 — one migration row for http_cosmo_prep covering three axes at once (endpoint
  honoured, credential identity moves, checksum may change), not three rows. reelee moves
  transitively through lacing, so that pair upgrades together.

Two traps worth carrying into implementation

• Never pass Config(ignore_configured_endpoint_urls=True). Measured: a no-op when an
  explicit endpoint is set, and it retargets an AWS_ENDPOINT_URL_S3 user to AWS when one is
  not. Two independent design drafts included it unconditionally.
• Re-enabling lacing/tests/test_artifact_store_s3.py (the cross-repo gate ADR-0008 wants):
  do not make it pass with anon=True. Under ADR-0012's amendment url_for returns None
  when anonymous, so the test would pass for the wrong reason — worse than the current exclusion.

Scope changes to note

anon='auto' is deferred to v1.x (#12), so v1 ships anon: bool. S3Connection accepts no
live boto3 client — checked, no dependent passes one — which makes "every s3dol store pickles"
an unconditional guarantee rather than a two-tier one. And botocore>=1.36 / s3transfer>=0.11.2
floors are now declared; both were unpinned while the design sets knobs that only exist from 1.36.

[thorwhalen]

Implementation session — 2026-08-11 (Layer A landed, v1 open and gated)

Landed: #19 — Layer A as an additive, non-breaking 0.1.x release (errors.py, presets.py, connection.py, _diagnose.py). This is ADR-0007 §5 step 0: s3dol.diagnose() is now on PyPI so every dependent can record what their environment resolves to before anything moves.

Open and deliberately held: #20 — the v1 surface (Layer B/C, sibling capability stores, ObjectHandle, free functions, the compat shim, s3dol.testing). Green, reviewed, gated on §5 step 2 — see the hold comment there.

Dependents pinned (§5 step 2): thorwhalen/lacing#26 ✅, thorwhalen/reelee#294 ✅, cosmograph-org/http_cosmo_prep#2 open (not ours to merge — and it is the one that both fails to declare the dependency and passes an explicit endpoint_url that v0 silently drops).

Design records that changed because implementation disagreed with them

[ADR-0011 §D3b-amended] — the guarded url_for shim cannot serve both of its call paths. dol.content_url hands over a key already mapped through the outer layers (its walk stops at the provider "because that layer applies its own"), while a direct delegated call hands over the raw outer key. Applying _id_of_key is right for the first and double-transforms the second (measured: p/x/x/g). It now applies dol's contract and refuses once the leaf has been wrapped, driven by a sticky probe flag that — unlike the wrapped_self weakref — survives the wrapper's death, so the refusal is reliable rather than intermittent.

This is the strongest available evidence for the family problem, and it is posted upstream as the P1 trial result: i2mint/dol#86 (comment). Summary: a keyed capability method is unfixable in the general case — not fragile, and not fixable by a better resolution primitive, because the ambiguity is in the call, not the resolution. dol's Option G engine makes the same method correct on all six shapes (including the temporary-wrapper and stacked-codec cases that defeated every prior option), because a flat boundary has one key domain and no chain for a caller to pre-resolve through.

Interpretive calls worth a second pair of eyes

• preset is a field on S3Connection (ADR-0012 D1's dataclass listing omits it, but D3 rung 2 and D4 require it to be nameable — treated as an oversight).
• soft rows report but do not pass their addressing style, reconciling "every row must state it" with "omit preserves the profile channel".
• The C1 unbound-template case raises MissingPresetParam (a subclass of MissingEndpoint, so either except works) rather than leaving a never-raised class — ADR-0004's own indictment of v0 was five never-raised exceptions.
• The module is _diagnose.py, not diagnose.py: a public submodule of that name shadows the s3dol.diagnose function after first import.

Adversarial review

A 4-lens refute panel produced 39 findings; every reproducible one is fixed with a regression test. The two most serious were both in the migration instrument itself: diagnose() echoed v1 credential kwargs verbatim (a denylist of two v0 names let every v1 shape through — now an allowlist), and the divergence table simultaneously cried wolf for the most common configuration and missed the real credential-identity move (both provenance labels truncate to AKIA… — now a pure identity predicate). Full list in #20's body.

Next

1. Land cosmograph-org/http_cosmo_prep#2, then run diagnose() in each dependent's real environment (§5 step 0b — needs a human).
2. Merge s3dol v1: Layer B/C, sibling capability stores, compat shim, testing (ADRs 0001–0012) #20. Release notes lead with the behaviour changes.
3. Then the v1.x queue in v1.x — deferred collections interfaces (uploads, versions, tags, bucket config, annotations, fsspec) #12, and the lacing cross-repo test gate (ADR-0008).

[thorwhalen]

Verified done: v1.0.0 shipped (commit 794d534, "s3dol v1 — Layer B/C, sibling capability stores, compat shim, testing"). All phases in this tracking issue's checklist are present in current master:

• P0/diagnose: s3dol/_diagnose.py::diagnose
• P3: s3dol/connection.py, s3dol/presets.py
• P4: s3dol/errors.py
• P5: s3dol/base.py
• P6: s3dol/values.py, s3dol/writes.py, s3dol/reads.py
• P7: s3dol/testing.py
• P8: s3dol/store.py is now an explicit "COMPAT SHIM — the v0 S3Store, forwarded onto the v1 machinery"

Deferred-scope items are correctly tracked separately in #12. Closing.