Robustly wrapping stores that have key- and value-taking methods

[thorwhalen]

Design rationale, consolidated so the next person does not re-derive it. Not a work item —
the work items are #83, #82, #18, #10, #6, #5, and PRs #84 / #85.

dol's core promise is that you can wrap a store with layers of transformations and keep using
it as a Mapping. That promise holds for the Mapping protocol and fails for everything else:
a wrapped store's non-dunder methods receive the outer, unmapped key, and its methods'
internal self[k] bypasses the transforms. Those are the two halves of one root cause, and a
13-package census says the family has quietly been living with both.

This writes down what we know, what we tried, and what each option actually costs — including
three options that look right and are not.

---

1. The mechanism, precisely

dol wraps by delegation (has-a): wrap_kvs(SomeClass) returns a Store subclass holding
an instance of SomeClass in self.store. The transform hooks
(_id_of_key / _key_of_id / _data_of_obj / _obj_of_data) live on the wrapper. The inner
instance never sees them.

Two consequences, in opposite directions:

	direction	issue
A method receives an unmapped key	outside → in	#83
A method's self[k] bypasses the transforms	inside → out	#18

And two delegation routes, which matters because a fix covering one is a silent no-op on the
other:

route	when	site
Store.__getattr__	instance-wraps, and mk_relative_path_store subclasses	dol/base.py:742
DelegatedAttribute.__get__	class-wraps; delegate_to installs one descriptor per attr of dir(wrapped)	dol/base.py:279, installed dol/base.py:416-480

Both return the method bound to the leaf.

Nothing raises. Capability detection cannot see it either — a @runtime_checkable Protocol
checks method presence, and since 3.12 isinstance uses getattr_static, so
isinstance(w, SupportsUrlFor) is True for a class-wrap and False for an instance-wrap. In
neither case does it say anything about whether the key is right.

---

2. Scale — the census

13 sibling packages surveyed, every claim re-verified by source read plus a runnable repro.
Index and per-package issues in #83.

The defect is overwhelmingly latent: it bites when a user applies a key codec, and most of
these packages never wrap their own stores. 12 survey claims were refuted outright. Being
accurate about this matters — an earlier draft of the s3dol write-up said "already destroying
data in production", and the evidence does not support that.

What it does support:

• focal — confirmed-live throughout, inherited from dol.filesys.Files.
• dol itself — Files(d).is_valid_key(k) is False for a key the store yields, breaking
  all(s.is_valid_key(k) for k in s). Fixed in fix: two delegation bugs in dol's own code (content_url, filesys key validation) #85.
• cosmodol.CosmosItems.replace — under a key codec, silently overwrites a different,
  real document in full.
• pydrivedol.GDReader.get_url — returns a URL for the wrong file and grants
  anyone/reader permission on it.
• xdol, unbox, lexis — the wrap_kvs will wrap the instance but self of instance is not wrapped #18 direction, already migrated to wrapped_self.

The shape that stays clean is instructive: azuredol's ContainerStore has essentially no
key-taking public methods. Its rich per-object surface lives on BlobHandle, which binds its
blob at construction, so a key codec over the store cannot corrupt it. It is not safe because of
where its prefix lives; it is safe because it has almost no seam to get wrong.

---

3. The option space

A — document it, use wrapped_self

Ships today, and it is what #18's design doc recommends for Phase 1.

But wrapped_self has its own silent failure mode. A delegated bound method holds no
reference to the wrapper, so when nothing else does, the wrapper is freed before the body runs
and _register_wrapper_backref's cleanup removes the registry entry — indistinguishable
from "never wrapped":

s = KeyCodecs.prefixed('x/')(BR('logs/'))
s.m_abs('b.txt')                                    # 'logs/x/b.txt'  correct
KeyCodecs.prefixed('x/')(BR('logs/')).m_abs('b.txt')
                                                    # 'logs/b.txt'    WRONG, silently

Where the leaf owns a prefix, the wrong answer is a plausible str, so a type check cannot
catch it. The predicate is "no live strong reference", not "temporary" — a temporary in a
reference cycle silently starts working, so the failure is intermittent. Reproduced on
CPython 3.10–3.14. Detail in #18.

Verdict: a real guardrail, and better than nothing — but not a correctness mechanism, and it
should stop being described as one.

B — declarative key-method registration

_key_methods = {'url_for': 0, 'delete_many': ('iter', 0)}, with the wrapper generating mapped
delegates.

dol already has this: wrap_kvs(ingoing_key_methods=…, outcoming_key_methods=…)
(dol/trans.py). It is untested (the TODO says so) and verified broken for leaf-defined
methods on both wrap paths — getattr(store_cls, name) at decoration time finds a
DelegatedAttribute or nothing, and the generated body calls super(store_cls, self).<name>(),
which does not consult Store.__getattr__. It fails loudly, at least. Building on it means
replacing it.

The deeper problem is that a method the author forgets to declare fails the same silent way. The
part that actually holds is not the registry but a reflective test that enumerates public
methods and fails on any undeclared one.

Verdict: the guard is worth more than the registry.

C — free functions

url_for(store, k) instead of store.url_for(k), resolving through the chain with
inner_most_key.

This is dol's own idiom (content_url, get_content, put_content, add_content), and it
sidesteps A's lifetime problem because the caller holds the store.

But it is not categorically safe. inner_most_key walks .store applying each layer's
_id_of_key, which breaks when a layer is not a dol Store:

chain	wire key	free function
any dol-shipped wrapper	—	correct
hand-rolled __getattr__-passthrough delegator	logs/b.txt	logs/logs/b.txt
key codec over a plain .store-holding middle layer	logs/x/b.txt	x/b.txt

A passthrough __getattr__ resolves _id_of_key to the leaf's bound method, so the walk
applies it twice; a middle layer without _id_of_key truncates the walk. Every dol-shipped
wrapper is safe — but dol/base.py:514-518 ships a documented hand-rolled Delegator recipe of
exactly the breaking shape.

Note also that dol's own instance of C, content_url, had the bug until #85: it did a flat
getattr(store, 'url_for')(key).

Verdict: more reliable than A, not reliable. Good for operations that are not keyed.

D — rebind delegated methods to the wrapper

Rejected, with running-code evidence, in misc/docs/dol_issue18_design.md §4. The fatal
defect is not blast radius: rebinding binds self to the innermost Wrap, so under a
Pipe stack it does not fix the case it exists to fix — 4 of the 6 sites that survey found
are Pipe stacks — and stacked-codec writes gain a partial-transform corruption surface. Plus
statically-undetectable crashes: a leaf method calling super().__getitem__(k) compiles
super(SomeClass, self) with self now a Wrap → TypeError.

Its Phase-1 instructions say: do not touch DelegatedAttribute.__get__, the delegate_to copy
loop, or the signature graft.

Verdict: do not re-propose without new information.

E — capabilities as parallel Mappings

Two designs get conflated under this name, and only one works.

As a Mapping-valued attribute (store.urls) — broken. A wrapper does not re-wrap such an
attribute:

KeyCodecs.prefixed('a/')(Leaf)({'a/b': 1}).urls    # {'a/b': ...}  INNER keys

The outer store's keys are ['b'], so store.urls[k] KeyErrors for every key it has. This is
#10's territory. (There is a precedent for the propagation half: Store.__init__
(dol/base.py:720-727) copies KeysView/ValuesView/ItemsView up from the leaf and binds
them to the outer store. Hard-coded to three names, but it is propagation plus outer
binding, which is simpler than re-wrapping.)

As a sibling store — correct by construction. Expose the capability as its own KvReader
over the same key space:

class BucketHandles(KvReader):
    def _id_of_key(self, k): return self.prefix + k
    def __getitem__(self, k): return ObjectHandle(self.bucket, self._id_of_key(k))

__getitem__ is the one thing dol maps correctly at every depth, so this needs no
inner_most_key, no wrapped_self, no guard. Verified correct unreferenced, under Pipe,
under cached_keys, under a value codec, and under the non-Store passthrough layer where C
is silently wrong.

Its cost is real: the user must wrap the sibling in parallel with the data store, because
deriving the codec chain onto a sibling is exactly #10.

Verdict: the best answer available today. s3dol adopted it
(ADR-0011).

F — is-a wrapping

Make wrap_kvs(Class) return a real subclass of Class. Then self inside a leaf method is
the wrapper, the transform hooks are on the MRO, and both #83 and #18 disappear — along with #6,
since the __signature__ graft exists only because Wrap is a generic shell rather than a real
subclass.

misc/docs/dol_issue18_design.md recommends this as the terminal direction for 0.4/1.0, with a
staged plan (opt-in mode='isa' first, default flip gated on a green dependents run). It is
x-large, coupled to #5, and its §9 still lists "commit to C as the terminal direction now?" as
an open question for the maintainer.

Verdict: the actual fix. Everything above is what you do until it lands.

---

4. What this implies for adapter authors today

The selection criterion that turned out to matter is not "which is the permanent answer" but
"which is correct on 0.3.x AND harmlessly redundant once is-a lands".

1. Have as few key-taking methods as possible — ideally none. This is the single biggest
   lever, and it is what makes azuredol robust. A capability that binds its key at
   construction (a handle) cannot be corrupted by a key codec at all.
2. Route keyed capabilities through __getitem__ — sibling stores (E-as-store). Correct by
   construction, no primitive.
3. Free functions for the non-keyed rest (C), knowing the non-Store-layer caveat.
4. wrapped_self only as a guardrail (A), never as a correctness argument.
5. Guard it with a reflective test (B's useful half) that fails on any new public method
   taking a key. Note a name heuristic is not enough: it misses delete(name, ...),
   delete_many(keys), batch(operations), sync_to(target) — several of the worst shapes.

---

5. Open questions

1. Is is-a (F) committed as the terminal direction? wrap_kvs will wrap the instance but self of instance is not wrapped #18's doc recommends it; its §9 still
   asks. Everything else is a holding pattern, so the answer changes how much to invest in the
   holding pattern.
2. Should wrapped_self's lost-reference hole be fixed, or left as a known limitation? A
   two-line change to _register_wrapper_backref makes it detectable-and-loud (detail in wrap_kvs will wrap the instance but self of instance is not wrapped #18).
   Cheap, but it hardens a mechanism F would delete.
3. Should dol grow the sibling-store pattern as a first-class thing — a helper that
   re-derives a wrapper chain onto a sibling store, which is Making conditional_data_trans better, and generally; recursively applying wrappers #10 — or stay out of it and let
   adapters ask users to wrap in parallel?
4. Should SupportsUrlFor stay method-shaped? It currently forces every backend to expose a
   keyed method, which is the shape this whole thread is about. A store-shaped or
   function-shaped seam would not.
5. Is there an inverse of inner_most_key? Mapping a key outward (leaf → caller) has no
   helper, and anything returning keys (prefixes, walk, a query) needs one.

References

Issues: #83 (this problem + census) · #18 (root cause, wrapped_self) · #82 (prefix
corruption) · #10 (recursive wrapping) · #6 (signature freeze) · #5 (wrapper class control).
PRs: #84, #85.
In-tree: misc/docs/dol_issue18_design.md, misc/docs/dol_issue10_design.md,
misc/docs/dol_content_metadata_bifurcation.md §2.2.
Downstream: s3dol#14,
s3dol ADR-0011.

[thorwhalen]

Option G — spec-carried boundary codecs on a flat proxy (design study + prototype)

Following up on the maintainer's proposal (2026-08-10, relayed off-thread — putting it on record here): study wrapt's object-proxy design; wrap incoming/outgoing keys and values in all methods per a specification (e.g. a Protocol annotated with KT/VT); generalize beyond KT/VT to arbitrary "types of interest"; accumulate wrap layers in two flat lists (encoders/decoders) and compile them for speed and validation; take care that method-calls-method never double-applies.

This is now evaluated as Option G in this thread's option space:

• Design study: misc/docs/dol_issue86_design.md (survived a 4-lens adversarial panel + an independent code review; corrections folded in and flagged)
• Prototype: dol/_interface_wrap.py (private, experimental, stdlib-only) + 39 tests — landed in Option G design study + prototype: spec-carried boundary codecs on a flat proxy (#86) #88

The short version

1. wrapt: take the lessons, not the goal. Verified on wrapt 2.3.0: its proxy has exactly our wrap_kvs will wrap the instance but self of instance is not wrapped #18 hole (self inside a wrapped method is the wrapped object — the proxy intercepts only the first hop), pickling refuses by default, and its __iter__ mistake is a ten-year cautionary tale about eagerly-defined dunders. Also: the pure-Python proxy measured ~5× faster than the C extension on CPython 3.12 — a dependency-free carrier costs nothing.
2. The flat compiled stack is the load-bearing idea. Re-wrapping extends a list instead of nesting an object, so — within pure-codec stacks — inner_most_key becomes a total fold, the inverse key mapping (§5.4 of the Key-transform wrappers delegate capability methods with the unmapped key (url_for etc. silently address the wrong object) #83 study — previously nonexistent) exists by construction, the leaf is a strong structural reference (§5.6), stacked wraps pickle by construction, and the measured ~420–510 ns/layer delegation tax collapses to one hop (3.6× faster at depth 6). Scope limit, stated plainly: filt_iter (key-set change) and cached_keys (stateful) are not codecs; mixed compositions still nest.
3. The typed spec is Option B, replaced — with B's verdict answered. "The guard is worth more than the registry": in G the guard is the mechanism — three wrap/compile-time refusal layers (undeclared attributes; unannotated params inside spec'd methods; unsupported shapes) — and the registry is derived from annotations, not maintained. The adapter-side reflective conformance test (ADR-0011 D5) stays; the wrap-time guard serves the other desk.
4. The boundary invariant has fine print. "A method correct on the bare leaf stays correct under the stack" holds iff the codec laws hold: decoder total+injective on the leaf's actual keys (our own prefixed/suffixed codecs violate this on out-of-band keys — the ADR-0006 corruption family), encoder/decoder mutual inverses, container args copied at the boundary. No-double-apply on internal self-calls is structural (verified with a counting encoder).
5. Headline finding — F's verdict needs a correction. Is-a wrapping does not fix Key-transform wrappers delegate capability methods with the unmapped key (url_for etc. silently address the wrong object) #83 for backend-direct method bodies (verified): a body like cosmodol.replace that hands its key straight to the backend still receives the outer key under is-a. So "both Key-transform wrappers delegate capability methods with the unmapped key (url_for etc. silently address the wrong object) #83 and wrap_kvs will wrap the instance but self of instance is not wrapped #18 disappear" overclaims; F and G serve disjoint populations (F: outer-domain bodies, wrap_kvs will wrap the instance but self of instance is not wrapped #18; G: leaf-domain bodies, Key-transform wrappers delegate capability methods with the unmapped key (url_for etc. silently address the wrong object) #83), and this thread's open question 1 ("is is-a committed as the terminal direction?") becomes "committed for which population". Bonus hazard for F's Phase 3: a leaf that owns _id_of_key (s3dol's shape) gets its prefix arithmetic shadowed by a naive is-a hook.
6. A decision rule for the two blessed patterns (proposed): spec expressibility is the line. Shapes the spec can express (scalar KT, Iterable[KT], tuple[KT, VT]…) MAY be methods iff spec'd; shapes it refuses (key-space-shifting returns, cross-keyspace values, keys-in-records/exceptions, view-dependent no-arg ops) MUST stay sibling stores / handles / free functions. Keeps §4 rung 1's incentive intact.
7. Of this thread's §5 open questions: Q5 is answered (the inverse exists by construction in the flat model); Q4 softens (a keyed method with a spec entry now has a correctness story, so SupportsUrlFor being method-shaped is less damning); Q1 splits per finding 5. And a new question 0 for the maintainer: what does wrap_kvs compile to at the endgame — is-a (the wrap_kvs will wrap the instance but self of instance is not wrapped #18 doc's Phase 3) or the flat engine (this doc's P2)? They currently contradict; a plausible synthesis is codec semantics → flat engine, class-decorator usage → is-a.

The doc's §9 rescopes migration honestly (P2 is not "wrap_kvs becomes a facade" — inner-layer wants_self/postget observe their layer's domain, which fusion cannot materialize; route 2 (mk_relative_path_store → Files) is separate), and §10 sets the mixed old/new stack policy (legacy Store = opaque leaf, with a warning).

[thorwhalen]

Decisions on the Option G open questions (maintainer, 2026-08-10)

All six questions from dol_issue86_design.md §11 are decided; §11 is now a decision record, and the code-level ones landed in #89:

• Q0 — wrap_kvs endgame: split synthesis. Codec/instance wrapping compiles to the flat engine; @wrap_kvs class-decoration becomes is-a — each mechanism owns the population it is uniquely correct for (backend-direct Key-transform wrappers delegate capability methods with the unmapped key (url_for etc. silently address the wrong object) #83 bodies vs outer-view wrap_kvs will wrap the instance but self of instance is not wrapped #18 bodies). This supersedes the "flat vs is-a" rivalry between this doc's P2 and the wrap_kvs will wrap the instance but self of instance is not wrapped #18 doc's Phase 3; the crisp fire-when rule is P2/P3 design work.
• Q1 — surface: private + facade. The engine stays private; MappingInterface (built-in spec) and kv_interface_wrap (wrap_kvs-shaped kwargs facade) shipped so the simple gesture never regresses. Export reconsidered after P1 adapter experience.
• Q2 — loudness default: 'exclude'. Wrap succeeds; using an undeclared attribute raises with guidance. 'raise' remains as strict mode. (Wrap-time raise on real leaves — dict: 11 publics, boto3: 122 — was judged to drive passthrough-as-habit.)
• Q3 — typed codecs: yes, now. Codec carries optional decoded_type/encoded_type; adjacent stack seams are validated at wrap time (outer encoded_type ≡ inner decoded_type), loud on mismatch, untagged unchecked. Tagging dol.trans.Codec waits for P2 (dependents gate).
• Q4 — eq/hash: outer-view eq, no hash when the spec'd surface covers __getitem__+__iter__ (a wrap equals a dict of its outer items; __hash__ is None). Store's own eq/hash incoherence queued for 0.4.
• Q5 — __class__ transparency: off; opt-in later, co-designed with Better way to control the wrapper class #5.

Next milestones per the (rescoped) plan: P1 — census-family adapters trial interface_wrap for spec-expressible capability surfaces (sibling stores stay blessed for refused shapes, per the §7 decision rule), plus the adapter-side conformance helper; P2 — the split-synthesis compatibility appendix (.store surface, signature graft, wrapped_self backref) before any wrap_kvs wiring.

[thorwhalen]

P1 adapter trial — interface_wrap on one surface (s3dol's url_for)

Narrowly scoped as planned: one method, the guarded url_for shim that s3dol ADR-0011 §D3b kept purely to satisfy dol.SupportsUrlFor. Sibling stores remain s3dol's blessed pattern; nothing else was migrated. Everything below was executed against dol/_interface_wrap.py at v0.3.63.

0. The trial got a sharper baseline than the one we designed against

Implementing s3dol v1 turned up a new fact about the delegated-method population, and it is worse than "the wrapper may be unreachable" (§D1a):

Two call paths reach one keyed method with different key domains, and nothing at the callee distinguishes them.

caller	what url_for receives
dol.content_url(wrapped, k)	the key already mapped through the outer layers — the walk stops at the provider "because that layer applies its own" (content.py::_url_for_provider_and_key)
wrapped.url_for(k) (delegation)	the raw outer key

For a leaf that owns a prefix, applying _id_of_key is right for the first and wrong for the second; re-resolving through the chain is right for the second and double-transforms the first (measured: a store scoped p/ under an x/ key codec produced p/x/x/g). Verified baseline:

wrapped['g']                      ok   b'V'
wrapped.url_for('g')              BAD  https://bucket.example/p/g?e=3600      <- unmapped
content_url(wrapped, 'g')         ok   https://bucket.example/p/x/g?e=3600

So wrapped_self is not merely a leaky guardrail — no implementation of the method is correct for both callers. s3dol now refuses (NotSupported, naming s3dol.urls(store)[k]) as soon as the leaf has been wrapped. That is a finding, not a design preference, and it strengthens §5 item 2 of the #83 study: a wrapper must be able to express "this method takes a key" and there must be exactly one key domain at the boundary.

1. Headline: G fixes this population. Cleanly.

One module-level Protocol, six methods, ~8 lines of spec — and every shape that fails under delegation passes:

proxy['g']                        ok   b'V'
proxy.url_for('g')                ok   https://bucket.example/p/x/g?e=3600
proxy.url_for('g', expires_in=60) ok   https://bucket.example/p/x/g?e=60     <- kwargs pass through
content_url(proxy, 'g')           ok   https://bucket.example/p/x/g?e=3600
list(proxy)                       ok   ['g']                                 <- outward mapping
temporary proxy, one expression   ok   https://bucket.example/p/x/g?e=3600   <- D1a's silent hole: gone
stacked codecs (x/ then y/)       ok   https://bucket.example/p/x/y/g?e=3600
pickle round-trip                 ok   b'V'

Why it works is structural, and worth stating explicitly: the ambiguity in §0 exists because there is a chain a caller can pre-resolve through. The flat model has no chain — one boundary, one key domain, one place a key can be encoded. content_url is then correct through a proxy by construction: its walk finds no .store, returns (proxy, k) immediately, and the proxy's own boundary does the only encoding that happens. The double-transform is not fixed; it is unrepresentable.

That is a concrete instance of the split-synthesis claim: F (is-a) would not have helped here at all — the leaf's url_for body is backend-direct and never calls self[k].

2. Spec-authoring cost: low, with one gotcha already documented

The spec is ~8 lines; roles matched by TypeVar name as designed; KT/VT on __getitem__/__iter__ gave outward mapping for free. The only friction was the known one: the spec must be module-level (a function-local Protocol + the deferred annotations it implies → NameError at wrap time). That is in the design doc; it cost one iteration anyway, which suggests the error message should say it outright.

3. Two things I'd change before P1 closes

(a) undeclared='exclude' does not cover private attributes — and for an adapter that is where the danger lives.

proxy.prefix        -> UndeclaredAttributeError   (good)
proxy._id_of_key('g') -> 'p/g'                    (silently the LEAF's domain)

_id_of_key/_key_of_id are exactly what dol.dig, inner_most_key, and every free-function key resolver walk. A proxy that forwards them verbatim will hand a chain-walker leaf-domain answers with no boundary applied — the same silent-wrong-key class the engine exists to end, reached through the back door. Options: exclude private names by default too; or treat _id_of_key/_key_of_id/store as reserved and either map or refuse them. Either way it should not be a passthrough by default.

(b) There is no stable, public way to ask "what is the leaf?" / "is this flat?"

I tried to assert flatness and hit AttributeError: 'LeafInterfaceProxy' object has no attribute '_leaf'. Adapter code and its tests want this — s3dol's own conformance test asserts structural properties of its stores. A tiny interface_leaf(proxy) / interface_stack(proxy) accessor pair would do, and it keeps people out of whatever the internal attribute is actually called.

4. Confirmed as designed (no action)

The codec laws (§2.5) bite in practice, exactly as written. With an out-of-band key in scope, a prefix codec's decoder is not injective and the boundary collapses two keys into one:

list(proxy) with an out-of-band key -> ['g', 'g']

That is dol#82 in one line. It confirms the doc's scoping: flat-model guarantees hold for law-abiding codecs, and filter-first composition is still the adapter's job. Worth keeping as the canonical one-liner example in the docs — it makes the fine print concrete in a way prose does not.

5. What the spec still refuses — s3dol's carry-forward list

These stayed sibling stores / handles / free functions, and I'd expect every blob adapter to have the same four:

shape	s3dol case
key-space-shifting return	sub(prefix) → a store in a new key space
a key embedded in a returned object	handles(s)[k] → an ObjectHandle holding the wire key
keys inside an exception payload	S3PartialFailure.failures is keyed by the keys that failed
strings that are not keys	prefixes() → directory strings a key codec would corrupt

The exception-payload row is the one I'd flag for §5 of the #83 study. The raise site genuinely has no boundary frame — but note s3dol could fix its own case, because the free function holds the store and can run the inverse walk itself. That is a hint about the general shape of the answer: the boundary can be re-entered from the outside when someone still holds the outer object, which is precisely what free functions (and sibling stores) have and delegated methods do not.

6. Verdict

For the population it serves, G is not a marginal improvement — it converts a method that cannot be made correct into one that is correct by construction, including the two shapes (temporary wrapper, stacked codecs) that defeated every prior option. s3dol keeps sibling stores as its blessed pattern per the spec-expressibility rule, and the url_for refusal stands as the honest answer under 0.3.x; but if/when the engine goes public, that refusal becomes a one-line spec entry instead.

(Trial script and full output are in the s3dol v1 work; s3dol PRs #19 and #20.)