s3dol v1: Layer B/C, sibling capability stores, compat shim, testing (ADRs 0001–0012)

README.md

@@ -7,6 +7,63 @@ To install:	```pip install s3dol```
 [Documentation](https://i2mint.github.io/s3dol/)
 
 
+## Quick start
+
+```python
+import s3dol
+
+s = s3dol.s3_store('my-bucket')                  # MutableMapping[str, bytes]
+s['hello.txt'] = b'world'
+s['hello.txt']                                   # b'world'
+list(s); 'hello.txt' in s; del s['hello.txt']
+
+s = s3dol.s3_store('my-bucket', prefix='logs/')  # scoped to a prefix
+s = s3dol.s3_store('open-data', anon=True)       # public bucket, no credentials
+s = s3dol.s3_store('b', preset='minio', endpoint_url='http://localhost:9000')
+```
+
+Starting from nothing (the default never creates a bucket from a typo):
+
+```python
+s = s3dol.s3_store('brand-new-bucket', on_missing_bucket='create')
+```
+
+**Big objects go through the same interface** — no `upload_multipart` method:
+
+```python
+s['video.mp4'] = s3dol.Filepath('/tmp/video.mp4')   # streamed, multipart above 8 MiB
+s['stream.bin'] = s3dol.Chunks(chunk_iterator)       # never fully in memory
+```
+
+**Keyed capabilities are stores you index**, not methods (a `dol` key wrapper
+hands a method the *unmapped* key — so s3dol has none):
+
+```python
+s3dol.handles(s)['video.mp4'].read(offset=0, length=1024)   # ranged read
+s3dol.handles(s)['video.mp4'].url(expires_in=3600)          # presigned URL
+s3dol.urls(s)['video.mp4']                                  # ...or directly
+s3dol.info(s)['video.mp4'].size                             # one HeadObject
+```
+
+Everything else is a free function taking the store first:
+
+```python
+s3dol.sub(s, 'folder/'); s3dol.prefixes(s)
+s3dol.delete_many(s, ['a', 'b']); s3dol.delete_bucket(endpoint, 'name', force=True)
+```
+
+**Test without a cloud** — no network, no docker, no moto:
+
+```python
+from s3dol.testing import mock_s3, run_conformance
+
+def test_my_service():
+    assert MediaService(mock_s3(data={'a.mp4': b'...'})).play('a.mp4')
+```
+
+`run_conformance(make_store)` is exported too: run the same laws against your
+own store (or a sibling `*dol` package's).
+
 ## Heads-up: v1 is coming — run `s3dol.diagnose()` now
 
 A major, behaviour-clarifying v1 is in progress
@@ -23,7 +80,7 @@ Before upgrading to v1, run the diagnosis with the same arguments you pass
 ```python
 import s3dol
 
-s3dol.diagnose(bucket_name="my-bucket", endpoint_url="...", path="...")
+s3dol.diagnose(bucket_name='my-bucket', endpoint_url='...', path='...')
 ```
 
 It prints what resolves (endpoint, region, signing, credential *source* — never

conftest.py

@@ -33,11 +33,18 @@
 _S3_FREE_TEST_STEMS = frozenset(
     {
         "test_url_for",
-        # Layer A (v1) suites: tier-1 pure + moto-mocked tier-2 — no endpoint.
+        # v1 suites: tier-1 pure + moto-mocked tier-2 — no live endpoint.
         "test_errors",
         "test_presets",
         "test_connection",
         "test_diagnose",
+        "test_values",
+        "test_keyed_surface",
+        "test_conformance",
+        "test_base",
+        "test_capabilities",
+        "test_store_shim",
+        "test_recipes",
     }
 )
 

misc/docs/decisions/0011-keyed-capability-surface.md

@@ -227,6 +227,39 @@ documented limitation, and the real fix is upstream: `content_url` must resolve
 chain **and** call the innermost `url_for` with the resolved key — resolving without that second
 half would double-transform.
 
+#### D3b-amended (2026-08-11, during implementation) — the guarded method cannot serve both call paths
+
+The guard above assumed one call path. There are **two**, and they hand the *same signature*
+**different key domains**:
+
+| caller | what `url_for` receives | correct action |
+|---|---|---|
+| `dol.content_url(wrapped, k)` | the key already mapped through the **outer** layers — because the walk stops at the layer owning `url_for` "*because that layer applies its own*" (`dol/content.py::_url_for_provider_and_key`) | apply `self._id_of_key` |
+| `wrapped.url_for(k)` (delegation) | the raw **outer** key | resolve through the whole chain |
+
+Applying `_id_of_key` is right for the first and wrong for the second; re-resolving through the
+chain is right for the second and **double-transforms** the first (measured: a store scoped to
+`p/` under a `x/` key codec produced `p/x/x/g`). Nothing at the callee distinguishes them —
+this is dol#83's defect reaching the one method the ADR kept.
+
+**Decision:** `url_for` applies `self._id_of_key(k)` (dol's stated contract), and **refuses**
+— `NotSupported`, naming `s3dol.urls(store)[k]` — as soon as the leaf has ever been wrapped.
+Consequences, stated plainly:
+
+- `dol.content_url` is correct for an **unwrapped** s3dol store (the common case, and the one
+  `dol/content.py`'s docstring shows) and **raises** for a wrapped one. Loud, never wrong.
+- The refusal is driven by a *sticky flag* set when dol probes the leaf (`hasattr(store,
+  'KeysView')` in `Store.__init__`), implemented as a descriptor because a `KvReader` leaf
+  would otherwise satisfy the probe from `MappingViewMixin` and never see it. Unlike the
+  `wrapped_self` weakref, the flag survives the wrapper's death — so the refusal is reliable,
+  not intermittent. The documented false positive stands: a once-wrapped leaf later used bare
+  also refuses.
+
+**This is the P1 evidence for dol#86/#83**: a keyed capability *method* is unfixable in the
+general case — not merely fragile, and not fixable by a better resolution primitive, because
+the ambiguity is in the *call*, not the resolution. Sibling stores (D2) route through
+`__getitem__`, which has one key domain and one caller, and so remain correct by construction.
+
 ### D4 — Destructive operations are free functions
 
 `delete_many(keys)` and `EndpointStore.delete(name, force=True)` both leave Layer B.

misc/docs/state-of-play.md

@@ -67,7 +67,20 @@ consumer's requirements.
 | Resolved design questions | **§7** / **#14** → [ADR-0011](decisions/0011-keyed-capability-surface.md) · **§8** / **#15** → [ADR-0012](decisions/0012-credential-and-endpoint-resolution.md) |
 | Open design questions | none — implementation is unblocked; see §11 |
 
-**Nothing has been implemented.** The repo still ships v0.1.9 unchanged.
+**Implementation is under way** (2026-08-11). Layer A landed as an additive
+0.1.x change (PR #19: `errors.py`, `presets.py`, `connection.py`, `_diagnose.py`
+— ADR-0007 §5 step 0, so dependents can record what resolves *before* anything
+moves), and the v1 surface followed on top (`values`/`writes`/`reads`/`base`/
+`capabilities`/`recipes`/`testing` + the `store.py` compat shim). Read §11 for
+what remains.
+
+One design amendment came out of implementation and is recorded where it
+belongs, in [ADR-0011 §D3b-amended](decisions/0011-keyed-capability-surface.md):
+the guarded `url_for` shim **cannot** serve both of its call paths, because
+`dol.content_url` and a direct delegated call hand the same signature different
+key domains. It now refuses once the leaf has been wrapped. That is the P1
+evidence for dol#86/#83 — a keyed capability *method* is unfixable in the
+general case, and not for want of a better resolution primitive.
 
 ## 3. The twelve ADRs, in one paragraph each
 

pyproject.toml

@@ -30,7 +30,12 @@ classifiers = [
     "Topic :: Software Development :: Libraries :: Python Modules",
 ]
 dependencies = [
-    "dol",
+    # A hard floor, not a preference (ADR-0006 §3's upstream policy: raise the
+    # floor the day each fix lands, and delete the local workaround in the same
+    # commit). v1 needs dol.inner_most_key exported (0.3.62), content_url
+    # resolving keys through the wrapper chain (0.3.62), and filesys
+    # is_valid_key fixed — all shipped by 0.3.63.
+    "dol>=0.3.63",
     "boto3",
     # ADR-0012: the endpoint env vars (AWS_ENDPOINT_URL_S3 / AWS_ENDPOINT_URL) do not exist
     # before botocore 1.31, and ADR-0003 §3's checksum knobs
@@ -49,12 +54,17 @@ test = [
     "pytest>=7.0",
     "pytest-cov>=4.0",
     "moto[s3]>=5.0",
+    # conformance assertion 2b fetches a presigned URL; moto patches
+    # `requests` but NOT urllib (which would leave the machine). Declared
+    # explicitly rather than relied on transitively via moto[s3].
+    "requests",
 ]
 dev = [
     "pytest>=7.0",
     "pytest-cov>=4.0",
     "ruff>=0.1.0",
     "moto[s3]>=5.0",
+    "requests",
 ]
 docs = [
     "sphinx>=6.0",
@@ -144,10 +154,14 @@ enabled = false
 [tool.wads.ci.testing]
 python_versions = [
     "3.10",
+    "3.12",
 ]
 pytest_args = [
     "-v",
     "--tb=short",
+    # Every documented example is executable against the in-memory fake, so
+    # doctests run in the default gate (ADR-0008).
+    "--doctest-modules",
 ]
 coverage_enabled = true
 coverage_threshold = 0

s3dol/__init__.py

@@ -1,26 +1,117 @@
-"""
-s3 (through boto3) with a simple (dict-like or list-like) interface
-"""
+"""s3dol — S3 and S3-compatible object storage behind a dict-like interface.
+
+The one-liner::
+
+    import s3dol
+    s = s3dol.s3_store('my-bucket')          # MutableMapping[str, bytes]
+    s['k'] = b'v'; s['k']; list(s); del s['k']
+
+Keyed capabilities are **sibling stores you index** (never methods — a dol key
+wrapper would hand a method the unmapped key; ADR-0011)::
+
+    s3dol.handles(s)['k']    # ObjectHandle (ranged reads, streams, metadata)
+    s3dol.urls(s)['k']       # presigned URL
+    s3dol.info(s)['k']       # ObjectInfo
+
+Non-keyed operations are free functions taking the store first::
+
+    s3dol.sub(s, 'folder/')          s3dol.prefixes(s)
+    s3dol.delete_many(s, keys)       s3dol.delete_bucket(endpoint, name, force=True)
 
-from s3dol.base import S3Dol
-from s3dol.store import S3Store
+Test without a cloud: ``from s3dol.testing import mock_s3, run_conformance``.
+Migrating from v0? ``s3dol.diagnose(**your_S3Store_kwargs)`` prints what
+resolves, from where, and whether v1 moves it. The legacy ``S3Store`` keeps
+working via ``s3dol.store`` (deprecated, removed in v2).
 
-# New (v1) layer-A names, loaded lazily (PEP 562) so `import s3dol` does not
-# pay for what a script does not use. `s3dol.diagnose(...)` is the ADR-0007 §5
-# step-0 migration safety mechanism — run it with your current S3Store kwargs
-# to see what resolves, from where, and whether the upcoming v1 moves it.
+Everything here loads lazily (PEP 562): ``import s3dol`` does not import boto3.
+"""
+
+# name -> module path (attribute of the same name), or (module path, attribute)
 _LAZY_ATTRIBUTES = {
-    "diagnose": ("s3dol._diagnose", "diagnose"),
+    # Layer C — recipes
+    "s3_store": "s3dol.recipes",
+    "S3Texts": "s3dol.recipes",
+    "S3Jsons": "s3dol.recipes",
+    "S3Pickles": "s3dol.recipes",
+    # Layer B — stores and the handle
+    "BucketCollection": "s3dol.base",
+    "BucketReader": "s3dol.base",
+    "BucketStore": "s3dol.base",
+    "EndpointCollection": "s3dol.base",
+    "EndpointReader": "s3dol.base",
+    "EndpointStore": "s3dol.base",
+    "ObjectHandle": "s3dol.base",
+    "ObjectInfo": "s3dol.base",
+    "S3Profiles": "s3dol.base",
+    # sibling capability stores + accessors (ADR-0011 D2)
+    "BucketHandles": "s3dol.capabilities",
+    "BucketUrls": "s3dol.capabilities",
+    "BucketInfo": "s3dol.capabilities",
+    "handles": "s3dol.capabilities",
+    "urls": "s3dol.capabilities",
+    "info": "s3dol.capabilities",
+    # free functions (ADR-0011 D3/D4)
+    "sub": "s3dol.capabilities",
+    "prefixes": "s3dol.capabilities",
+    "delete_many": "s3dol.capabilities",
+    "delete_bucket": "s3dol.capabilities",
+    # value refs (ADR-0005)
+    "Filepath": "s3dol.values",
+    "Chunks": "s3dol.values",
+    "Streamable": "s3dol.values",
+    # Layer A — connection + presets
     "S3Connection": "s3dol.connection",
     "resolve": "s3dol.connection",
     "Resolution": "s3dol.connection",
+    "StaticCredentials": "s3dol.connection",
+    "ProfileCredentials": "s3dol.connection",
+    "CallableCredentials": "s3dol.connection",
     "Preset": "s3dol.presets",
     "Capabilities": "s3dol.presets",
     "register_preset": ("s3dol.presets", "register"),
+    # errors (the taxonomy — `except s3dol.S3Error` catches the package)
+    "S3Error": "s3dol.errors",
+    "ObjectNotFound": "s3dol.errors",
+    "BucketNotFound": "s3dol.errors",
+    "KeyNotValid": "s3dol.errors",
+    "ObjectArchived": "s3dol.errors",
+    "AccessDenied": "s3dol.errors",
+    "CredentialsError": "s3dol.errors",
+    "NotSupported": "s3dol.errors",
+    "BucketNotEmpty": "s3dol.errors",
+    "S3PartialFailure": "s3dol.errors",
+    "ConfigurationError": "s3dol.errors",
+    "S3DolWarning": "s3dol.errors",
+    # diagnostics
+    "diagnose": ("s3dol._diagnose", "diagnose"),
+    # legacy (deprecated; removed in v2)
+    "S3Store": "s3dol.store",
 }
 
+__all__ = sorted(_LAZY_ATTRIBUTES) + ["S3Dol", "__version__"]
+
 
 def __getattr__(name):
+    if name == "__version__":
+        import importlib.metadata
+
+        try:
+            return importlib.metadata.version("s3dol")
+        except importlib.metadata.PackageNotFoundError:
+            return "unknown"
+    if name == "S3Dol":
+        # v0's profiles->buckets->keys explorer, renamed (ADR-0007 §1).
+        import warnings
+
+        warnings.warn(
+            "s3dol.S3Dol is deprecated (removed in v2): it is now "
+            "s3dol.S3Profiles — its keys are AWS profile names.",
+            DeprecationWarning,
+            stacklevel=2,
+        )
+        from s3dol.base import S3Profiles
+
+        return S3Profiles
     try:
         target = _LAZY_ATTRIBUTES[name]
     except KeyError:
@@ -33,12 +124,12 @@ def __getattr__(name):
         module_path, attribute = target, name
     value = getattr(importlib.import_module(module_path), attribute)
     # Cache the resolved attribute. This ALSO deliberately overrides the
-    # submodule binding that `import s3dol.diagnose` just created on this
-    # package — without it, `s3dol.diagnose` would resolve to the module on
-    # every access after the first, shadowing the function.
+    # submodule binding a same-named submodule import creates on this package
+    # (e.g. `info` vs a hypothetical info module) — without it, the second
+    # access would resolve to the module and shadow the function.
     globals()[name] = value
     return value
 
 
 def __dir__():
-    return sorted(set(globals()) | set(_LAZY_ATTRIBUTES))
+    return sorted(set(globals()) | set(__all__))