diff --git a/.claude/skills/dol-store-building/SKILL.md b/.claude/skills/dol-store-building/SKILL.md new file mode 100644 index 00000000..d179e4d6 --- /dev/null +++ b/.claude/skills/dol-store-building/SKILL.md @@ -0,0 +1,125 @@ +--- +name: dol-store-building +description: "Build a dol store: wrap any storage backend (files, S3, DB, dict, an API) behind a uniform dict-like (MutableMapping) interface, adding key and value transforms/serialization. Use when a user wants to give a backend a dict interface, add JSON/pickle/gzip (or custom) serialization to a store, transform or filter keys, compose codecs, cache a slow store, or asks 'how do I use dol to ...'. Covers wrap_kvs (the core), the ValueCodecs/KeyCodecs namespaces, Pipe composition, the ready-made file stores (Files/TextFiles/JsonFiles/PickleFiles), filt_iter, the test-with-dict-then-swap-backend workflow, and self-aware transforms via FirstArgIsMapping. For authoring interactive scaffolds see the /new-store, /add-codec, /explain-store commands; for modifying dol's internals see dol-dev-wrap-kvs." +--- + +# Building a dol store + +dol turns any storage backend into a `dict`-like object: `s[key]` reads, `s[key] = val` +writes, `del s[key]` deletes, `for k in s` / `len(s)` / `k in s` explore. You write your +logic against this uniform interface and swap backends freely. The core move is **wrapping +a backend with key/value transforms**. + +## The golden workflow: prototype with `dict`, then swap the backend + +Always build and test with a plain `dict` first, then swap in the real backend — the +transform code is identical: +```python +from dol import wrap_kvs +import json + +# 1. logic first, dict backend +S = wrap_kvs(dict, obj_of_data=json.loads, data_of_obj=json.dumps) +s = S(); s['x'] = {'a': 1}; assert s['x'] == {'a': 1} + +# 2. same transforms, real backend +from dol import Files +s = wrap_kvs(Files('/data'), obj_of_data=json.loads, data_of_obj=json.dumps) +``` + +## `wrap_kvs` — the core, and its `X_of_Y` naming + +`X_of_Y(y) -> x`. Outgoing (read) transforms produce what the user sees; ingoing (write) +transforms produce what the backend stores. They come in inverse pairs: + +| kwarg | direction | signature | use | +|---|---|---|---| +| `key_of_id` | read (key out) | `k = key_of_id(_id)` | backend id → user key | +| `id_of_key` | write (key in) | `_id = id_of_key(k)` | user key → backend id | +| `obj_of_data` | read (value out) | `obj = obj_of_data(data)` | deserialize | +| `data_of_obj` | write (value in) | `data = data_of_obj(obj)` | serialize | +| `postget` | read, key-aware | `obj = postget(k, data)` | deserialize depending on key (e.g. by extension) | +| `preset` | write, key-aware | `data = preset(k, obj)` | serialize depending on key | + +Use `obj_of_data`/`data_of_obj` when the transform is the same for all values; use +`postget`/`preset` when it depends on the key. + +## Prefer ready-made codecs over hand-rolled lambdas + +```python +from dol import ValueCodecs, KeyCodecs, Pipe +ValueCodecs.json() # json.dumps / json.loads +ValueCodecs.pickle() # pickle +ValueCodecs.gzip() # compress/decompress +KeyCodecs.suffixed('.json') # add/strip a key suffix +KeyCodecs.prefixed('ns:') # add/strip a key prefix + +# Compose with + or Pipe (order = application order on the backend side) +MyStore = Pipe(KeyCodecs.suffixed('.pkl'), ValueCodecs.pickle() + ValueCodecs.gzip())(dict) +``` + +## Ready-made file stores (skip wrap_kvs when one fits) + +```python +from dol import Files, TextFiles, JsonFiles, PickleFiles +Files('/data') # keys=relative paths, values=bytes +TextFiles('/data') # values=str +JsonFiles('/data') # values=json-decoded objects +``` + +## Filtering the key space + +```python +from dol import filt_iter +s = filt_iter(store, filt=lambda k: k.endswith('.json')) +# ready-made variants: +filt_iter.suffixes('.json'); filt_iter.prefixes('user/'); filt_iter.regex(r'\d{4}') +``` + +## Caching a slow store + +```python +from dol import cache_vals, cache_this +fast = cache_vals(slow_store) # in-memory read cache +class C: + @cache_this(cache='_c') # cache an expensive property/method + def expensive(self): ... +``` + +## Transforms that need the store itself: `FirstArgIsMapping` + +Most transforms are pure `f(value)`. When a transform genuinely needs the store instance +(its config, root path, etc.), mark it so dol passes the store as the first arg — instead +of relying on parameter names: +```python +from dol import wrap_kvs, FirstArgIsMapping +def resolve(self, data): # first arg is the store + return f"{self.root}/{data}" +s = wrap_kvs(store, obj_of_data=FirstArgIsMapping(resolve)) +``` +Note: passing a bare `def f(self, data)` (first param named `self`/`store`/`mapping`, ≥2 +required params) also works via dol's heuristic — but `FirstArgIsMapping` is explicit and +robust (a plain unary builtin like `bytes.decode` is correctly treated as `f(data)`). + +## Read-only vs read-write + +Subclass `KvReader` for read-only stores, `KvPersister` for read-write, or just use +`wrap_kvs` on a backend that is/ isn't writable. Note `clear()` is disabled on persisters +by default (guard against wiping a backend); re-enable deliberately if you must. + +## Gotchas + +- **`bytes.decode` as `obj_of_data`** now works, but prefer `ValueCodecs.str_to_bytes()` + or `lambda b: b.decode()` for clarity. +- **`clear()` is disabled** on `KvPersister` — deliberate. +- **A wrapped class's own methods see the *unwrapped* store as `self`** — if you write a + method that does `self[k]` inside a `@wrap_kvs`-decorated class, re-wrap `self` (advanced; + see the dol-dev-wrap-kvs skill / Issue #18). +- **Compose, don't subclass**, when adding transforms: `wrap_kvs`/codecs/`Pipe` compose + cleanly; subclassing `Store` is only for custom hook protocols. + +## When to reach for the commands + +- `/new-store ` — scaffold a new store class/factory interactively. +- `/add-codec ` — add or create a codec. +- `/explain-store ` — trace an existing store's transform pipeline. diff --git a/.gitignore b/.gitignore index 28741e2e..e51ceed5 100644 --- a/.gitignore +++ b/.gitignore @@ -117,3 +117,6 @@ venv.bak/ # Notebook playground vf.ipynb + +# Local-only ecosystem inventory (names private packages) +misc/data/ diff --git a/CLAUDE.md b/CLAUDE.md index 5b4455ee..b2f703a6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -185,15 +185,38 @@ Run tests: `pytest dol/tests/` |----------|----------| | [general_design.md](misc/docs/general_design.md) | Language-agnostic design: what dol is, the KV pipeline, layered composition, patterns | | [dol_design.md](misc/docs/dol_design.md) | Python architecture: class hierarchy, `wrap_kvs` deep dive, `Codec`/`Sig`/`Pipe`, critique | +| [dol_architecture_map.md](misc/docs/dol_architecture_map.md) | Code-verified structural map: module/dependency graph, public API, class hierarchy, `wrap_kvs`/codec machinery deep dive, ranked tech debt. **Start here for refactors.** | | [issues_and_discussions.md](misc/docs/issues_and_discussions.md) | GitHub issues/discussions themes, known limitations, open design questions | +| [dol_issues_report.md](misc/docs/dol_issues_report.md) | Prioritized issue triage + wave-by-wave tackle order | | [frontend_dol_ideas.md](misc/docs/frontend_dol_ideas.md) | `zoddal` design: TypeScript KV interface, adapters, Zod bridge, zod-collection-ui integration | +> A **local-only** ecosystem inventory (gitignored) lives in `misc/data/`: dol's 76 +> dependents, their usages (file:line), a pre-PR test-gate order + runner, and the +> `wrap_kvs` blast-radius scan. Regenerate with the scripts there. + +--- + +## Agent Skills & Commands (`.claude/`) + +**Dev skills** (`.claude/skills/`, for working *on* dol): +- `dol-dev-wrap-kvs` — the `wrap_kvs`/`store_decorator`/`Store.wrap` machinery: the + signature-conditioning rule, `FirstArgIsMapping`, the delegation architecture + `self`/ + signature traps (#18/#6), and the mandatory dependents test-gate. Read before touching + `trans.py`/`base.py`. +- `dol-dev-portability` — Windows/POSIX landmines for path/key code. + +**Consumer skills** (`.claude/skills/`, for *using* dol): +- `dol-store-building` — wrap any backend behind a dict interface: `wrap_kvs`, codecs, the + ready-made file stores, `filt_iter`, caching, and self-aware transforms. + +**Commands** (`.claude/commands/`): `/new-store`, `/add-codec`, `/explain-store` — interactive scaffolds. + --- ## Known Limitations / Gotchas -- **`wrap_kvs` + `self` inside methods**: When a `wrap_kvs`-decorated class uses `self[k]` in its own methods, `self` is the unwrapped instance. Re-apply the wrapper to `self` if transforms are needed (Issue #18). +- **`wrap_kvs` + `self` inside methods**: When a `wrap_kvs`-decorated class uses `self[k]` in its own methods, `self` is the unwrapped instance. Re-apply the wrapper to `self` if transforms are needed (Issue #18, still open — delegation architecture). - **`clear()` is disabled** on `KvPersister`. Call `ensure_clear_to_kv_store(store)` to re-enable. - **No async support** in core. Use synchronous wrappers for async backends (thread pool, etc.). -- **`bytes.decode` as `obj_of_data`** causes issues — use `lambda b: b.decode()` instead (Issue #9). -- **Windows paths**: Some path-related code has Unix assumptions. Issues #52, #58 track this. +- **Transforms wanting the store**: a transform is called `f(self, data)` only if its first param is named `self`/`store`/`mapping` **and** it has ≥2 required params; otherwise `f(data)`. Mark explicitly with `FirstArgIsMapping(f)`. (`bytes.decode` as `obj_of_data` now works — Issue #9 fixed.) +- **Windows paths**: cross-platform fixes landed (Issues #40/#52/#58 resolved, CI green). See the `dol-dev-portability` skill before touching path/key code. diff --git a/misc/docs/dol_architecture_map.md b/misc/docs/dol_architecture_map.md new file mode 100644 index 00000000..7a4d0541 --- /dev/null +++ b/misc/docs/dol_architecture_map.md @@ -0,0 +1,609 @@ +# dol Architecture & Design Map + +> An authoritative, code-verified map of the `dol` library, intended for (1) a +> redesign/refactoring effort and (2) authoring "dev skills" (AI-agent tooling for +> developers working **on** dol). All non-obvious claims cite `file:line` against the +> **current** source (verified 2026-07, version 0.3.46, Python ≥ 3.10). + +--- + +## 1. Purpose & scope + +This document is the **structural/mechanical** reference for dol's Python source: every +module, the public API surface, the class hierarchy, and — in depth — the +`wrap_kvs`/`store_decorator`/codec machinery that is the heart of the library. It is +deliberately factual and line-cited so it can be trusted when refactoring or when writing +agent tooling that edits dol source. + +It **complements, does not repeat** the existing `misc/docs`: + +| For… | Read | This doc adds | +|---|---|---| +| Language-agnostic "what/why" (middleware, KV pipeline, Russian dolls) | [general_design.md](general_design.md) | — | +| Python architecture narrative + design critique | [dol_design.md](dol_design.md) | Verifies it against code; flags staleness (§ below) | +| GitHub issues/discussions themes | [issues_and_discussions.md](issues_and_discussions.md) | Corrects the Issue #9 root-cause (§5.4) | +| The split-store / content-metadata problem | [dol_content_metadata_bifurcation.md](dol_content_metadata_bifurcation.md) | — | +| Dead code / coverage tracker | [code-quality-improvements.md](code-quality-improvements.md) | Adds new debt found (§11) | +| Doc index | [dol_misc_docs_guide.md](dol_misc_docs_guide.md) | Should be updated to list this file | + +### Staleness / inaccuracies found in existing docs + +- **`issues_and_discussions.md` §1a stated the wrong root cause for Issue #9** (now + corrected in that doc). It said `wrap_kvs` "checks whether `obj_of_data` has 1 or 2+ + required args". The real (pre-fix) decision was the transform's **first parameter + *name*** ∈ `{"self","store","mapping"}` (`self_names`). Issue #9/#12 is now **FIXED** + (name **and** ≥2 required params + the `FirstArgIsMapping` marker) — see §5.4. +- **`dol_design.md` cites `wrap_kvs` at `trans.py:1801` and `store_decorator` at + `trans.py:130`.** `store_decorator` is still `:130`; `wrap_kvs` is now `:1813` + (decorator) / `:1814` (def). `Codec` is now `:3374` (doc says `:3362`). +- **`dol_design.md`'s `cache_this` examples are stale.** The current signature is + `cache_this(func=None, *, cache, key, pre_cache, as_property, ignore, serialize, + deserialize)` (`dol/caching.py:1055`). The whole `KeyStrategy`/`CachedProperty`/ + `CachedMethod` subsystem (`caching.py:126–1053`) is undocumented in existing docs. +- **`code-quality-improvements.md` LOC figures are per-*statement* (coverage), not raw + lines** — it lists `trans.py` as "789 lines" etc. Raw line counts are much larger + (see §2). Its dead-code items were spot-checked and are still present (§11). +- **`dol_design.md` claims `Store.__getattr__` returns `getattr(self.store, attr)`.** The + real implementation guards against pickling recursion via + `getattr(object.__getattribute__(self, "store"), attr)` (`dol/base.py:617`). + +--- + +## 2. Module map + +LOC = raw `wc -l`. "Internal deps" = other `dol.*` modules imported at **module load +time** (docstring-only and inline-function imports excluded). + +| Module | LOC | Purpose | Key public exports | Internal deps | +|---|---:|---|---|---| +| `__init__.py` | 208 | Public API surface; deprecation shim via module `__getattr__` | (re-exports — see §3) | everything | +| `base.py` | 1140 | Class hierarchy + hook protocol; `kv_walk`; delegation machinery | `Collection`, `KvReader`, `KvPersister`, `Store`, `MappingViewMixin`, `kv_walk`, `Stream` | errors, signatures, util | +| `trans.py` | 3492 | **The core.** `wrap_kvs`, `store_decorator`, `filt_iter`, `cached_keys`, `Codec`/`ValueCodec`/`KeyCodec`, `kv_wrap`, path-access adders | `wrap_kvs`, `filt_iter`, `cached_keys`, `kv_wrap`, `store_decorator`, `flatten`, `add_path_get/access`, `mk_read_only`, `Codec`* | base, errors, signatures, util | +| `caching.py` | 2675 | Caching layers: `cache_this`, `cache_vals`, `store_cached`, `WriteBackChainMap`, key-strategy protocol | `cache_this`, `cache_vals`, `store_cached`, `WriteBackChainMap`, `flush_on_exit`, `ensure_clear_to_kv_store` | base, trans, util | +| `paths.py` | 2270 | Path/nested access: `KeyTemplate`, `KeyPath`, `path_get/set/filter`, relative-path stores | `KeyTemplate`, `KeyPath`, `path_get`, `path_set`, `path_filter`, `mk_relative_path_store`, `flatten_dict`, `leaf_paths`, `add_prefix_filtering` | base, dig, explicit, naming, trans, util | +| `naming.py` | 1243 | **Older** parametrized-key system: `StrTupleDict` (str↔tuple↔dict). Overlaps `KeyTemplate` (§7) | `StrTupleDict`, `mk_store_from_path_format_store_cls` | base, errors, signatures, trans, util | +| `sources.py` | 1172 | KV views over disparate sources; composite stores | `FlatReader`, `FanoutReader`, `FanoutPersister`, `CascadedStores`, `MultiSource`, `SequenceKvReader`, `FuncReader`, `ObjReader`, `Attrs` | base, caching, signatures, trans, util | +| `filesys.py` | 867 | Filesystem stores | `Files`, `FilesReader`, `TextFiles`, `PickleFiles`, `JsonFiles`, `Jsons`, `DirReader`, `mk_dirs_if_missing`, `subfolder_stores` | base, dig, kv_codecs, naming, paths, trans, trash | +| `kv_codecs.py` | 596 | Ready-made codec namespaces | `ValueCodecs`, `KeyCodecs` (`KeyValueCodecs` exists but **unexported**) | paths, signatures, trans, util, zipfiledol | +| `signatures.py` | 5403 | `Sig` signature-calculus engine (used everywhere) | `Sig`, `KO`/`PK`/… kind aliases, `Param`, `call_forgivingly` | (none) | +| `util.py` | 2356 | Grab-bag utilities | `Pipe`, `lazyprop`, `partialclass`, `groupby`, `regroupby`, `instance_checker`, `LiteralVal`, `chain_get`, `written_bytes` | (none — self-contained) | +| `zipfiledol.py` | 994 | Zip/tar archive stores + compression codecs | `ZipFiles`, `ZipReader`, `FilesOfZip`, `FlatZipFilesReader`, `zip_compress`, `tar_compress` | base, errors, explicit, filesys, paths, sources, trans, util | +| `appendable.py` | 673 | Append semantics for stores | `appendable`, `mk_item2kv_for`, `Extender` | trans, util | +| `explicit.py` | 291 | Stores driven by explicit key data | `ExplicitKeyMap`, `KeysReader`, `invertible_maps` | base, sources, trans, util | +| `tools.py` | 585 | Misc store add-ons | `store_aggregate`, `iSliceStore`, `Forest` | base, filesys, trans | +| `misc.py` | 462 | Read/write misc sources (URLs, dropbox) | `MiscGetter`, `MiscGetterAndSetter`, `get_obj` | filesys, trans, util, zipfiledol | +| `dig.py` | 206 | Introspect wrapper layers | `trace_getitem`, `inner_most`, `layers` | trans | +| `mixins.py` | 248 | Legacy mixin classes (mostly superseded by `wrap_kvs`) | `ReadOnlyMixin`, `IdentityKvWrapMixin`, `OverWritesNotAllowedMixin` | errors, util | +| `errors.py` | 170 | Exception hierarchy | `KeyValidationError`, `OperationNotAllowed`, `WritesNotAllowed`, … | (none) | +| `trash.py` | 237 | Cross-platform trash/recycle delete | `get_platform_trash_func`, `make_safe_delete_func` | (none) | +| `recipes.py` | 5 | Thin alias module | `search_paths` (= `paths.path_filter`) | paths | + +`*` `Codec`/`ValueCodec`/`KeyCodec`/`KeyValueCodec` live in `trans.py` but are **not** +top-level `dol` exports; import from `dol.trans` (see §3, §6). + +### Internal dependency graph + +Foundation layer (no internal deps): `signatures`, `util`, `errors`, `trash`. + +```mermaid +graph TD + signatures[signatures] + util[util] + errors[errors] + trash[trash] + base --> errors & signatures & util + trans --> base & errors & signatures & util + caching --> base & trans & util + naming --> base & errors & signatures & trans & util + paths --> base & dig & explicit & naming & trans & util + dig --> trans + explicit --> base & sources & trans & util + sources --> base & caching & signatures & trans & util + kv_codecs --> paths & signatures & trans & util & zipfiledol + zipfiledol --> base & errors & explicit & filesys & paths & sources & trans & util + filesys --> base & dig & kv_codecs & naming & paths & trans & trash + appendable --> trans & util + mixins --> errors & util + tools --> base & filesys & trans + misc --> filesys & trans & util & zipfiledol + recipes --> paths +``` + +Notes on the graph: +- **`base` and `trans` are the two hubs.** Almost everything depends on `trans`; + `trans` depends only on `base` + the foundation. There is **no** circular import + between `trans` and `paths` (the `from dol.paths` lines in `trans.py` are inside + doctests only — `trans.py:2661,2793`). +- **`paths` ↔ `explicit` are mutually entangled** (`paths` imports `explicit`; + `explicit` imports `sources` which is independent) — but `paths` also imports + `naming` and `dig`, making it one of the most coupled modules. +- **`zipfiledol` and `filesys` are the heaviest leaves**, each pulling in 6–8 internal + modules. `filesys` imports `kv_codecs` which imports `zipfiledol` — a long chain, but + acyclic. + +--- + +## 3. Public API surface (`dol/__init__.py`) + +Grouped by concern. All names below are importable as `from dol import `. + +**Base classes / protocol** (`base.py`): `Collection`, `MappingViewMixin`, `KvReader`, +`KvPersister`, `Reader` (alias of `KvReader`), `Persister` (alias of `KvPersister`), +`Store`, `kv_walk`, `KT`, `VT`, `BaseKeysView`, `BaseValuesView`, `BaseItemsView`. + +**Wrappers / transforms** (`trans.py`): `wrap_kvs`, `kv_wrap`, `filt_iter`, +`cached_keys`, `cache_iter` (deprecating), `add_decoder`, `add_ipython_key_completions`, +`insert_hash_method`, `add_path_get`, `add_path_access`, `flatten`, `disable_delitem`, +`disable_setitem`, `mk_read_only`, `add_aliases`, `insert_aliases`, +`add_missing_key_handling`, `store_decorator`, `redirect_getattr_to_getitem`. + +**Codecs** (`kv_codecs.py`): `ValueCodecs`, `KeyCodecs`. + +**Caching** (`caching.py`): `cache_this`, `add_extension`, `lru_cache_method`, +`WriteBackChainMap`, `mk_cached_store` (old alias), `cache_vals`, `store_cached`, +`store_cached_with_single_key`, `ensure_clear_to_kv_store`, `flush_on_exit`, +`mk_write_cached_store`. + +**Paths / naming** (`paths.py`, `naming.py`): `flatten_dict`, `leaf_paths`, +`KeyTemplate`, `mk_relative_path_store`, `KeyPath`, `paths_getter`, `path_get`, +`path_set`, `path_filter`, `add_prefix_filtering`, `StrTupleDict`, +`mk_store_from_path_format_store_cls`. + +**Sources / composite** (`sources.py`): `FlatReader`, `SequenceKvReader`, `FuncReader`, +`Attrs`, `ObjReader`, `FanoutReader`, `FanoutPersister`, `CascadedStores`, `MultiSource`. + +**Filesystem** (`filesys.py`): `Files`, `FilesReader`, `TextFiles`, `PickleFiles`, +`JsonFiles`, `Jsons`, `DirReader`, `ensure_dir`, `mk_dirs_if_missing`, +`MakeMissingDirsStoreMixin`, `resolve_path`, `resolve_dir`, `temp_dir`, +`create_directories`, `process_path`, `subfolder_stores`. + +**Zip** (`zipfiledol.py`): `ZipReader`, `ZipInfoReader`, `FilesOfZip`, +`FileStreamsOfZip`, `FlatZipFilesReader`, `ZipFiles`, `ZipStore` (alias), +`ZipFileStreamsReader`, `zip_compress`, `zip_decompress`, `to_zip_file`, +`remove_mac_junk_from_zip`, `tar_compress`, `tar_decompress`. + +**Utils** (`util.py`): `Pipe`, `lazyprop`, `partialclass`, `groupby`, `regroupby`, +`igroupby`, `instance_checker`, `chain_get`, `non_colliding_key`, `get_app_folder`, +`get_app_config_folder`, `AttributeMapping`, `AttributeMutableMapping`, +`not_a_mac_junk_path`, `written_bytes`, `written_key`, `read_from_bytes`. + +**Other**: `store_aggregate` (`tools.py`), `mk_item2kv_for`, `appendable` +(`appendable.py`), `trace_getitem` (`dig.py`), `ExplicitKeyMap`, `invertible_maps`, +`KeysReader` (`explicit.py`). + +**Deprecation shim**: `__init__.py:197` defines a module-level `__getattr__` that maps the +removed `get_app_data_folder` → `get_app_config_folder` with a `DeprecationWarning`. + +### Export gaps to flag + +- **Documented / expected but NOT top-level exported**: `Codec`, `ValueCodec`, + `KeyCodec`, `KeyValueCodec` (must use `from dol.trans import …`). `llms.txt` lists them + under "Core API / trans.py" which reads as if they were `dol`-level. +- **Exists but unexported**: `KeyValueCodecs` namespace (`kv_codecs.py:575`) — and its two + factories `key_based`/`extension_based` are **unimplemented stubs** (§6, §11). +- **`Stream`** (`base.py:1021`) — a layer-able stream interface — is public-quality but not + exported anywhere. +- ~~**`FirstArgIsMapping`** is defined but unused~~ — **wired in as of the Issue #9/#12 fix** + (2026-07): consumed via `_resolve_self_convention` at all four call sites and exported + from `dol`. (Historical: it was dead code with `# TODO: Use this for it's intent!`.) + +--- + +## 4. Class hierarchy & hook protocol + +``` +collections.abc.Collection (ABC: __iter__, __contains__, __len__) + └─ dol.base.Collection base.py:86 + head(); default __len__/__contains__ by iteration + └─ dol.base.KvReader base.py:156 (MappingViewMixin, Collection, Mapping) + │ + __getitem__, keys/values/items; __reversed__ → NotImplementedError + └─ dol.base.KvPersister base.py:196 (KvReader, MutableMapping) + __setitem__/__delitem__; clear() DISABLED + └─ dol.base.Store base.py:469 wraps self.store; adds the 4 transform hooks +``` + +Aliases: `Reader = KvReader` (`base.py:191`), `Persister = KvPersister` (`:239`), +`KvStore = Store` (`:737`). + +**Mixins & helpers (base.py):** +- `MappingViewMixin` (`:141`) — swap `KeysView`/`ValuesView`/`ItemsView` *class attributes* + to customize views instead of overriding `.keys()/.values()/.items()`. +- `DelegatedAttribute` (`:251`) + `delegate_to` (`:291`) + `delegator_wrap` (`:366`) — + the delegation engine. `Store.wrap = classmethod(partial(delegator_wrap, + delegation_attr="store"))` (`:611`) lets any Store carry its own wrapping method. +- `KeyValidationABC` (`:978`), `Stream` (`:1021`), `kv_walk` (`:768`). + +**The hook protocol (the customization surface of `Store`):** + +| Hook (default = `static_identity_method`) | Direction | Called by | +|---|---|---| +| `_id_of_key(k) → _id` | ingoing key | `__getitem__`, `__setitem__`, `__delitem__`, `__contains__` | +| `_key_of_id(_id) → k` | outgoing key | `__iter__` | +| `_data_of_obj(obj) → data` | ingoing value | `__setitem__` | +| `_obj_of_data(data) → obj` | outgoing value | `__getitem__` | + +Defaults assigned at `base.py:600–603`. `__getitem__` (`:631`) also routes +`_errors_that_trigger_missing` (default `(KeyError,)`, `:607`) to `__missing__` if present. +`Store.__getattr__` (`:613`) delegates everything else to `self.store` using +`object.__getattribute__` to survive pickling; `__getstate__`/`__setstate__` +(`:723`/`:730`) persist only `_state_attrs = ["store", "_class_wrapper"]`. + +**Three documented ways to inject hooks** (Store docstring, `base.py:469–578`): subclass +& override; assign callables to `_id_of_key` etc. on class/instance; or use `wrap_kvs` +(preferred). Convention per `.claude/rules/dol-conventions.md`: **prefer `wrap_kvs` over +subclassing `Store`.** + +--- + +## 5. The `wrap_kvs` / `store_decorator` machinery — deep dive + +This is the heart of dol and the root of Issues #9/#12/#18. Everything below is verified +against `dol/trans.py`. + +### 5.1 `store_decorator` — the 4-way meta-decorator (`trans.py:130`) + +`store_decorator(func)` takes a class-transforming `func(store=None, *, **kw)` and returns +a `wrapper` usable **four ways**: class-decorator, class-decorator-factory, +instance-decorator, instance-decorator-factory. Mechanism: + +1. It computes an enriched signature `wrapper_sig` = `Sig(func)` merged with the + "wrapper assignment" params `__module__, __name__, __qualname__, __doc__, + __annotations__, __defaults__, __kwdefaults__` (all keyword-only, default `None`) — + `trans.py:357`. These let callers rename/re-doc the produced class. +2. `wrapper(store=None, **kwargs)` (`:363`): if `store is None` → return + `partial(_func_wrapping_store_in_cls_if_not_type, **kwargs)` (the **factory** branch); + else call it directly. +3. `_func_wrapping_store_in_cls_if_not_type` (`:324`) is where **instance vs class** is + resolved: if `store` is **not a type** (an instance), it builds `WrapperStore = + func(Store, **kwargs)` then returns `WrapperStore(store_instance)` — i.e. the instance + is first wrapped in a fresh `Store` subclass (`:337–338`). If `store` **is** a type, + it asserts all-but-first args are keyword-only (`:340`) and calls `func(store, **kwargs)`. +4. After producing `r`, it copies over any explicitly-passed dunder specials (`:346`). + +Consequence to know (documented in the docstring, `:219`): decorating an **instance** +returns an object whose type is `dol.base.Store` (or a subclass), **not** the original +type. `b.store == a` but `isinstance(b, type(a))` is `False`. + +`double_up_as_factory` (`:36`) is the simpler cousin (direct-or-factory, no +instance-wrapping) used for plain function decorators; it validates first-arg-defaults- +to-`None` and all-else-keyword-only (`:90`). + +### 5.2 `wrap_kvs` (`trans.py:1813` decorator / `:1814` def) + +Decorated with `@store_decorator`, so it inherits the full 4-way behavior. Body is short +(`:1968–1978`): default `name` to the store's qualname, assemble `kwargs`, call +`_handle_codecs(kwargs)` to normalize codec/encoder/decoder aliases into the canonical +`key_of_id`/`id_of_key`/`obj_of_data`/`data_of_obj` (`:1981`), then delegate to +`_wrap_store(_wrap_kvs, kwargs)`. + +Parameters (canonical + aliases resolved by `_handle_codecs`): +- Canonical: `key_of_id`, `id_of_key`, `obj_of_data`, `data_of_obj`, `preset`, `postget`. +- Codec shortcuts: `key_codec`/`value_codec` (a `Codec` → its `.encoder`/`.decoder` are + split into the canonical pair), and the four `*_encoder`/`*_decoder` aliases. + `_handle_codecs` raises `ValueError` if you pass both a codec and the canonical it maps + to (`:2000`, `:2007`, …). +- Advanced: `outcoming_key_methods`, `outcoming_value_methods`, `ingoing_key_methods`, + `ingoing_value_methods` — extra method names to also wrap alongside the defaults. + +### 5.3 `_wrap_kvs` — applying transforms to the class (`trans.py:2124`) + +For each of the four canonical transforms it wraps the corresponding hook method +(plus any extra `*_methods`): `_key_of_id`/`_obj_of_data` via `_wrap_outcoming` +(`:2141–2145`); `_id_of_key`/`_data_of_obj` via `_wrap_ingoing` (`:2147–2151`). +`postget`/`preset` are special-cased (`:2156–2190`): they replace `__getitem__`/ +`__setitem__` directly so the transform receives `(k, data)`. + +### 5.4 The signature-based conditioning — how `(data)` vs `(self, data)` is decided + +This was the crux of Issue #9/#12. **Status: RESOLVED (2026-07)** — the mechanism below is +the *current* (fixed) behavior; the historical bug is preserved for context. + +**(A) For the four canonical transforms** (`key_of_id`, `id_of_key`, `obj_of_data`, +`data_of_obj`), `_wrap_outcoming` and `_wrap_ingoing` branch on a single resolver +**`_resolve_self_convention(trans_func) -> (func, wants_self)`**: +- `wants_self == False` → the wrapped method calls `trans_func()`. +- `wants_self == True` → it calls `trans_func(self, )`. + +`_resolve_self_convention` (a) unwraps an explicit `FirstArgIsMapping` marker and forces +`wants_self=True`, else (b) falls back to `_has_unbound_self`. The **fixed** rule is: + +> `wants_self ⟺ first parameter name ∈ self_names = {"self","store","mapping"}` +> **AND `_num_required_positional_params(params) >= 2`**. + +The arity clause is the fix. **Before**, the decision was name-only — so a unary callable +whose first param merely *happened* to be named `self` (e.g. `bytes.decode`) was mis-called +as `trans_func(store, data)`: +``` +# OLD (buggy): _has_unbound_self(bytes.decode) -> True (first param named 'self') +wrap_kvs(dict, obj_of_data=bytes.decode)()['k'] +# → TypeError: descriptor 'decode' for 'bytes' objects doesn't apply to a 'dict' object +# NOW (fixed): bytes.decode has only 1 required positional -> treated as unary -> works. +``` +A genuine self-aware transform `def f(self, data)` (2 required) is unaffected. If +`signature()` raises `ValueError` (a signature-less builtin), `_has_unbound_self` returns +`False`. The old note that this was an "arg-count check" was wrong; it is now name **and** +arity. + +**(B) For `postget`/`preset`**, `_wrap_kvs` resolves the marker first, then does its +`num_of_args(...) < 2` arity validation on the *unwrapped* function, then branches on the +resolved `wants_self`. + +**The explicit escape hatch — now wired in:** `FirstArgIsMapping(f)` (a `LiteralVal` +subclass in `trans.py`) forces the `(self, data)` convention regardless of names/arity. +It was dead code (`# TODO: Use this for it's intent!`); the fix consumes it via +`_resolve_self_convention` at all four call sites and exports it from `dol`. This realizes +Issue #12's proposed solution. + +**Backward-compat evidence:** an AST scan of dol + all 76 dependents found **0** +behavior-changing call sites (12 genuine self-convention transforms, all ≥2 required, are +preserved); a recall-gap scan of attribute/imported-name transforms across 13 heavy users +found 0 more; and a baseline-vs-modified dependents test-gate across 25 repos showed 0 +pass→fail regressions. See `dol_issues_report.md` and the `dol-dev-wrap-kvs` skill. + +**Remaining redesign note:** the name-based fallback is kept for compatibility; the clean +long-term direction is explicit-marker-only (drop `self_names`), a future breaking change. + +### 5.5 Related: Issue #18 ("`self` is unwrapped") + +`_wrap_outcoming`/`_wrap_ingoing` wrap **methods on the generated class** via +`super(store_cls, self).(...)`. When a method *inside* a `wrap_kvs`-decorated +class calls `self[k]`, `self` is the inner (unwrapped) instance, so the transform pipeline +is bypassed. Workaround (per CLAUDE.md gotchas): re-apply the wrapper to `self` inside the +method, or route through the wrapped instance explicitly. + +### 5.6 `kv_wrap` — the alternate interface (`trans.py:2395`) + +`kv_wrap(trans_obj)` builds a wrapper from an object carrying `_key_of_id`/`_id_of_key`/ +`_obj_of_data`/`_data_of_obj` attributes (a "trans object"), with chainable +`.outcoming_keys(...)`, `.ingoing_keys(...)`, `.outcoming_vals(...)`, `.ingoing_vals(...)` +attributes (`_kv_wrap_*` at `:2198–2348`). Same underlying `_wrap_outcoming`/`_wrap_ingoing`. + +--- + +## 6. Codec system + +Defined at the **end of `trans.py`** (`:3365–3451`); ready-made namespaces in +`kv_codecs.py`. + +**Dataclasses** (`trans.py:3374`): +```python +@dataclass +class Codec(Generic[DecodedType, EncodedType]): + encoder: Callable # decoded → encoded (the "write"/ingoing side) + decoder: Callable # encoded → decoded (the "read"/outgoing side) + __iter__ = (encoder, decoder) # so `enc, dec = codec` works + compose_with(other) → Pipe both sides # __add__ (:3382) + invert() → swap encoder/decoder # __invert__ (:3389) +``` +Subclasses are **callable store-wrappers** (each just calls `wrap_kvs`): +- `ValueCodec(obj)` → `wrap_kvs(obj, data_of_obj=encoder, obj_of_data=decoder)` (`:3403`) +- `KeyCodec(obj)` → `wrap_kvs(obj, id_of_key=encoder, key_of_id=decoder)` (`:3408`) +- `KeyValueCodec(obj)` → `wrap_kvs(obj, preset=encoder, postget=decoder)` (`:3413`) + +**Composition**: `codec_a + codec_b` composes encoders left→right and decoders right→left +(`compose_with`, `:3382`) — order matters and is inverse on the two sides so round-trips +hold. `Pipe` (`util.py:638`) is the underlying left-to-right function composition; +`ValueCodecs.pickle() + ValueCodecs.gzip()` means "pickle then gzip on write, gunzip then +unpickle on read". Codecs can also be threaded through `Pipe(...)` as store wrappers. + +**`kv_codecs.py` namespaces** (all classes subclass `CodecCollection`, which is +**non-instantiable** — `:227`; `.default` sub-namespace holds pre-called default codecs, +populated by `@_add_default_codecs`, `:254`): +- `ValueCodecs` (`:263`): `pickle`, `json`, `csv`, `csv_dict`, `gzip`, `bz2`, `lzma`, + `zipfile`, `tarfile`, `base64`, `urlsafe_b64`, `codecs`, `quopri`, `plistlib`, + `xml_etree`, `str_to_bytes`, `stringio`, `bytesio`, `single_nested_value`, + `tuple_of_dict`. Each is built by `value_wrap` (`= codec_wrap(ValueCodec, …)`, `:222`) + which uses `Sig` to give the factory a signature merged from encoder+decoder params. +- `KeyCodecs` (`:427`): `affixed`, `suffixed`, `prefixed`, `common_prefixed`, + `mapped_keys`. Affix codecs go through `affix_key_codec` (`trans.py:3439`). +- `KeyValueCodecs` (`:575`, **unexported**): `key_based`, `extension_based` — **both are + empty stubs** (`extension_based`'s `ext_mapping` is unused; `:596–597`). Dead/incomplete. + +The CSV codecs use the "**signature-template**" pattern: `@Sig`-decorated no-op functions +(`_csv_rw_sig` etc., `:36`) exist only so their parameter lists can be composed onto the +real encode/decode functions. Vulture flags their params as unused — this is by design +(see `code-quality-improvements.md`). + +--- + +## 7. Paths & naming subsystem — the duplication is real (Discussion #21) + +There are **two overlapping parametrized-key systems**, and Discussion #21's concern is +**still valid**: + +| Capability | `paths.KeyTemplate` (`paths.py:1658`) | `naming.StrTupleDict` (`naming.py:432`) | +|---|---|---| +| Template with named fields + per-field regex | yes (`field_patterns`) | yes (`format_dict`) | +| `str_to_dict` / `dict_to_str` | yes | yes | +| `str_to_tuple` / `tuple_to_str` | yes | yes | +| `str_to_namedtuple` / `str_to_simple_str` | yes | partial | +| per-field `from_str`/`to_str` casts | yes (`from_str_funcs`/`to_str_funcs`) | yes (`process_info_dict`) | +| `.key_codec(src, tgt)` → a `KeyCodec` store-wrapper | **yes** | no | +| `.filt_iter` / `.clone` conveniences | **yes** | no | +| Age / status | newer, richer, actively used | older; only `StrTupleDict` + `mk_store_from_path_format_store_cls` exported; **31% test coverage** (lowest in the package) | + +**Verdict**: `KeyTemplate` supersedes `StrTupleDict` for essentially all new use. `naming.py` +is a redesign-shrink candidate (deprecate `StrTupleDict` → thin adapter over `KeyTemplate`, +or move its still-used bits — `mk_pattern_from_template_and_format_dict`, used by +`filesys.py:11` — into `paths.py`). + +**Other path machinery in `paths.py`**: +- `KeyPath` (`:956`) — a path object with a configurable separator; usable as a + `key_of_id`/`id_of_key`. +- `path_get` (`:405`) / `path_set` (`:712`) / `path_filter` (`:836`) — nested get/set/ + search. Note there is **also** a private `_path_get` (`:216`) and `chain_get` + (`util.py:298`) doing overlapping "walk a path of getitems" work — three + implementations of the same idea, exactly what Discussion #21 flagged. +- `mk_relative_path_store` (`:1139`), `PrefixRelativizationMixin` (`:1036`), + `add_prefix_filtering` (`:1336`), `prefixless_view` (`:1270`) — the relative/prefix + family. +- `flatten_dict` (`:99`, via `flattened_dict_items`) and `leaf_paths` (`:134`, via + `_leaf_paths_recursive`) — nested→flat. Note these use their **own** recursion, not + `kv_walk`; only `path_filter`/`search_paths` are built on `kv_walk` — another instance + of the same-idea-implemented-N-times debt. + +Also note `trans.flatten` (`trans.py:2867`) flattens a *store of stores* (levels-aware), +while `paths.flatten_dict` flattens a plain nested `Mapping` — two "flatten"s with +different semantics; a naming-collision trap for agents. + +--- + +## 8. Caching subsystem (`caching.py`) + +The subsystem is far larger than the existing docs suggest (2675 lines). Key pieces: + +**`cache_this` (`:1055`)** — unified property/method cache. Signature: +`cache_this(func=None, *, cache=None, key=None, pre_cache=False, as_property=None, +ignore=None, serialize=None, deserialize=None)`. Auto-detects property vs method from the +signature unless `as_property` forces it. `cache` may be a `MutableMapping`, an attribute +**name** (string) resolved on the instance, or a **callable `self → MutableMapping`** +(enables per-instance persistent caches, e.g. `cache=lambda self: Files(f'/cache/{self.id}')`). +Backed by: +- **`KeyStrategy` protocol** (`:126`) + implementations `ExplicitKey` (`:165`), + `ApplyToMethodName` (`:188`), `InstanceProp` (`:211`), `ApplyToInstance` (`:233`), + `FromMethodArgs` (`:255`), `CompositeKey` (`:284`), registered via + `@register_key_strategy` (`:158`). This is the pluggable key-generation layer that + decides *what key* a cached value is stored under. **None of this is in the existing docs.** +- `CachedProperty` (`:464`) and `CachedMethod` (`:807`) — the two descriptor classes + `cache_this` dispatches to. + +**Store-level caching**: +- `cache_vals` (`:1902`, alias `mk_cached_store`) — read-through value cache in front of a + slow store. +- `mk_sourced_store` (`:2000`) — cache-aside with a `source` fallback. +- `store_cached(store, key_func)` (`:2161`) / `store_cached_with_single_key` (`:2223`) — + function memoization into an arbitrary store. +- `mk_write_cached_store` (`:2386`) + `flush_on_exit` (`:2355`) — buffered writes flushed + on context exit. +- `WriteBackChainMap` (`:2528`) — `ChainMap` where writes hit the first map and reads fall + through. +- `ensure_clear_to_kv_store` (`:2293`) — re-enable the disabled `clear()`. + +**Stacking behavior (Issue #50)**: `cache_this` decorators are descriptor-based and key on +the method name / arg-derived key; stacking multiple `@cache_this` on one method is **not** +robustly supported (key conflicts / invalidation). `lru_cache_method` (`:1596`) and +`cached_method` (`:1541`) are the lighter in-memory alternatives. For agents: prefer a +single `cache_this` with an explicit `key`/`cache` over stacking. + +--- + +## 9. Patterns catalog (idioms an agent/dev must know) + +| Pattern | Mini-example | Why | +|---|---|---| +| **`X_of_Y` naming** | `id_of_key(k)→_id`, `key_of_id(_id)→k` | Directionality is explicit; always paired. Outer=`key`/`obj`, inner=`_id`/`data`. | +| **Test with `dict`, swap backend** | `S = wrap_kvs(dict, …); …; S = wrap_kvs(Files('/d'), …)` | Same transforms, real backend. Core testing convention. | +| **4-way `store_decorator` usage** | `@filt_iter(filt=f)` on a class / `filt_iter(inst, filt=f)` / `filt_iter(filt=f)` factory | One decorator, class-or-instance, with-or-without params. | +| **`Pipe` composition** | `Pipe(json.dumps, str.encode, gzip.compress)` | Left-to-right function chain (`util.py:638`). | +| **Codec composition** | `ValueCodecs.pickle() + ValueCodecs.gzip()` | Encoders compose L→R, decoders R→L; round-trip safe. | +| **Stack `wrap_kvs` (Russian dolls)** | key layer, then value layer, then filter layer | Each layer independent/composable. | +| **`postget`/`preset` for key-conditioned values** | `postget=lambda k,v: json.loads(v) if k.endswith('.json') else v` | Use only when the value transform depends on the key. | +| **Re-wrap `self` inside methods** | inside a `wrap_kvs`-class method, wrap `self` again before `self[k]` | Works around Issue #18 (self is unwrapped). | +| **Avoid `bytes.decode` as a transform** | use `lambda b: b.decode()` | `bytes.decode`'s first param is named `self` → misclassified (§5.4, Issue #9). | +| **Customize views via class attr** | `MyStore.KeysView = MyKeysView` | Don't override `.keys()`; override the view class (`MappingViewMixin`). | +| **Re-enable `clear()`** | `ensure_clear_to_kv_store(store)` | `clear()` is disabled on `KvPersister`. | + +--- + +## 10. Extension points & hooks + +- **New store — preferred**: `wrap_kvs(backend, …)` (class or instance). Test with `dict` + first. (`.claude/rules/dol-conventions.md`.) +- **New store — read-only**: subclass `KvReader` implementing `__getitem__`, `__iter__` + (optionally `__len__`, `__contains__`). Read-write: subclass `KvPersister` adding + `__setitem__`, `__delitem__`. +- **New store — with hooks**: subclass `Store`, override `_id_of_key`/`_key_of_id`/ + `_data_of_obj`/`_obj_of_data`. (Discouraged vs `wrap_kvs` unless the hook protocol is + genuinely needed.) +- **New codec**: build a `ValueCodec`/`KeyCodec`/`KeyValueCodec(encoder=…, decoder=…)`, or + add a factory to a `CodecCollection` namespace using `value_wrap`/`key_wrap` + (`kv_codecs.py:222`). Compose with `+`. +- **New store decorator**: write `func(store=None, *, **kw)` and wrap with + `@store_decorator` for free 4-way behavior. +- **New key-cache strategy**: implement the `KeyStrategy` protocol and + `@register_key_strategy` (`caching.py:126,158`). +- **"Hooks for optimized ops" (Discussion #24) — NOT yet implemented.** The intended + design: dol tooling (`filt_iter`, `update`, sort/paginate) would check for a + backend-provided fast method (e.g. `_filter_`, a fast-`sync` protocol) before falling + back to Python-level iteration — analogous to how `__len__` optimizes `len()`. The plug + point today would be in `_filt_iter` (`trans.py:1403`) and `Store.update`/`__iter__`. + This is greenfield for the redesign. + +--- + +## 11. Tech-debt & redesign opportunities (ranked by leverage) + +1. **Signature-conditioning fragility (§5.4).** The name-based `(data)` vs `(self, data)` + heuristic (`self_names`, `_has_unbound_self`) is the single highest-leverage fix. Wire + in the already-defined `FirstArgIsMapping` marker (`trans.py:2113`) as the explicit + opt-in and make the name heuristic a deprecated fallback. Fixes Issue #9/#12, removes a + class of silent bugs, and de-risks every `wrap_kvs` call. *High impact, moderate effort.* +2. **`trans.py` is a 3492-line kitchen sink.** It holds the core wrappers, filtering, + codecs, path-access adders, alias inserters, missing-key handlers, and hashing. Split + into `trans_core` (wrap machinery), `codecs` (move `Codec`/`ValueCodec`/… out — they're + logically `kv_codecs`'), and `store_addons` (path-get/access, aliases, hash). *High + impact on maintainability + testability.* +3. **`naming.py` ↔ `paths.py` duplication (§7).** Deprecate `StrTupleDict` in favor of + `KeyTemplate`; consolidate the three path-get impls (`path_get`, `_path_get`, + `chain_get`). `naming.py` has the lowest coverage in the package (31%). *Medium impact, + removes ~1200 lines of parallel logic.* +4. **`signatures.py` (5403 lines) and `caching.py` (2675) module-size hotspots.** `Sig` is + load-bearing everywhere and a hard dependency to test around; consider carving the + rarely-used arithmetic out. `caching.py` bundles the key-strategy protocol, two + descriptor classes, and ~10 store-cache helpers — extractable. *Medium.* +5. **Dead / incomplete code** (confirmed present): + - `KeyValueCodecs.key_based` / `.extension_based` — empty stubs (`kv_codecs.py:580,589`). + - `FirstArgIsMapping` — defined, never used (`trans.py:2113`). + - `util.delegate_as` — raises `NotImplementedError` with unreachable code after + (`util.py:1598`). + - `stream_util.skip_lines` — `n_lines_to_skip` param unused (`base.py:1016`). + - `zipfiledol.py:267` `pass` after `raise`; unused `disable_deletes` (`trans.py:551`). + - Duplicate `identity`/`identity_func` defined 3× (`caching.py:87,121`; `sources.py:28`; + `util.py:159`) and `HashableMixin`/`HashableDict` duplicated across + `util.py`/`caching.py`/`mixins.py`. + *Low-effort cleanup; some already tracked in `code-quality-improvements.md`.* +6. **LSP violation: disabled `clear()`** on a declared `MutableMapping` (`base.py:224`). + `dict(store)` / anything calling `.clear()` breaks. Consider a read-only marker type in + the hierarchy rather than method-nulling. *Design-level; breaking.* +7. **No async, no generics, ABC-not-Protocol** — the three additive modernizations from + `dol_design.md`'s critique (§Design Critique) remain open and non-breaking if done as + additions. + +--- + +## 12. Notes for dev-skill authors (top things when modifying dol source) + +1. **Tests live in `dol/tests/`** (`test_trans.py`, `test_caching.py`, `test_paths.py`, + `test_filesys.py`, …). Run `pytest dol/tests/`. **Doctests are first-class** — most + functions document via runnable doctests; `pytest --doctest-modules dol/` matters. Never + break a doctest silently. +2. **The `(self, data)` trap (§5.4).** Any transform whose first parameter is named + `self`, `store`, or `mapping` will be called as `f(store_instance, value)`. This is the + most common way to accidentally break `wrap_kvs`. When writing or reviewing transforms, + check the first parameter name. Prefer `lambda x: …` over bound/unbound builtins. +3. **The "self is unwrapped" trap (§5.5, Issue #18).** Inside a `wrap_kvs`-decorated class, + `self[k]` bypasses the transforms. Don't assume in-class `self[...]` is transformed. +4. **Decorating an instance changes its type to `Store`** (§5.1). `wrap_kvs(some_dict, …)` + returns a `Store`, not a `dict`; `isinstance` checks and `type().__name__` will surprise + you. The original is at `.store`. +5. **`store_decorator` requires all-but-first args keyword-only** and first arg defaulting + to `None`. Violating this asserts at decoration/first-call time (`trans.py:340`, `:92`). + Follow the same rule for new decorators (matches CLAUDE.md's keyword-only convention). +6. **`Codec`/`ValueCodec`/`KeyCodec` are in `trans.py`, not exported at top level.** Import + from `dol.trans`. Ready-made ones: `dol.ValueCodecs` / `dol.KeyCodecs`. +7. **Keep core dependency-free.** `dol` core imports nothing outside stdlib. New optional + deps go behind lazy imports with a helpful error (see how `kv_codecs` lazily imports + `pickle`/`gzip`/`plistlib` inside the class body, `:315,356`). +8. **`Sig` (`signatures.py`) is everywhere** — `store_decorator`, codec factories, and + `wrap_kvs` all rewrite signatures with it. If a wrapper's signature looks wrong, suspect + `Sig` merge logic. Don't read all 5403 lines; use `Sig(func).names/.defaults/.kinds`. +9. **Two different "flatten"s and three "path get"s** (§7). Disambiguate before touching: + `trans.flatten` (store-of-stores) vs `paths.flatten_dict` (nested Mapping); + `path_get`/`_path_get`/`chain_get`. +10. **Every module must keep its top-level docstring** (all 20 currently have one — verified; + they are auto-extracted for docs). When adding a module or editing one, preserve/enhance + it (per CLAUDE.md). And update `dol_misc_docs_guide.md` if you add a doc here. + +--- + +*Generated as a structural companion to the `misc/docs/` design set. Cross-links above are +intra-repo. Line numbers are current as of the verification date; re-check after any large +`trans.py`/`caching.py` refactor.* diff --git a/misc/docs/dol_issues_report.md b/misc/docs/dol_issues_report.md new file mode 100644 index 00000000..9227308a --- /dev/null +++ b/misc/docs/dol_issues_report.md @@ -0,0 +1,162 @@ +# dol — Issues Triage & Tackle-Order Report + +> **Purpose:** a scannable, prioritized map of dol's open GitHub issues — what is +> already resolved (and should be closed), and in what order to tackle the rest. +> Companion to [issues_and_discussions.md](issues_and_discussions.md) (themes/history), +> [dol_architecture_map.md](dol_architecture_map.md) (code-verified mechanics), and the +> local-only ecosystem inventory (`misc/data/`, gitignored — dependents & the pre-PR +> test gate). +> +> Prepared 2026-07-02 against dol `0.3.46` / HEAD `4a995b8`. Every "verified" verdict +> below was reproduced or refuted by running code, not inferred from titles. + +--- + +## TL;DR + +- **Close now (verified resolved): #40, #52, #58.** Three open issues were already fixed by + merged PRs but never closed. (A fourth, #50, was already closed 2025-10-10.) Closing these + leaves **14 legacy issues** to work through — plus #67–#70 filed by this study. +- **The single highest-leverage cluster is `wrap_kvs`'s signature conditioning: #9 → #12 + → #18 (+ #5, #6).** One root cause (guessing a transform's calling convention from its + first *parameter name*) spawns a whole bug class. Fixing it is a **breaking change** and + `wrap_kvs` is used by **32 of 76** ecosystem dependents, so it must go through the + dependents test-gate. This is where redesign effort should concentrate first. +- Everything else is correctness fragilities (**#14, #3**), design/feature enhancements + (**#10, #16, #56, #2**), and small wins (**#13, #15, #1**). + +--- + +## Update (2026-07): Wave 1 progress + +- **#9 + #12 — FIXED** (branch `study/dol-tightening`). `_has_unbound_self` now requires + first-param-name ∈ `self_names` **AND** ≥2 required positional params, so unary builtins + like `bytes.decode` are correctly treated as `f(data)`; the previously-dead + `FirstArgIsMapping` marker is wired in via `_resolve_self_convention` (all 4 call sites) + and exported. Validated backward-compatible by AST scan (0 breaking sites across dol + 76 + dependents), recall-gap scan (0 more), a baseline-vs-modified dependents test-gate (0 + pass→fail across 25 repos), and a 6-lens adversarial review (CLEAN). Regression tests + + doctests added. See `dol_architecture_map.md` §5.4 and the `dol-dev-wrap-kvs` skill. +- **#18, #6 — assessed, still open.** Confirmed to be *distinct* mechanisms from #9 (the + #9 fix does not touch them): #18 is the delegation/has-a architecture (`self` inside a + wrapped class's methods is the inner store); #6 is `Store.wrap` setting an inherited + `__signature__`. Both need separate, more invasive fixes — deferred. + +--- + +## 1. Reconcile first — close/confirm these (verified resolved) + +| # | Title | Evidence | Action | +|---|-------|----------|--------| +| **#40** | `re.error: incomplete escape \U` | Root-cause fixed: `naming.py:265–291` now `re.escape`s literal template text and explicitly handles Windows `C:\Users\…` backslash paths (the exact traceback site). Fixed via PR #64/#65. | **Close** (fixed) | +| **#50** | Stacking `cache_this` decorators | Implemented in `caching.py` (stacking-aware `__set_name__` propagation, `:550`, `:692`) + 4 regression tests (`test_caching.py::…test_basic_stacking/test_triple_stacking/…`). Merged PR #57. | Already closed 2025-10-10 — listed for completeness | +| **#52** | Make dol tests windows compatible | Duplicate of #58; superseded by the cross-platform PRs #64/#65. | **Close** (duplicate of #58) | +| **#58** | Make dol tests work on windows | **Windows CI re-run 2026-07-02 → green** (run `28614299483`). 28 failures → 0. Code fixes in `naming.py`, `filesys.py`, `filt_iter`/`filter_regex`. | **Close** (resolved) — see caveat → new issue for auto-trigger | + +> **Caveat carried forward:** `.github/workflows/windows_ci.yml` is `workflow_dispatch` +> (manual) only, so nothing guards against Windows regressions on future PRs. Captured as a +> **new issue** (see §4) rather than left implicit in #58. + +--- + +## 2. Tackle order (the 14 that remain) + +Waves are ordered by leverage. Within a wave, do items top-to-bottom. + +### Wave 1 — The `wrap_kvs` core knot ⟵ *start here; unblocks the redesign* + +This is one architectural problem wearing five hats. See +[dol_architecture_map.md §5.4](dol_architecture_map.md) for the mechanics. + +| # | P | Type | Role in the cluster | +|---|---|------|---------------------| +| **#12** | high | enhancement / refactor / breaking | **The fix vehicle.** Proposes an explicit `FirstArgIsMapping` marker so a transform opts *in* to the `(self, data)` convention instead of dol guessing. **The class already exists but is unused** (`trans.py:2113`, `class FirstArgIsMapping(LiteralVal)`, tagged `# TODO: Use this for it's intent!`). Decide the mechanism here. | +| **#9** | high | bug / breaking | **The flagship symptom.** `wrap_kvs(store, obj_of_data=bytes.decode)` fails while `…=lambda x: x.decode()` works. **Verified root cause:** dol decides `(data)` vs `(self, data)` by whether the transform's *first parameter name* ∈ `{"self","store","mapping"}` (`trans.py:1617`, `_has_unbound_self` `:424`, `_first_param_is_an_instance_param` `:419`). `bytes.decode`'s first param is literally named `self`, so it misfires. **Not** an arg-count check (older docs said so — corrected). Fix by adopting #12. | +| **#18** | high | bug | Same root cause, different face: a `wrap_kvs`-decorated class calling `self[k]` internally hits the *unwrapped* instance (transforms bypassed). **Verified reproducible.** Either fold into the #12 fix or standardize the "re-wrap self" pattern + document. | +| **#6** | medium | bug / refactor | `Store.wrap` freezes a subclass's `__init__` signature to the base class's (`signature(B) == signature(A)`, dropping B's own params). **Verified reproducible.** Sibling of the same signature-machinery family (`signatures.py` + `Store.wrap`). | +| **#5** | medium | enhancement / refactor | "Better way to control the wrapper class" — the `wrapper=` argument threaded through every `Store.wrap` caller; proposal to use a hook on the object instead. Do alongside #6. | + +**Why first:** `wrap_kvs` (32 pkgs), `Store` (12), `KvReader` (22) top the dependent-usage +table. Any change here is the biggest blast radius in the library → must be paired with the +pre-PR dependents test-gate and a deprecation path. Doing it first means later work builds +on a sound core. + +### Wave 2 — Correctness fragilities + +| # | P | Type | Note | +|---|---|------|------| +| **#14** | medium | bug | `mk_dirs_if_missing` fragilities: (a) only makes dirs on *write*, so `list()`/`get` on a fresh store raise FS errors instead of empty/`KeyError`; (b) `mk_dirs_if_missing(wrap_kvs(G))` breaks (`FileNotFoundError: ''`) while `wrap_kvs(mk_dirs_if_missing(G))` works — an **ordering fragility tied to Wave 1**. Sequence after #9/#12. | +| **#3** | low | bug | `FileBytesReader(gettempdir())` — **still fails, but the failure has evolved**: now a `PermissionError` walking macOS-protected temp subdirs (`…/com.apple.appleaccountd/TemporaryItems/`). Reframe as *recursive-walk should skip or gracefully handle unreadable dirs*. Small, self-contained. | + +### Wave 3 — Design / feature enhancements + +| # | P | Type | Note | +|---|---|------|------| +| **#10** | medium | enhancement | Recursively applying wrappers to nested stores (`conditional_data_trans`); the "store of stores" DNA-propagation problem. Overlaps #2 (both want `kv_walk`-driven traversal). | +| **#16** | medium | enhancement / paths | `KeyPath` write-through / autovivification (`s[1][2][3] = v` when intermediate keys don't exist). Needs contextual per-level factories. | +| **#56** | low | enhancement | Fast `update`/sync between heterogeneous stores (avoid item-by-item copy). Needs a duck-typed "fast-sync" protocol negotiation; clean design open. | +| **#2** | medium | documentation / enhancement | `kv_walk`: docs, tests, and a `store_decorator` that surfaces walk-paths as a flat `KvReader`/`KvPersister`. Pairs with #10. | + +### Wave 4 — Small wins & docs + +| # | P | Type | Note | +|---|---|------|------| +| **#13** | low | enhancement / good-first-issue | `confirm_overwrite` preset for `wrap_kvs` (prompt before overwriting a differing value). Self-contained; good onboarding task. | +| **#15** | low | enhancement | `AttrContainer` tab-completion in PyCharm (works in Jupyter). Niche IDE-specific; needs a `__dir__`/dunder investigation. | +| **#1** | low | documentation | Long-running doc wishlist (`partial`-based custom wrappers; verify postget/obj_of_data order). Largely absorbed by `llms.txt`/`CLAUDE.md`/`misc/docs`; prune to what's still missing. | + +--- + +## 3. The dependency map between issues + +``` +#12 (marker mechanism) ──┬──▶ #9 (bytes.decode etc.) + ├──▶ #18 (self not wrapped) + └──▶ #14 (mk_dirs ordering fragility, partial) +#6 (subclass signature) ──── #5 (wrapper control) [Store.wrap family] +#10 (recursive wrap) ─────── #2 (kv_walk tooling) [nested traversal] +#16, #56 standalone +``` +Do **#12 before #9/#18**; do **#9/#18 before #14**; **#2 and #10 together**; **#5 with #6**. + +--- + +## 4. New issues to file (from this study + the architecture pass) + +Grounded in the code audit — see [dol_architecture_map.md §11](dol_architecture_map.md). +Filed 2026-07-02 as part of this study: + +- **[#67](https://github.com/i2mint/dol/issues/67)** — Windows CI is manual-only + not + regression-guarded; make `windows_ci.yml` run on push/PR (or add a Windows leg to + `ci.yml`). Carries #58's residual caveat. +- **[#68](https://github.com/i2mint/dol/issues/68)** — `KeyValueCodecs.key_based` / + `.extension_based` are empty stubs (`kv_codecs.py:580,589`) that silently return `None`; + the namespace is also unexported — implement or remove. +- **[#69](https://github.com/i2mint/dol/issues/69)** — consolidate the three path-get + implementations (`path_get`, `_path_get`, `chain_get`) and the two unrelated `flatten`s + (`trans.flatten` vs `paths.flatten_dict`) — a concrete slice of Discussion #21. +- **[#70](https://github.com/i2mint/dol/issues/70)** — refactor: split oversized modules + (`trans.py` 3492 LOC, `signatures.py` 5403, `caching.py` 2675) and dedup triplicated + helpers (`identity`/`identity_func` ×3; `HashableMixin`/`HashableDict`). + +> **`FirstArgIsMapping` is dead code** (`trans.py:2113`, tagged `# TODO: Use this for it's +> intent!`) — *not* filed separately; it is the concrete implementation vehicle for **#12** +> and is noted in #12's thread to keep the design decision and its implementation together. + +--- + +## 5. Reference: verification log + +| Issue | How verified | Result | +|-------|--------------|--------| +| #40 | Read `naming.py:265–291`; re-ran the escape logic | Fixed | +| #50 | Read `caching.py` stacking code + `test_caching.py` | Fixed (4 tests) | +| #52/#58 | Triggered `windows_ci.yml` on master | Green (run 28614299483) | +| #9 | Ran `wrap_kvs(d, obj_of_data=bytes.decode)` | Fails (TypeError) — reproduced | +| #6 | Ran the `Store.wrap` subclass-signature snippet | Reproduced | +| #3 | Ran `FileBytesReader(gettempdir())` | Fails (PermissionError, evolved) | +| #18 | Documented in CLAUDE.md; mechanism confirmed in `trans.py` | Reproducible | + +Backend usage weights (why blast radius matters) come from the local ecosystem scan: +`wrap_kvs` 32 pkgs · `Files` 24 · `KvReader` 22 · `Pipe` 18 · `filt_iter` 15 · `Store` 12. +Full inventory: `misc/data/dol_ecosystem_dependents.md` (gitignored). diff --git a/misc/docs/dol_misc_docs_guide.md b/misc/docs/dol_misc_docs_guide.md index f3f1d66d..8fe3e5f9 100644 --- a/misc/docs/dol_misc_docs_guide.md +++ b/misc/docs/dol_misc_docs_guide.md @@ -4,9 +4,19 @@ These documents describe **dol**, a Python library for building uniform `dict`-l - **`general_design.md`** (~220 lines) — Language-agnostic architecture of dol. Covers the key insight (language-native KV interfaces), the interface hierarchy (`Collection → KvReader → KvPersister → Store`), the middleware principle (dol sits between domain code and storage), and the KV transform pipeline (`key_of_id`, `id_of_key`, `obj_of_data`, `data_of_obj`). -- **`dol_design.md`** (~540 lines) — Python-specific implementation details. Covers the class hierarchy (rooted in `collections.abc`), the `Store` class with its 4 transform hooks, `wrap_kvs` (the core transformation function), key/value codecs (`ValueCodecs`, `KeyCodecs`), path-based stores, caching (`cache_this`), and the `Pipe` composition utility. The reference implementation doc. +- **`dol_design.md`** (~540 lines) — Python-specific implementation details. Covers the class hierarchy (rooted in `collections.abc`), the `Store` class with its 4 transform hooks, `wrap_kvs` (the core transformation function), key/value codecs (`ValueCodecs`, `KeyCodecs`), path-based stores, caching (`cache_this`), and the `Pipe` composition utility. The reference implementation doc. *(Some line-cites are stale; see `dol_architecture_map.md` for code-verified numbers.)* -- **`issues_and_discussions.md`** (~260 lines) — Themes from dol's GitHub issues/discussions. Major topics: `wrap_kvs` design tensions (signature-based conditioning, `self` not being the wrapped instance, recursive wrapping), the builtin codec ecosystem, path key handling, store composition patterns, and API ergonomics debates. +- **`dol_architecture_map.md`** (~600 lines) — Code-verified structural/mechanical map of the current source: per-module table + internal dependency graph, the exact public API surface, the class hierarchy, and a deep dive on the `wrap_kvs`/`store_decorator`/codec machinery (including the precise signature-conditioning logic behind Issue #9). Ends with a ranked tech-debt list and "notes for dev-skill authors." **Start here when refactoring or building agent tooling on dol.** + +- **`issues_and_discussions.md`** (~270 lines) — Themes from dol's GitHub issues/discussions. Major topics: `wrap_kvs` design tensions (signature-based conditioning, `self` not being the wrapped instance, recursive wrapping), the builtin codec ecosystem, path key handling, store composition patterns, and API ergonomics debates. Kept roughly current (resolved issues flagged). + +- **`dol_issues_report.md`** (~150 lines) — Actionable triage: which open issues are already resolved (close them) and a wave-by-wave **tackle order** for the rest, with an inter-issue dependency graph. The "what to work on next" companion to `issues_and_discussions.md`. + +- **`dol_content_metadata_bifurcation.md`** (~450 lines) — Design study of the content/metadata split-store problem (a store whose values carry both payload and metadata). - **`code-quality-improvements.md`** (~230 lines) — Technical debt tracker for dol: dead code, unused parameters, incomplete implementations, test coverage gaps. Operational/maintenance reference. +> A **local-only** ecosystem inventory lives under `misc/data/` (gitignored: it names +> private dependents). It maps dol's 76 local dependents, their dol usages (file:line), +> and a pre-PR test-gate order — regenerate with `misc/data/scan_dol_usages.py`. + diff --git a/misc/docs/issues_and_discussions.md b/misc/docs/issues_and_discussions.md index d72375fe..6c9024fe 100644 --- a/misc/docs/issues_and_discussions.md +++ b/misc/docs/issues_and_discussions.md @@ -2,7 +2,17 @@ This document summarizes the major themes from GitHub issues and discussions in the [i2mint/dol](https://github.com/i2mint/dol) repository. The emphasis is on **design and architecture** themes, since many issues are dev/design discussions rather than bug reports. -Sources: GitHub issues (as of early 2026) and GitHub discussions. +Sources: GitHub issues and discussions. + +> **Updated 2026-07-02.** For an actionable, prioritized *tackle-order* view (which +> issues to close, and in what order to fix the rest), see the companion +> [dol_issues_report.md](dol_issues_report.md). For code-verified mechanics of the +> `wrap_kvs` machinery referenced throughout, see +> [dol_architecture_map.md](dol_architecture_map.md). +> +> **Resolved & now closed:** #40 (Windows regex escape — fixed in `naming.py`), #52 & #58 +> (Windows tests — CI green as of 2026-07-02). Already-closed and worth noting as done: +> #50 (stacking `cache_this` — merged PR #57). --- @@ -20,7 +30,16 @@ wrap_kvs(store, obj_of_data=lambda x: bytes.decode(x)) # works wrap_kvs(store, obj_of_data=bytes.decode) # fails! ``` -**Root cause**: The code checks whether `obj_of_data` has 1 or 2+ required args, and applies it as `obj_of_data(data)` or `obj_of_data(self, data)` accordingly. This "Postelization" (being liberal in what you accept) leads to bugs. +**Root cause** (code-verified 2026-07): dol decides whether to call the transform as +`obj_of_data(data)` or `obj_of_data(self, data)` by inspecting the transform's **first +parameter *name***, not its argument *count*. If that name is in +`self_names = frozenset(["self", "store", "mapping"])` (`trans.py:1617`; logic in +`_first_param_is_an_instance_param` `:419` and `_has_unbound_self` `:424`), dol assumes the +function wants the store as its first argument. `bytes.decode`'s first parameter is literally +named `self`, so it misfires — dol calls `bytes.decode(store, data)` and gets +*"descriptor 'decode' for 'bytes' objects doesn't apply to a 'Store' object"*. This +"Postelization" (being liberal in what you accept) leads to bugs. *(An earlier version of +this doc mis-stated the root cause as an arg-count check — corrected here.)* **Discussion #34** ("Clean way of Postelizing callbacks") proposes a more principled solution: use an explicit marker (e.g., a `Literal` type or wrapper class) to signal "this function takes `self`", instead of inferring it from the signature. @@ -224,12 +243,16 @@ Proposals for improving AI agent assistance with dol — relevant to the `CLAUDE ## Theme 10: Cross-Platform and Compatibility -### Issue #58, #52: Windows compatibility +### Issue #58, #52, #40: Windows compatibility — RESOLVED (2026-07) -Several tests fail on Windows due to: -- Path separator differences (`/` vs `\`) -- Temp file handling -- Regex patterns with backslashes (Issue #40: `re.error: incomplete escape \U at position 2`) +Historically several tests failed on Windows due to path-separator differences (`/` vs `\`), +temp-file handling, and regex patterns with backslashes (`re.error: incomplete escape \U`). +Fixed across PRs #60 (Py3.12 escapes), #64 (`filter_regex` compiled as regex, not path +template), and #65 (cross-platform path/regex/separator fixes: 28 failures → 0). A fresh +`windows_ci.yml` run on `master` (2026-07-02) is **green**. + +**Residual:** `windows_ci.yml` is `workflow_dispatch` (manual) only, so nothing guards +against future Windows regressions — tracked as a new issue (make it run on push/PR). ### Issue #59 (CLOSED): Python 3.12 compatibility @@ -261,3 +284,7 @@ Fixed. dol now works with Python 3.12. | #8 | `FlatReader` refactored and stabilized | | #47 | Simpler affix codecs (partially addressed) | | #59 | Python 3.12 compatibility fixed | +| #40 | Windows regex-escape crash fixed in `naming.py` (PR #64/#65) | +| #50 | Stacking `cache_this` decorators enabled (PR #57, + 4 tests) | +| #52 | Windows test compatibility (duplicate of #58) | +| #58 | Windows tests pass; CI green 2026-07-02 (PR #64/#65) |