From 653fe0fdfd437862378398b0b137a5bb55c14ef7 Mon Sep 17 00:00:00 2001 From: Thor Whalen <1906276+thorwhalen@users.noreply.github.com> Date: Fri, 21 Aug 2026 13:22:16 +0100 Subject: [PATCH] docs: master index, flat-model reference, roadmap SSOT; record fleet-program intents (#91-#94) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - misc/docs/README.md: master index — big picture (two architectures), reading paths per audience, full doc index with status labels. Absorbs and replaces dol_misc_docs_guide.md. - misc/docs/dol_flat_model.md: the flat model reference (Option G) — (leaf, spec, stack), flatten-and-compile, loudness, codec laws, scope limits, the Q0-Q5 decisions, code map. Adversarially verified against code (example executed verbatim; one caching overclaim corrected). - misc/docs/dol_roadmap.md: sequencing SSOT — tracks A-E, supersedes the issues-report waves; links the new intent issues #91 (fleet ledger), #92 (vocabulary horizon), #93 (layer kinds + fast-ops), #94 (cruft audit). - Targeted supersession banners: dol_design.md (4 sections + corrected the refuted pickling claim), general_design.md (layers: semantic vs runtime), dol_issues_report.md (historical snapshot). - CLAUDE.md: doc index refreshed (README/flat-model/roadmap rows, #16 marked shipped, dependents count 76 -> 85 per the 2026-08-03 scan). Claude-Session: https://claude.ai/code/session_01FLZ8T5a6Y4P3u25yC1JD9R --- CLAUDE.md | 22 +- misc/docs/README.md | 111 +++++++++ misc/docs/dol_architecture_map.md | 5 +- misc/docs/dol_design.md | 41 +++- misc/docs/dol_flat_model.md | 359 ++++++++++++++++++++++++++++++ misc/docs/dol_issues_report.md | 5 + misc/docs/dol_misc_docs_guide.md | 26 --- misc/docs/dol_roadmap.md | 113 ++++++++++ misc/docs/general_design.md | 6 + 9 files changed, 653 insertions(+), 35 deletions(-) create mode 100644 misc/docs/README.md create mode 100644 misc/docs/dol_flat_model.md delete mode 100644 misc/docs/dol_misc_docs_guide.md create mode 100644 misc/docs/dol_roadmap.md diff --git a/CLAUDE.md b/CLAUDE.md index 0dd46324..1d8d4c26 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -204,23 +204,33 @@ locally and fail in CI. ## Documentation Index (`misc/docs/`) +**Master index: [misc/docs/README.md](misc/docs/README.md)** — overview of the two +architectures (shipping delegation vs the flat model), reading paths per audience, +and the full doc index with status labels. The table below is the quick version. + | Document | Contents | |----------|----------| +| [README.md](misc/docs/README.md) | **Start here.** Master index: big picture, reading paths, status labels | +| [dol_flat_model.md](misc/docs/dol_flat_model.md) | **The flat model reference** (Option G): a wrap is `(leaf, spec, stack)` — flat compiled codec stack, typed interface specs, loudness policies, codec laws, scope limits, the six decisions Q0–Q5, code map for `dol/_interface_wrap.py` (private, experimental) | +| [dol_roadmap.md](misc/docs/dol_roadmap.md) | **Sequencing SSOT**: tracks A–E (flat engine P0–P3, is-a, paths engine, fleet program #91/#92/#94, hygiene) | | [general_design.md](misc/docs/general_design.md) | Language-agnostic design: what dol is, the KV pipeline, layered composition, patterns | -| [dol_design.md](misc/docs/dol_design.md) | Python architecture: class hierarchy, `wrap_kvs` deep dive, `Codec`/`Sig`/`Pipe`, critique | +| [dol_design.md](misc/docs/dol_design.md) | Python architecture: class hierarchy, `wrap_kvs` deep dive, `Codec`/`Sig`/`Pipe`, critique. **Partially superseded** — see its banner; prefer the architecture map for mechanics | | [dol_architecture_map.md](misc/docs/dol_architecture_map.md) | Code-verified structural map: module/dependency graph, public API, class hierarchy, `wrap_kvs`/codec machinery deep dive, ranked tech debt. **Start here for refactors.** | | [issues_and_discussions.md](misc/docs/issues_and_discussions.md) | GitHub issues/discussions themes, known limitations, open design questions | -| [dol_issues_report.md](misc/docs/dol_issues_report.md) | Prioritized issue triage + wave-by-wave tackle order | -| [dol_issue16_design.md](misc/docs/dol_issue16_design.md) | Issue #16 design: optional key-path write-through / autovivification — opt-in `create_missing`, contextual per-level factory, the `path_set_writeback` boundary engine + persistent-store write-back protocol, scoped plan. Design-only (no code yet). | +| [dol_issues_report.md](misc/docs/dol_issues_report.md) | Historical triage snapshot (2026-07-02); sequencing superseded by `dol_roadmap.md`, verification log still cited | +| [dol_issue16_design.md](misc/docs/dol_issue16_design.md) | Issue #16 design: optional key-path write-through / autovivification — opt-in `create_missing`, contextual per-level factory, the `path_set_writeback` boundary engine + persistent-store write-back protocol. **Implementation shipped** (see Known Limitations below). | | [dol_issue18_design.md](misc/docs/dol_issue18_design.md) | Issue #18 design: `self`-not-wrapped delegation trap — `wrapped_self` (shipped) now, is-a wrapping (deferred, major) later. | | [dol_issue83_design.md](misc/docs/dol_issue83_design.md) | Issue #83 design study: the **inverse** of #18 — a delegated method *receives* the unmapped key. Two delegation routes (a fix for one is a no-op on the other), a 13-package census (mostly **latent**; 12 claims refuted), and options A–F with verified costs: `wrapped_self` has its own silent hole (degrades with no live strong reference), chain-walking free functions break on non-`Store` layers, and the only form correct *by construction* is routing the capability through `__getitem__` as a sibling store. **§5 is the carry-forward list for a future redesign.** | | [dol_issue86_design.md](misc/docs/dol_issue86_design.md) | Discussion #86 design study: **Option G** — spec-carried boundary codecs on a flat proxy (wrapt lessons, KT/VT-annotated interface specs, flatten-and-compile codec stacks). Prototype in `dol/_interface_wrap.py` (private, experimental). Headline: is-a does **not** fix #83 for backend-direct method bodies — F and G serve disjoint populations; codec laws (§2.5) are the boundary invariant's fine print; flat-model guarantees are scoped to pure-codec stacks (filters/caches still nest). Open question 0: who owns the wrap_kvs endgame. | | [dol_issue10_design.md](misc/docs/dol_issue10_design.md) | Issues #10 + #2 (paired) design: recursive wrapping of nested stores (`recursive_wrap`) + a flat `KvReader`/`KvPersister` view (`flat_store`), sharing one `(path,key,value)` descent frontier and reusing the #16 `path_set_writeback` engine. Load-bearing fix: the recursion read-surface and the write-back boundary must be **different** objects (naive `boundary=self` infinite-loops). Model-2 read + write-into-existing in scope; persistent creation deferred to P3. Design-only (no code yet). | | [frontend_dol_ideas.md](misc/docs/frontend_dol_ideas.md) | `zoddal` design: TypeScript KV interface, adapters, Zod bridge, zod-collection-ui integration | +| [dol_content_metadata_bifurcation.md](misc/docs/dol_content_metadata_bifurcation.md) | The content/metadata split-store problem (feeds issue #80) | +| [code-quality-improvements.md](misc/docs/code-quality-improvements.md) | Tech-debt tracker: dead code, coverage gaps (feeds issue #94) | -> A **local-only** ecosystem inventory (gitignored) lives in `misc/data/`: dol's 76 -> dependents, their usages (file:line), a pre-PR test-gate order + runner, and the -> `wrap_kvs` blast-radius scan. Regenerate with the scripts there. +> A **local-only** ecosystem inventory (gitignored) lives in `misc/data/`: dol's 85 +> direct dependents (as of the 2026-08-03 scan), their usages (file:line), a pre-PR +> test-gate order + runner, and the `wrap_kvs` blast-radius scan. Regenerate with +> the scripts there. Growing this into a versioned fleet ledger is issue #91. --- diff --git a/misc/docs/README.md b/misc/docs/README.md new file mode 100644 index 00000000..a8011d8a --- /dev/null +++ b/misc/docs/README.md @@ -0,0 +1,111 @@ +# dol design documentation — start here + +`dol` is a pure-Python, dependency-free toolkit for putting a uniform `dict`-like +(Mapping) face on any storage backend — files, S3, databases, in-memory — by +composing **key transforms**, **value transforms** (codecs), **filters**, and +**caches** over a raw backend. Domain code speaks `store[key]`; the layers decide +what that means physically. + +This folder is the design memory of the project: reference docs (how it works), +design studies (why it works that way), decision records, and the roadmap. This +README is the master index — read the two paragraphs below for the big picture, +pick your reading path, then dig in. + +## The one thing to understand first: two architectures + +**Today (shipping)**: dol wraps by **has-a delegation** — each `wrap_kvs` / +`Store` layer is an object holding the previous one. It works, it's everywhere +(85 local dependent packages), and it has known structural traps: inside a +wrapped method `self` is the *unwrapped* store (#18), a delegated method +*receives* untransformed keys (#83), stacked wraps don't pickle, and there is no +inverse key mapping. + +**The decided direction (experimental)**: the **flat model** — a wrap is +`(leaf, spec, stack)`: one proxy, a typed interface spec saying where keys/values +occur in *every* method, and a flat, compiled list of codec layers. Re-wrapping +extends the list; it never nests. The endgame (decision Q0, 2026-08-10) is a +**split synthesis**: codec/instance wrapping compiles to the flat engine; +class-decoration becomes is-a. Both mechanisms exist because they fix *disjoint* +populations of the delegation traps. Narrative walkthrough: discussion +[#90](https://github.com/i2mint/dol/discussions/90). + +Where things stand and what's next: **[dol_roadmap.md](dol_roadmap.md)** (the +sequencing SSOT). + +## Reading paths + +**I want to build stores with dol (consumer).** +Repo-root `README.md` and `llms.txt` → the project `CLAUDE.md` core patterns (or +the `dol-store-building` skill) → [general_design.md](general_design.md) for the +conceptual model. Dip into [dol_design.md](dol_design.md)'s worked examples when +"why does this work" strikes. You should never need the issue docs; the gotchas +that matter (`wrapped_self`, `create_missing`) are in `CLAUDE.md`'s Known +Limitations. + +**I'm changing dol itself (maintainer / refactorer).** +[dol_architecture_map.md](dol_architecture_map.md) first (code-verified map) → +the `dol-dev-wrap-kvs` skill before touching `trans.py`/`base.py` → +[dol_roadmap.md](dol_roadmap.md) for what's in flight → then the design-study +chain for your area (see the index). Before any PR touching `wrap_kvs`: the +dependents test-gate (local `misc/data/` inventory). + +**I'm an agent needing orientation.** +`llms.txt` → project `CLAUDE.md` → stop. Escalate to +[dol_architecture_map.md](dol_architecture_map.md) §2–3 (navigation) and §9 +(idioms). Prefer the architecture map over [dol_design.md](dol_design.md) for +citation-grade mechanics — the latter is partially superseded (see its banner). + +**I want the design history (how we got here).** +[issues_and_discussions.md](issues_and_discussions.md) → +[dol_issues_report.md](dol_issues_report.md) (2026-07 triage snapshot) → the +study arc: [dol_issue18_design.md](dol_issue18_design.md) → +[dol_issue83_design.md](dol_issue83_design.md) → +[dol_issue86_design.md](dol_issue86_design.md) → discussion +[#90](https://github.com/i2mint/dol/discussions/90). The paths thread runs in +parallel: [dol_issue16_design.md](dol_issue16_design.md) → +[dol_issue10_design.md](dol_issue10_design.md). + +## Index + +### Reference (how it works) + +| Doc | Status | Contents | +|---|---|---| +| [general_design.md](general_design.md) | current (one scoped note) | Language-agnostic architecture: KV interfaces, interface hierarchy, middleware principle, the KV transform pipeline. The timeless layer — note in *Layered Composition* distinguishing semantic layers from runtime nesting. | +| [dol_design.md](dol_design.md) | **partially superseded** — see banner | Python implementation narrative: class hierarchy, `Store` hooks, `wrap_kvs`, codecs, `Sig`, `Pipe`. Still the only home of the `Sig` narrative, the design-critique register, `double_up_as_factory`, and the mid-altitude worked examples. Mechanics claims: prefer the architecture map. | +| [dol_architecture_map.md](dol_architecture_map.md) | current | Code-verified structural map + ranked tech debt. **Start here for refactors.** | +| [dol_flat_model.md](dol_flat_model.md) | current (engine: experimental/private) | **The flat model reference**: `(leaf, spec, stack)`, role codecs, flatten-and-compile, loudness, codec laws, scope limits, the six decisions, code map. | + +### Sequencing + +| Doc | Status | Contents | +|---|---|---| +| [dol_roadmap.md](dol_roadmap.md) | **living SSOT** | Tracks A–E: flat engine P0–P3, is-a, paths engine, the fleet program (#91 ledger / #92 vocabulary / #94 cruft), hygiene. Supersedes the issues-report waves. | +| [dol_issues_report.md](dol_issues_report.md) | historical snapshot (2026-07-02) | The triage that ordered the program: close-list, dependency map, verification log. Sequencing superseded by the roadmap; evidence still cited. | +| [issues_and_discussions.md](issues_and_discussions.md) | current-ish | Themes from GitHub issues/discussions. | + +### Design studies & decision records (why it works that way) + +| Doc | Contents | +|---|---| +| [dol_issue18_design.md](dol_issue18_design.md) | The delegation trap (`self` unwrapped inside methods): `wrapped_self` (shipped), is-a plan (Track B), rebind rejection. | +| [dol_issue83_design.md](dol_issue83_design.md) | The inverse trap (methods *receive* unmapped keys): 13-package census, options A–F, the carry-forward list (§5). | +| [dol_issue86_design.md](dol_issue86_design.md) | **Option G decision record**: the flat model's design study, adversarial-panel evidence, codec laws, Q0–Q5 decisions (§11), migration P0–P3. | +| [dol_issue16_design.md](dol_issue16_design.md) | Key-path write-through / autovivification (shipped): the `path_set_writeback` boundary engine. | +| [dol_issue10_design.md](dol_issue10_design.md) | Recursive wrapping + flat store view (designed, code pending); reuses the #16 engine. | +| [dol_content_metadata_bifurcation.md](dol_content_metadata_bifurcation.md) | The content/metadata split-store problem (feeds #80). | + +### Operational / peripheral + +| Doc | Contents | +|---|---| +| [code-quality-improvements.md](code-quality-improvements.md) | Tech-debt tracker (dead code, coverage gaps). Feeds #94. | +| [CHANGELOG.md](CHANGELOG.md) | Change log. | +| [frontend_dol_ideas.md](frontend_dol_ideas.md) | `zoddal`: the TypeScript/Zod incarnation of the dol idea. | +| [generate_llms_txt_instruction.md](generate_llms_txt_instruction.md) | How the repo's `llms.txt` files are generated. | + +> A **local-only** ecosystem inventory (gitignored) lives in `misc/data/`: dol's +> 85 direct dependents, their usages (file:line), a pre-PR test-gate order + +> runner, and the `wrap_kvs` blast-radius scan. Regenerate with the scripts +> there. Growing this into a versioned fleet ledger is +> [#91](https://github.com/i2mint/dol/issues/91). diff --git a/misc/docs/dol_architecture_map.md b/misc/docs/dol_architecture_map.md index 7a4d0541..315bcc98 100644 --- a/misc/docs/dol_architecture_map.md +++ b/misc/docs/dol_architecture_map.md @@ -24,7 +24,7 @@ It **complements, does not repeat** the existing `misc/docs`: | GitHub issues/discussions themes | [issues_and_discussions.md](issues_and_discussions.md) | Corrects the Issue #9 root-cause (§5.4) | | The split-store / content-metadata problem | [dol_content_metadata_bifurcation.md](dol_content_metadata_bifurcation.md) | — | | Dead code / coverage tracker | [code-quality-improvements.md](code-quality-improvements.md) | Adds new debt found (§11) | -| Doc index | [dol_misc_docs_guide.md](dol_misc_docs_guide.md) | Should be updated to list this file | +| Doc index | [README.md](README.md) | The master index; routes here for refactors | ### Staleness / inaccuracies found in existing docs @@ -600,7 +600,8 @@ single `cache_this` with an explicit `key`/`cache` over stacking. `path_get`/`_path_get`/`chain_get`. 10. **Every module must keep its top-level docstring** (all 20 currently have one — verified; they are auto-extracted for docs). When adding a module or editing one, preserve/enhance - it (per CLAUDE.md). And update `dol_misc_docs_guide.md` if you add a doc here. + it (per CLAUDE.md). And update `misc/docs/README.md` (the master index) if you + add a doc here. --- diff --git a/misc/docs/dol_design.md b/misc/docs/dol_design.md index 708eeb0c..b8825f2b 100644 --- a/misc/docs/dol_design.md +++ b/misc/docs/dol_design.md @@ -2,6 +2,15 @@ This document describes the Python-specific implementation of dol's design. For the language-agnostic concepts, see [general_design.md](general_design.md). +> **Status (2026-08): partially superseded.** This doc describes the *shipping* +> delegation architecture, which remains what users run today — but the redesign +> program ([dol_flat_model.md](dol_flat_model.md), +> [dol_issue86_design.md](dol_issue86_design.md), decisions Q0–Q5) has superseded +> several sections; each carries an inline note. For citation-grade mechanics +> prefer [dol_architecture_map.md](dol_architecture_map.md). Still uniquely here: +> the `Sig` narrative, the design-critique register, `double_up_as_factory`, and +> the worked examples. Index: [README.md](README.md). + --- ## Class Hierarchy @@ -65,6 +74,13 @@ The hooks default to identity (no-op), so `Store(dict())` behaves exactly like a ## `wrap_kvs`: The Core Transformation Function +> **Note (2026-08):** accurate for the shipping architecture, with two updates: +> the method-transform parameters documented below (`outcoming_key_methods` / +> `ingoing_key_methods` …) are **verified broken** and replaced by typed interface +> specs in the flat model ([dol_issue86_design.md](dol_issue86_design.md) TL;DR 1); +> and per decision Q0, codec/instance wrapping compiles to the flat engine at the +> endgame ([dol_flat_model.md](dol_flat_model.md)). + Located in `dol/trans.py:1801`. The most important function in the library. ```python @@ -152,6 +168,12 @@ store = wrap_kvs(store, obj_of_data=json.loads, data_of_obj=json.dumps) ## `store_decorator`: The Meta-Decorator +> **Note (2026-08):** the 4-way usage taxonomy survives, but the instance branch +> ("wraps it in `Store` first") is what changes under decision Q0: instance +> wrapping compiles to a flat proxy at the endgame, not a fresh `Store` subclass. +> The P2 compatibility questions are listed in +> [dol_roadmap.md](dol_roadmap.md) Track A. + Located in `dol/trans.py:130`. Enables writing a class-transforming function once and using it in 4 ways: ```python @@ -331,6 +353,14 @@ def my_func(*args, **kwargs): ... ## Delegation Pattern +> **Note (2026-08): the most-superseded section of this doc.** The snippet below +> is simplified (the real implementation guards via `object.__getattribute__`, +> see the architecture map), and delegation-as-the-architecture is what the flat +> carrier replaces for codec stacks: capability mirroring plus the +> `undeclared='exclude'` policy (decision Q2) eliminate the verified +> `DelegatedAttribute` raw-data leaks (`__or__`/`copy`/`fromkeys`). Delegation +> remains the shipping mechanism, and remains the model for filter/cache layers. + `Store` uses the delegation pattern: it holds a reference to an inner store (`self.store`) and delegates all storage operations to it. Attribute access falls through via `__getattr__`: ```python @@ -461,6 +491,12 @@ kt.dict_to_key({"user": "john", "year": "2024", "month": "01"}) # 'john/2024/01 ## Design Critique and Alternatives +> **Note (2026-08):** this register predates the redesign; several critiques have +> since been *realized* rather than refuted — item 1's Protocol suggestion and +> item 6's missing generics are now the flat model's core mechanism +> (KT/VT-annotated Protocol specs, typed codecs per decision Q3). Items 2 +> (`clear()`/LSP), 3 (naming — now issue #92), and 5 (async) remain live. + ### 1. ABC Inheritance vs. Protocols **Current approach**: Classes inherit from `collections.abc.Mapping`, `MutableMapping`, etc. @@ -536,7 +572,10 @@ kt.dict_to_key({"user": "john", "year": "2024", "month": "01"}) # 'john/2024/01 `wrap_kvs` is powerful but creates anonymous classes at runtime, which has implications: - `type(store).__name__` may not be meaningful -- Pickling can be tricky (though dol handles this via `__reduce__`) +- Pickling is **broken today** for decorator-form, `Files`, and stacked wraps (one + root cause: class-name shadowing — verified in the #86 study's 7-case matrix); + instance wraps and a single class-wrap work. The flat proxy repairs the stacked + case by construction. - Debugging stack traces show generic names For performance-critical code or when pickling is needed, direct subclassing is still more reliable. diff --git a/misc/docs/dol_flat_model.md b/misc/docs/dol_flat_model.md new file mode 100644 index 00000000..3c3ca8e7 --- /dev/null +++ b/misc/docs/dol_flat_model.md @@ -0,0 +1,359 @@ +# The flat model: spec-carried boundary codecs on a flat proxy + +> **Status: EXPERIMENTAL** (the roadmap's P0 prototype, now in P1 trials). The +> engine lives in `dol/_interface_wrap.py` — a +> private module, not exported from `dol`. Nothing in `base.py`/`trans.py` changed. +> This doc is the *reference*: how the mechanism works, what it guarantees, and what +> it refuses. The *rationale* — why this design won, the adversarial evidence, and +> the decision record — is [dol_issue86_design.md](dol_issue86_design.md). The +> roadmap for growing it into dol's main wrapping machinery is +> [dol_roadmap.md](dol_roadmap.md). Behavior is pinned by +> `dol/tests/test_interface_wrap.py` (~45 tests). Accurate as of v0.3.63. + +## Why this exists (one paragraph) + +dol's shipping wrap machinery (`wrap_kvs`, `Store`) composes by **has-a delegation**: +each wrap is an object holding the previous one. That architecture has a two-sided +delegation trap — inside a wrapped method, `self` is the *unwrapped* inner store +(Issue [#18](https://github.com/i2mint/dol/issues/18)), and a delegated non-dunder +method *receives* the outer, unmapped key (Issue +[#83](https://github.com/i2mint/dol/issues/83)) — plus per-layer delegation cost, +broken pickling for stacked wraps, and no inverse key mapping. The flat model +(Option G of discussion [#86](https://github.com/i2mint/dol/discussions/86)) +restructures wrapping so the #83 class of bug is impossible *by construction* for +the population it serves, and the other problems dissolve as side effects. + +## The model: a wrap is `(leaf, spec, stack)` + +One proxy object, however many layers: + +| Part | What it is | Where it lives on the proxy | +|---|---|---| +| **leaf** | The raw backend (a `dict`, a `Files`, a boto3-backed store…). Always directly reachable; never a proxy-of-proxy. | `_self_leaf`, exposed as `__wrapped__` | +| **spec** | An interface declaration: per method, *where* the "types of interest" (roles: KT, VT, or any TypeVar name) occur in arguments and returns. | `_self_spec` (an `InterfaceSpec`) | +| **stack** | The flat list of transformer layers. Each layer is a dict `{role: Codec(encoder, decoder)}`. | `_self_stack` (a tuple, innermost-first) | + +Method bodies always run against the leaf itself — `self` inside a leaf method **is +the leaf** — so internal `self[k]` / `self.x()` calls stay *below* the boundary and +transforms apply **exactly once**, at the boundary. This is structural, not +careful-coding: the no-double-apply property was verified with a counting encoder +through a spec'd method that internally calls another spec'd method. + +### The transformer pair: `Codec` + +```python +@dataclass(frozen=True) +class Codec: + encoder: Callable[[Any], Any] # outer -> inner (arguments going in) + decoder: Callable[[Any], Any] # inner -> outer (results coming out) + decoded_type: Optional[type] = None # outer-facing type tag (optional) + encoded_type: Optional[type] = None # leafward type tag (optional) +``` + +`encoder` handles what goes in: keys on `__getitem__`/`__setitem__`/`__delitem__`/ +`__contains__`/any spec'd method with a `KT` parameter; values on `__setitem__`. +`decoder` handles what comes out: `__getitem__`'s value, `__iter__`'s keys, any +spec'd return carrying a role. `Iterator[...]`-annotated returns are decoded +**lazily** (via `map`); `Iterable[...]`-annotated *arguments* are **materialized** +to a list (an `Iterable` contract implies the leaf may re-iterate). + +Note the vocabulary: **encoder/decoder**, not `id_of_key`/`key_of_id`. The flat +engine adopts the target naming language from day one (see the vocabulary-migration +intent in [dol_roadmap.md](dol_roadmap.md)). + +### The stack: extend, never nest + +Re-wrapping an already-wrapped object appends a layer to the flat tuple over the +**same leaf** (from `interface_wrap`): + +```python +if isinstance(obj, InterfaceProxy): + leaf = obj._self_leaf # same leaf — never a proxy-of-proxy + base_stack = tuple(obj._self_stack) # copy the existing flat list + ... +stack = base_stack + ((dict(codecs),) if codecs else ()) # extend, don't nest +``` + +Wrapping is **copy-not-mutate**: the old proxy stays valid, and an iterator +obtained before a new wrap keeps the pipelines it was compiled with. A 6-deep wrap +is still *one* object: `len(proxy._self_stack) == 6` and `proxy.__wrapped__` is the +raw backend. + +**Ordering convention** (the one detail that will bite you if you touch this code): +`_self_stack` is **innermost-first** — index 0 is the first layer applied to the +leaf. Encoding therefore iterates `reversed(stack)` (outermost layer encodes +first, moving inward); decoding iterates `stack` in order (innermost decodes +first, moving outward). + +### Flatten-and-compile + +At wrap time the stack is compiled **once**: + +```python +def _fused_role_funcs(stack, *, direction): + roles = set() + for layer in stack: + roles.update(layer) + out = {} + for role in roles: + if direction == "encode": + funcs = [layer[role].encoder for layer in reversed(stack) if role in layer] + else: + funcs = [layer[role].decoder for layer in stack if role in layer] + out[role] = _fuse(funcs) # left-to-right composition into ONE callable + return out +``` + +Per role, all encoders fuse into one callable and all decoders into another +(`_fuse` drops identity functions). Then, per spec'd method the leaf actually has, +`_compile_method_plan` bakes a **boundary plan**: encode the role-carrying +arguments → call the *leaf-bound* method → decode the return. Plans live in +`_self_plans`; the generated proxy class dispatches every spec'd method to its +plan. + +Plan compilation has three shapes: a positional-only plan for 3.10 builtin slots +with no inspectable signature (`dict.__getitem__`); a fast path when the only +role'd parameter is the first positional; and a general path using +`Signature.bind`. The **spec's signature is the outer contract** — calls bind +against it, so the leaf's own parameter names are irrelevant (`dict` calls its key +`key`; the spec may say `k`). + +The "compile for validation" half is `_validate_stack_seams`: where two adjacent +layers carry type tags for the same role, the outer layer's `encoded_type` must be +the inner layer's `decoded_type`, else the wrap refuses loudly. Untagged codecs +stay unchecked (progressive disclosure). + +### What the flat list buys + +Each of these was a named open problem under the nested model: + +- **The inverse key mapping exists by construction** — + `proxy._decode_role('KT', inner_key)` (the missing primitive of the #83 design + doc §5.4, needed by anything that *returns* keys: listings, `walk`, prefix + queries). The nested model has no inverse at all. +- **`inner_most_key` becomes a total fold** — `proxy._encode_role('KT', k)`; no + `.store`-chain walking, no non-`Store`-layer hazard, because there are no layers + at runtime. +- **The leaf is a strong structural reference** — `proxy.__wrapped__`, no weakref + registry. +- **Stacked wraps pickle by construction** — `__reduce__` recompiles from + `(leaf, spec_source, stack, policy)`; no dynamic class in the payload. (Lambda + codecs fail loudly; codecs must be module-level/picklable.) +- **One boundary hop at any depth** — measured 3.6× faster than nested `wrap_kvs` + at depth 6 for `__getitem__`, 3.4× for iteration. + +## The spec side + +The spec is what lets the boundary cover **all** methods, not just the Mapping +dunders — which is precisely what kills the #83 bug class (a delegated method like +`url_for(k)` receiving the outer, unmapped key). + +### Annotated form + +```python +from typing import Protocol, TypeVar, Iterator, Iterable +KT, VT = TypeVar('KT'), TypeVar('VT') + +class BucketInterface(Protocol[KT, VT]): + def __getitem__(self, k: KT) -> VT: ... # Mapping dunders are ordinary + def __iter__(self) -> Iterator[KT]: ... # spec entries - no privileged + def __contains__(self, k: KT) -> bool: ... # surface + def url_for(self, k: KT, *, expires_in: int = 3600) -> str: ... + def delete_many(self, keys: Iterable[KT]) -> None: ... + def items_page(self) -> Iterator[tuple[KT, VT]]: ... +``` + +Compilation walks `get_type_hints` + `signature` per method, recording the *paths* +at which role TypeVars occur. Supported shapes: bare, `list/set/frozenset/tuple/ +dict[...]`, `Iterable/Iterator[...]` (nesting included), `Optional[...]`, +`*args: KT` (elementwise). Everything else refuses **at wrap time** +(`UnsupportedSpecShape`) — including roles inside `Callable[[KT], ...]` +(contravariant positions) and `**kwargs: VT` (no annotation channel for keyword +names). Roles are matched **by TypeVar name, not identity** — deliberate: identity +matching would silently classify a user's same-named `KT` as "not a key", the +exact silent hole the mechanism exists to kill. + +### Dict form (no typing required) + +```python +spec = {'__getitem__': {0: 'KT', 'return': 'VT'}, + '__setitem__': {0: 'KT', 1: 'VT'}} # int keys = positional index +``` + +Same compiled algebra, same loudness rules. + +### Three refusal layers (loudness) + +The failure mode this design targets is **silence-by-omission**, so omission is +loud at every level the mechanism can see: + +1. **Undeclared public leaf attributes** — default policy `undeclared='exclude'`: + the wrap succeeds, and every *use* of an undeclared attribute raises + `UndeclaredAttributeError` with guidance (refusal at the moment of danger). + `'raise'` is strict mode (wrap-time refusal naming the attributes); + `'passthrough'` / `passthrough={...}` forward verbatim, explicitly. +2. **Unannotated parameters in spec'd methods** — `UnderAnnotatedSpecError` at + compile time (an unannotated key parameter would silently receive outer keys). +3. **Unsupported shapes** — `UnsupportedSpecShape` at compile time: refuse rather + than guess (properties/classmethods in specs included). + +## The proxy carrier + +A generated class per `(leaf type, spec, surface)` — cached for source-backed +(Protocol) specs; dict-form specs build a fresh class per wrap (id-keyed caching +was rejected over GC id-reuse collisions). The class namespace +contains **exactly the spec'd methods the leaf actually has** — capability +mirroring (a leaf without `__len__` yields a proxy without `__len__`) — and +nothing else: + +- A dunder outside the spec does not exist on the proxy — `proxy | other` on a + dict-leaf wrap raises `TypeError` instead of silently returning raw inner data + (today's class-wrap `DelegatedAttribute`s leak `__or__`/`copy`/`fromkeys`). +- Explicit dunder access via `__getattr__` raises plain `AttributeError` — a + forwarded raw `leaf.__contains__` would answer in the wrong key domain. +- A `__getitem__`-only spec gets a poisoned `__iter__` that raises `TypeError`, + blocking CPython's legacy sequence protocol from pushing integer keys through + the key codec. +- When the surface covers `__getitem__` + `__iter__`, an injected `__eq__` + compares **outer views** (a wrap equals a dict holding its outer items) and + `__hash__` is `None`. A `__getitem__`-only wrap keeps identity equality. +- Single-underscore attribute access is forwarded to the leaf; proxy-own state + lives under `_self_*` names (wrapt's lesson). + +## The codec laws (the invariant's fine print) + +The headline invariant — *a method correct on the bare leaf stays correct under +any codec stack* — holds **iff**: + +1. **The key decoder is total and injective on the leaf's actually-occurring + keys.** dol's own `prefixed`/`suffixed` codecs violate this on out-of-band keys + (decoding `'z.txt'` under a `'data/'` prefix codec). Filtering first — + `Pipe(filt_iter.prefixes(p), KeyCodecs.prefixed(p))` — remains the blessed + guard, and note that composition is a *mixed* stack (the filter is not a codec + layer). +2. **Encoder and decoder are mutual inverses on both domains.** Under a one-sided + codec, a method that returns its own key argument returns a *different* key. + The optional `decoded_type`/`encoded_type` tags are the enforcement hook + growing toward these laws. +3. **Container-shaped role arguments are copied at the boundary** (encode builds a + new list/dict/tuple), so a leaf method that *mutates* its argument in place + loses that side channel — rare, real, documented. + +Lazy iterator decoding adds a temporal caveat: a data-dependent decode failure +raises at *consumption* time, arbitrarily far from the call — loud but late. + +## Scope limits (read before extrapolating) + +- **Pure-codec stacks only.** The flat vocabulary today represents *codecs*: pure, + invertible, per-role transforms. `filt_iter` (changes the key **set**) and + `cached_keys` (stateful) are **not** codecs — mixed compositions still nest, and + the flat-model guarantees (inverse mapping, `__wrapped__` = raw backend, pickle + uniformity) are scoped to the codec layers. Extending the layer vocabulary is a + roadmap item, not a given ([dol_roadmap.md](dol_roadmap.md)). +- **Legacy `Store` = opaque leaf.** `interface_wrap` over a legacy `dol` Store + works but warns: the Store (and its `.store` chain) is treated as an opaque + leaf; guarantees apply only to the layers above it. +- **Instances only.** Class-wrapping raises `TypeError` (future work). Inherited + Protocol methods are skipped (TypeVar substitution is future work). No + `postget`/`preset` (key-aware value transforms) yet. +- **Populations.** Method bodies split three ways, and the flat model serves one: + - *Leaf-domain bodies* (adapter methods that talk to the backend: `url_for`, + `replace`, `delete_many`) — **the flat model's population**; boundary + transformation is the only mechanism in the #83 option space that serves it. + - *Outer-domain bodies* (extension methods over the wrapped view) — served by + `wrapped_self` today, is-a wrapping later (#18). The flat model deliberately + does not touch them. + - *View-blind bodies* (no key in the signature, semantics still depend on the + outer view, e.g. a `sync_to`) — no argument/return mechanism can serve these; + design pressure says have fewer of them. +- **The sibling-store decision rule**: *spec expressibility is the line.* A + capability whose shape the spec can express may be a method **iff spec'd**; a + shape the spec refuses (key-space-shifting returns, cross-keyspace values, keys + embedded in returned records, view-blind operations) must remain a sibling + store, handle, or free function. + +## The six decisions (Q0–Q5, maintainer, 2026-08-10) + +Recorded in [dol_issue86_design.md §11](dol_issue86_design.md); implemented in +PR [#89](https://github.com/i2mint/dol/pull/89) where code was called for: + +| # | Question | Decision | +|---|---|---| +| Q0 | Who owns the `wrap_kvs` endgame? | **Split synthesis**: codec/instance wrapping compiles to the flat engine; `@wrap_kvs` class-decoration becomes is-a. Each mechanism owns the population it is uniquely correct for. (Design work: P2/P3.) | +| Q1 | Public surface | **Private + facade**: engine stays private; built-in `MappingInterface` spec + `kv_interface_wrap` facade ship. Export reconsidered after P1 adapter trials. | +| Q2 | Loudness default | **`'exclude'`** (was `'raise'`): wrap succeeds, undeclared *use* raises with guidance. | +| Q3 | Typed codecs | **Yes, now**: optional `decoded_type`/`encoded_type` tags + seam validation. Tagging `dol.trans.Codec` is P2 (dependents gate). | +| Q4 | eq/hash | **Outer-view `__eq__`, no `__hash__`** when the surface covers traversal. `Store`'s own eq/hash incoherence queued for 0.4. | +| Q5 | `__class__` transparency | **Off today**; opt-in later, co-designed with #5. | + +## Is `Store` obsolete? No — three senses of no + +1. **Today**: `Store`/`wrap_kvs` are the only public, shipping mechanism. The flat + engine is private and used by nothing else in dol. +2. **At the decided endgame** (Q0): `wrap_kvs`'s codec/instance semantics compile + to the flat engine, but class decoration becomes is-a — `Store`'s role changes + rather than disappears. A compatibility appendix (what `.store` returns — dol + core reads it in ≥51 places — the #6 signature graft, the `wrapped_self` + backref) must be answered before any wiring. +3. **Permanently**: filters and caches are not codecs; unless the layer vocabulary + is extended (P3 / roadmap), `Store`-style nesting survives for them even in the + endgame. + +## Using it today + +```python +from dol._interface_wrap import interface_wrap, kv_interface_wrap, Codec + +# The wrap_kvs-shaped facade (built-in Mapping spec): +s = kv_interface_wrap( + {}, + id_of_key=lambda k: k + '.json', # facade keeps wrap_kvs vocabulary... + key_of_id=lambda k: k[:-5], + data_of_obj=str, + obj_of_data=int, +) +s['a'] = 1 # leaf now holds {'a.json': '1'} +assert list(s) == ['a'] and s == {'a': 1} + +# The general gesture (any spec, any roles): +s = interface_wrap(leaf, spec=BucketInterface, + codecs=dict(KT=Codec(encoder=..., decoder=...), + VT=Codec(encoder=..., decoder=...)), + passthrough={'wire'}) +``` + +Caveats: `kv_interface_wrap` transforms are **plain unary callables** — no +`wrap_kvs`-style `f(self, x)` signature inference, no `FirstArgIsMapping`. The +Mapping *mixin* methods (`get`, `keys`, `items`, `update`, …) are deliberately not +in the built-in spec — under the default policy they're hidden-and-loud. + +## Code map + +All in `dol/_interface_wrap.py` (private): + +| Identifier | Role | +|---|---| +| `Codec` | frozen dataclass: encoder/decoder + optional type tags | +| `InterfaceSpec` | compiled spec; `.from_annotated(Protocol)`, `.from_dict(...)` | +| `_find_role_sites` / `_transformer_for_path` | locate roles in annotations; build structure-mapping callables | +| `_fuse` / `_fused_role_funcs` | flatten-and-compile the stack into per-role pipelines | +| `_validate_stack_seams` | typed-codec seam validation (Q3) | +| `_compile_method_plan` | bake one boundary plan: encode args → call leaf → decode return | +| `_outer_signature` | the spec's signature is the outer contract | +| `InterfaceProxy` | carrier base: `__wrapped__`, `_encode_role`, `_decode_role`, `__reduce__`, `__getattr__` policy | +| `_build_proxy_class` | generated class per (leaf type, spec, surface); cached for source-backed specs only | +| `interface_wrap` | the entry point: normalize spec/stack, validate, compile, assemble | +| `MappingInterface` / `kv_interface_wrap` | built-in Mapping spec + `wrap_kvs`-shaped facade (Q1) | +| `InterfaceWrapError`, `UnsupportedSpecShape`, `UnderAnnotatedSpecError`, `UndeclaredAttributeError` | the loudness surface | + +## History and companions + +- Discussion [#90](https://github.com/i2mint/dol/discussions/90) — the complete + narrative walkthrough of the #86/Option G cycle. +- [dol_issue86_design.md](dol_issue86_design.md) — the design study + decision + record (adversarial-panel evidence, verification log, migration P0–P3). +- [dol_issue83_design.md](dol_issue83_design.md) — the census + options A–F + + carry-forward list this design answers. +- [dol_issue18_design.md](dol_issue18_design.md) — the other half (outer-domain + bodies): `wrapped_self` now, is-a later. +- [dol_roadmap.md](dol_roadmap.md) — where this goes next. diff --git a/misc/docs/dol_issues_report.md b/misc/docs/dol_issues_report.md index f5fbead3..316eae9c 100644 --- a/misc/docs/dol_issues_report.md +++ b/misc/docs/dol_issues_report.md @@ -1,5 +1,10 @@ # dol — Issues Triage & Tackle-Order Report +> **Status: historical snapshot (2026-07-02).** The wave/tackle order in §2 is +> superseded by [dol_roadmap.md](dol_roadmap.md) (the sequencing SSOT). The +> close-list, dependency map, and verification log remain the cited evidence +> record. +> > **Purpose:** a scannable, prioritized map of dol's open GitHub issues — what is > already resolved (and should be closed), and in what order to tackle the rest. > Companion to [issues_and_discussions.md](issues_and_discussions.md) (themes/history), diff --git a/misc/docs/dol_misc_docs_guide.md b/misc/docs/dol_misc_docs_guide.md deleted file mode 100644 index f36da530..00000000 --- a/misc/docs/dol_misc_docs_guide.md +++ /dev/null @@ -1,26 +0,0 @@ -### `dol` — Python Data Object Layer - -These documents describe **dol**, a Python library for building uniform `dict`-like (MutableMapping) interfaces to any storage backend. The core idea: map CRUD operations onto Python's native mapping protocol (`__getitem__`, `__setitem__`, `__delitem__`, `__iter__`). - -- **`general_design.md`** (~220 lines) — Language-agnostic architecture of dol. Covers the key insight (language-native KV interfaces), the interface hierarchy (`Collection → KvReader → KvPersister → Store`), the middleware principle (dol sits between domain code and storage), and the KV transform pipeline (`key_of_id`, `id_of_key`, `obj_of_data`, `data_of_obj`). - -- **`dol_design.md`** (~540 lines) — Python-specific implementation details. Covers the class hierarchy (rooted in `collections.abc`), the `Store` class with its 4 transform hooks, `wrap_kvs` (the core transformation function), key/value codecs (`ValueCodecs`, `KeyCodecs`), path-based stores, caching (`cache_this`), and the `Pipe` composition utility. The reference implementation doc. *(Some line-cites are stale; see `dol_architecture_map.md` for code-verified numbers.)* - -- **`dol_architecture_map.md`** (~600 lines) — Code-verified structural/mechanical map of the current source: per-module table + internal dependency graph, the exact public API surface, the class hierarchy, and a deep dive on the `wrap_kvs`/`store_decorator`/codec machinery (including the precise signature-conditioning logic behind Issue #9). Ends with a ranked tech-debt list and "notes for dev-skill authors." **Start here when refactoring or building agent tooling on dol.** - -- **`issues_and_discussions.md`** (~270 lines) — Themes from dol's GitHub issues/discussions. Major topics: `wrap_kvs` design tensions (signature-based conditioning, `self` not being the wrapped instance, recursive wrapping), the builtin codec ecosystem, path key handling, store composition patterns, and API ergonomics debates. Kept roughly current (resolved issues flagged). - -- **`dol_issues_report.md`** (~150 lines) — Actionable triage: which open issues are already resolved (close them) and a wave-by-wave **tackle order** for the rest, with an inter-issue dependency graph. The "what to work on next" companion to `issues_and_discussions.md`. - -- **`dol_issue18_design.md`** (~250 lines) — Decision-ready design for Issue #18 (`self` is the unwrapped inner store inside a delegation-wrapped class's own methods). Mechanism, a 26-site ecosystem blast-radius classification (0 break, 6 latent bugs), four fix designs with adversarial judging, an empirical verification log, and the staged recommendation: ship `wrapped_self()` now (Phase 1, backward-compatible), commit to is-a wrapping later (closes #18 + #6), reject method-rebinding. - -- **`dol_issue83_design.md`** (~300 lines) — Design study for Issue #83, the **inverse** of #18: a delegated method *receives* the outer, unmapped key. Documents the two delegation routes (`Store.__getattr__` and `DelegatedAttribute.__get__` — a fix for one is a silent no-op on the other), a 13-package ecosystem census (overwhelmingly **latent**, and 12 survey claims refuted), and options A–F each with the running-code evidence for its cost: `wrapped_self` degrades silently when nothing holds a strong reference to the wrapper, chain-walking free functions break on a non-`Store` layer, and the only form correct *by construction* is exposing the capability as a sibling store keyed through `__getitem__`. **§5 is the explicit carry-forward list for a future redesign of the wrapping machinery.** - -- **`dol_content_metadata_bifurcation.md`** (~450 lines) — Design study of the content/metadata split-store problem (a store whose values carry both payload and metadata). - -- **`code-quality-improvements.md`** (~230 lines) — Technical debt tracker for dol: dead code, unused parameters, incomplete implementations, test coverage gaps. Operational/maintenance reference. - -> A **local-only** ecosystem inventory lives under `misc/data/` (gitignored: it names -> private dependents). It maps dol's 76 local dependents, their dol usages (file:line), -> and a pre-PR test-gate order — regenerate with `misc/data/scan_dol_usages.py`. - diff --git a/misc/docs/dol_roadmap.md b/misc/docs/dol_roadmap.md new file mode 100644 index 00000000..7c6919b2 --- /dev/null +++ b/misc/docs/dol_roadmap.md @@ -0,0 +1,113 @@ +# dol roadmap — the sequencing SSOT + +> **Status: living document.** This is the single source of truth for *what happens +> in which order* across dol's redesign program. It supersedes the wave/tackle order +> of [dol_issues_report.md](dol_issues_report.md) §2 (kept as the 2026-07-02 triage +> snapshot — its close-list, dependency map, and verification log remain the +> evidence record). Design content lives in the linked docs; this file only +> sequences it. Last updated 2026-08-21. + +## The program in one paragraph + +dol's wrapping machinery is migrating from **has-a delegation** (nested wrapper +objects, the #18/#83 delegation traps) to a **two-mechanism endgame** decided on +2026-08-10 (decision Q0, [dol_issue86_design.md §11](dol_issue86_design.md)): +codec/instance wrapping compiles to the **flat boundary-codec engine** +([dol_flat_model.md](dol_flat_model.md)), and `@wrap_kvs` class-decoration becomes +**is-a** ([dol_issue18_design.md](dol_issue18_design.md)). In parallel: the paths / +boundary-write-back engine matures (#16 shipped, #10/#2 designed), and a +fleet-facing program (ledger → vocabulary migration → cruft audit) prepares the +ecosystem for the eventual breaking surface change. Discussion +[#90](https://github.com/i2mint/dol/discussions/90) is the complete narrative +walkthrough of how we got here. + +## Track A — the flat engine (Option G) + +| Phase | Status | What | +|---|---|---| +| **P0 — prototype** | **done** (v0.3.62–63, PRs [#88](https://github.com/i2mint/dol/pull/88)/[#89](https://github.com/i2mint/dol/pull/89)) | Private `dol/_interface_wrap.py` + ~45 tests; the six decisions Q0–Q5 implemented where code was called for (Q2 `exclude` default, Q3 typed-codec seams, Q4 outer-view eq, Q1 `MappingInterface` + `kv_interface_wrap` facade). | +| **P1 — adapter trials** | **in progress** | Census-family adapters trial `interface_wrap` on spec-expressible surfaces; sibling stores per the spec-expressibility rule. First trial (s3dol `url_for`) posted to [#86](https://github.com/i2mint/dol/discussions/86) on 2026-08-11. Remaining: more adapters, the adapter-side conformance helper, the legacy-path warning idea. Export of the engine is reconsidered only after P1 evidence (decision Q1). | +| **P2 — the wrap_kvs facade** | not started; **gated on the compatibility appendix** | A wrap_kvs facade over the engine for flat-equivalent configurations (falling back to nesting, loudly, where a layer observes its domain). Must answer first: does the product subclass `Store`; what does `.store` return (≥51 lines in dol core read it); is the #6 signature graft kept; does the engine register the `wrapped_self` backref. Also P2: tag `dol.trans.Codec` with the Q3 type tags (dependents test-gate). | +| **P3 — layer vocabulary** | intent recorded → **[#93](https://github.com/i2mint/dol/issues/93)** | Extend the flat stack beyond pure codecs: filter / cache / contextual-codec / interceptor layer kinds, and spec-registered fast-ops ([#24](https://github.com/i2mint/dol/discussions/24), [#56](https://github.com/i2mint/dol/issues/56)). See Track D interplay: the layer-kind design should inherit the new vocabulary from #92. | + +## Track B — is-a class decoration + +The other half of the Q0 split synthesis: `@wrap_kvs`-as-class-decorator becomes +is-a wrapping, absorbing the #6 (signature freeze) and #5 (wrapper-class control) +concerns. Design: [dol_issue18_design.md](dol_issue18_design.md) Phase 3, with two +constraints discovered since: is-a does **not** fix #83's backend-direct bodies +(disjoint populations — that's Track A's job), and a naive is-a hook shadows a +leaf that owns `_id_of_key` (verified, [dol_issue86_design.md §8](dol_issue86_design.md)). +Not scheduled before P2 lands; the crisp "which mechanism fires when" rule is +P2/P3 design work. Open issues folded in here: [#18](https://github.com/i2mint/dol/issues/18), +[#5](https://github.com/i2mint/dol/issues/5). + +## Track C — the paths / boundary-write-back engine + +- **#16 — shipped** (v0.3.55): `create_missing`/`autoviv`, the `path_set_writeback` + boundary engine in `dol/_paths_core.py`. Design record: + [dol_issue16_design.md](dol_issue16_design.md). +- **#10 + #2 — designed, code pending**: recursive wrapping (`recursive_wrap`) + a + flat `KvReader`/`KvPersister` view (`flat_store`) sharing one descent frontier + and reusing the #16 engine. Design: [dol_issue10_design.md](dol_issue10_design.md). + Persistent store-of-stores creation is deferred inside that design (its P3). + +## Track D — the fleet program (intent recorded 2026-08-21) + +Maintainer intents, recorded as issues; sequencing is **ledger first** — both +other legs consume it. + +1. **[#91 — fleet usage ledger](https://github.com/i2mint/dol/issues/91)**: grow + the local import-census (85 direct dependents, 2026-08-03 scan) into a + versioned ledger: full commit pins + baseline test results + restore helper; + AST-level subclass and kwarg census; zero-usage report; successive scans. +2. **[#92 — vocabulary horizon](https://github.com/i2mint/dol/issues/92)**: + `{kind}_{encoder|decoder|codec}` replaces the `X_of_Y` language (and + `kv_wrap`'s `outcoming_*`/`ingoing_*` — three vocabularies to reconcile). + Strategy: successor surface, not rename-in-place; `wrap_kvs` stays as a + back-compat facade over it. Execution gated on P1 evidence + P2 facade + #91. + New surfaces adopt the target vocabulary from day one (the flat engine's + `Codec(encoder, decoder)` already does). +3. **[#94 — cruft audit](https://github.com/i2mint/dol/issues/94)**: relocate + fleet-unused, low-value members (→ xdol / tests / recipes). Current scan: + 86/147 `__init__` exports with zero detected fleet usage (61 public >1 year). + Gated on #91's authoritative census (star-imports + submodule-only compat + surface). Complementary to [#70](https://github.com/i2mint/dol/issues/70). + +## Track E — hygiene and remaining triage + +- Open issues not yet claimed by a track: [#69](https://github.com/i2mint/dol/issues/69) + (path-get/flatten dedup), [#70](https://github.com/i2mint/dol/issues/70) (module + splitting), [#15](https://github.com/i2mint/dol/issues/15) (AttrContainer tab + completion), [#2](https://github.com/i2mint/dol/issues/2) (kv_walk docs), + [#1](https://github.com/i2mint/dol/issues/1) (documentation ideas). +- **Untriaged** (filed after the 2026-07-02 triage): [#80](https://github.com/i2mint/dol/issues/80) + (record/metadata DataProvider layer — see + [dol_content_metadata_bifurcation.md](dol_content_metadata_bifurcation.md)), + [#82](https://github.com/i2mint/dol/issues/82) (prefix-relativization boundary bug). +- Repo cleanup notes: the remote branch refs for merged PRs #88/#89 and the + 2026-07-11 `stash/*` branches (full-tree WIP snapshots predating the whole + #83/#86 cycle) are stale and can be pruned after a quick skim for salvage. + +## Dependency sketch + +``` +P0 ──► P1 ──► P2 ──► P3 (#93) + │ ▲ + │ (vocabulary from #92 informs layer-kind naming) + ▼ │ + Track B (is-a) │ + │ +#91 (ledger) ──► #92 (vocabulary execution) ──► fleet migration + └─────────► #94 (cruft audit) +Track C (#10/#2 implementation) — independent; reuses the #16 engine +``` + +## Decided inputs (do not re-litigate; re-open only with new evidence) + +Q0 split synthesis · Q1 private + facade · Q2 `exclude` loudness default · +Q3 typed codecs now · Q4 outer-view eq, no hash · Q5 no `__class__` transparency. +Full record: [dol_issue86_design.md §11](dol_issue86_design.md); narrative: +[#90](https://github.com/i2mint/dol/discussions/90). Also standing: the rebind +family is rejected (twice, on strengthened grounds — §6 of the same doc); +`clear()` stays disabled on `KvPersister`. diff --git a/misc/docs/general_design.md b/misc/docs/general_design.md index 5c919309..510a6256 100644 --- a/misc/docs/general_design.md +++ b/misc/docs/general_design.md @@ -124,6 +124,12 @@ def postget(k, v): ## Layered Composition (Russian Dolls) +> **Scope note (2026-08):** layers are dol's *semantic* model — always true. As +> *runtime structure*, nesting is how the shipping architecture works; under the +> decided flat-model endgame, pure-codec stacks compile to a single flat proxy +> with fused transform pipelines (one boundary hop), while filters and caches +> keep nesting. See [dol_flat_model.md](dol_flat_model.md). + The name "dol" evokes Russian dolls: layers of wrappers, each adding a transformation. A store is built by stacking layers: ```