Skip to content

CLAUDE.md

Latest commit

 

History

History
266 lines (191 loc) · 13.8 KB

CLAUDE.md

File metadata and controls

266 lines (191 loc) · 13.8 KB
Raw

dol — AI Agent Guide

dol is a pure-Python (no dependencies) toolkit for wrapping any storage backend (files, S3, databases, dicts) behind a uniform dict-like interface. Version 0.3.38. Python ≥ 3.10.

For a comprehensive agent-readable API reference, see llms-full.txt. For a quick orientation, see llms.txt.

Key Files

File What's in it
dol/base.py Collection, KvReader, KvPersister, Store — the class hierarchy
dol/trans.py wrap_kvs (core), store_decorator, filt_iter, cached_keys, Codec, kv_wrap
dol/kv_codecs.py ValueCodecs, KeyCodecs — ready-made codec namespaces
dol/caching.py cache_this, cache_vals, store_cached, WriteBackChainMap
dol/paths.py KeyTemplate, mk_relative_path_store, KeyPath, path_get/set/filter
dol/filesys.py Files, TextFiles, JsonFiles, PickleFiles — filesystem stores
dol/sources.py FlatReader, FanoutReader/Persister, CascadedStores
dol/signatures.py Sig — signature arithmetic
dol/util.py Pipe, lazyprop, partialclass, groupby
dol/__init__.py Public API — all exports live here

Core Pattern: Building Stores

The fundamental operation is wrapping a backend with transforms:

from dol import wrap_kvs, Files
import json

# Add JSON serialization to a file store
JsonFileStore = wrap_kvs(Files, obj_of_data=json.loads, data_of_obj=json.dumps)

# Or wrap an instance
s = wrap_kvs(dict(), id_of_key=lambda k: k.upper(), key_of_id=str.lower)

wrap_kvs parameters:

  • key_of_id / id_of_key — outgoing/incoming key transforms
  • obj_of_data / data_of_obj — outgoing/incoming value transforms
  • postget(key, data) → obj — value transform that knows the key (for reads)
  • preset(key, obj) → data — value transform that knows the key (for writes)
  • key_codec / value_codec — Codec objects (encoder+decoder pair)

Core Conventions

  • X_of_Y naming: key_of_id = "give me a key, you give me an id" (outgoing). id_of_key = "give me an id, you give me a key" (incoming). Always pairs.
  • KvReader for read-only: subclass KvReader (not KvPersister) when writes aren't needed.
  • KvPersister for read-write: clear() is disabled — override only if you're sure.
  • Test with dict, deploy with real backend: wrap_kvs(dict, ...) first, then swap dict for Files, a DB store, etc.
  • Transforms are pure functions: they should be stateless and not have side effects.

How to Create a New Store

Option 1: wrap_kvs (preferred for most cases)

from dol import wrap_kvs

MyStore = wrap_kvs(
    dict,
    id_of_key=lambda k: k + ".json",
    key_of_id=lambda _id: _id[:-5],
    obj_of_data=json.loads,
    data_of_obj=json.dumps,
)

Option 2: Subclass KvReader/KvPersister

from dol.base import KvReader


class MyReader(KvReader):
    def __getitem__(self, k): ...
    def __iter__(self): ...
    def __len__(self): ...  # optional, falls back to iteration count

Option 3: Subclass Store (when you need transform hooks)

from dol.base import Store


class MyStore(Store):
    def _id_of_key(self, k):
        return k.upper()

    def _key_of_id(self, _id):
        return _id.lower()

    def _data_of_obj(self, obj):
        return json.dumps(obj)

    def _obj_of_data(self, data):
        return json.loads(data)

Ready-Made Codecs

from dol import ValueCodecs, KeyCodecs, Pipe

# Common value codecs
ValueCodecs.pickle()  # pickle.dumps / pickle.loads
ValueCodecs.json()  # json.dumps / json.loads
ValueCodecs.gzip()  # compress/decompress
ValueCodecs.str_to_bytes()  # encode/decode

# Key codecs
KeyCodecs.suffixed(".pkl")  # add/strip suffix
KeyCodecs.prefixed("ns:")  # add/strip prefix

# Chain with Pipe
MyStore = Pipe(KeyCodecs.suffixed(".pkl"), ValueCodecs.pickle())(dict)

Store Decorators

Most tools in trans.py use @store_decorator, making them work 4 ways:

from dol import filt_iter, cached_keys


# As class decorator
@filt_iter(filt=lambda k: k.endswith(".json"))
class MyStore(dict): ...


# As instance wrapper
s = filt_iter(my_store, filt=lambda k: k.endswith(".json"))

# As factory
json_only = filt_iter(filt=lambda k: k.endswith(".json"))
s = json_only(my_store)

Caching

from dol import cache_this, cache_vals, store_cached


# Cache a property or method
class MyClass:
    @cache_this
    def expensive(self):
        return sum(range(1_000_000))


# Cache fetched values from a slow store
fast = cache_vals(slow_store)


# Persist function results across sessions
@store_cached(JsonFiles("/cache"))
def compute(x, y):
    return slow_computation(x, y)

Testing Approach

Always prototype with dict as the backend:

# 1. Test logic with dict
s = wrap_kvs(dict(), obj_of_data=json.loads, data_of_obj=json.dumps)
s["key"] = {"a": 1}
assert s["key"] == {"a": 1}

# 2. Swap to real backend
from dol import Files

s = wrap_kvs(Files("/data"), obj_of_data=json.loads, data_of_obj=json.dumps)

Run tests: a bare pytest (from the repo root) runs exactly what CI runs — the dol/tests/ unit tests and every module doctest, with CI's doctest flags. Narrow it with pytest dol/tests/ (unit tests only) or pytest dol/caching.py (one module's doctests). Do not add NORMALIZE_WHITESPACE to doctest_optionflags: CI does not pass it, so doctests relying on it would pass locally and fail in CI.

Documentation Index (misc/docs/)

Master index: misc/docs/README.md — overview of the two architectures (shipping delegation vs the flat model), reading paths per audience, and the full doc index with status labels. The table below is the quick version.

Document Contents
README.md Start here. Master index: big picture, reading paths, status labels
dol_flat_model.md The flat model reference (Option G): a wrap is (leaf, spec, stack) — flat compiled codec stack, typed interface specs, loudness policies, codec laws, scope limits, the six decisions Q0–Q5, code map for dol/_interface_wrap.py (private, experimental)
dol_roadmap.md Sequencing SSOT: tracks A–E (flat engine P0–P3, is-a, paths engine, fleet program #91/#92/#94, hygiene)
general_design.md Language-agnostic design: what dol is, the KV pipeline, layered composition, patterns
dol_design.md Python architecture: class hierarchy, wrap_kvs deep dive, Codec/Sig/Pipe, critique. Partially superseded — see its banner; prefer the architecture map for mechanics
dol_architecture_map.md Code-verified structural map: module/dependency graph, public API, class hierarchy, wrap_kvs/codec machinery deep dive, ranked tech debt. Start here for refactors.
issues_and_discussions.md GitHub issues/discussions themes, known limitations, open design questions
dol_issues_report.md Historical triage snapshot (2026-07-02); sequencing superseded by dol_roadmap.md, verification log still cited
dol_issue16_design.md Issue #16 design: optional key-path write-through / autovivification — opt-in create_missing, contextual per-level factory, the path_set_writeback boundary engine + persistent-store write-back protocol. Implementation shipped (see Known Limitations below).
dol_issue18_design.md Issue #18 design: self-not-wrapped delegation trap — wrapped_self (shipped) now, is-a wrapping (deferred, major) later.
dol_issue83_design.md Issue #83 design study: the inverse of #18 — a delegated method receives the unmapped key. Two delegation routes (a fix for one is a no-op on the other), a 13-package census (mostly latent; 12 claims refuted), and options A–F with verified costs: wrapped_self has its own silent hole (degrades with no live strong reference), chain-walking free functions break on non-Store layers, and the only form correct by construction is routing the capability through __getitem__ as a sibling store. §5 is the carry-forward list for a future redesign.
dol_issue86_design.md Discussion #86 design study: Option G — spec-carried boundary codecs on a flat proxy (wrapt lessons, KT/VT-annotated interface specs, flatten-and-compile codec stacks). Prototype in dol/_interface_wrap.py (private, experimental). Headline: is-a does not fix #83 for backend-direct method bodies — F and G serve disjoint populations; codec laws (§2.5) are the boundary invariant's fine print; flat-model guarantees are scoped to pure-codec stacks (filters/caches still nest). Open question 0: who owns the wrap_kvs endgame.
dol_issue10_design.md Issues #10 + #2 (paired) design: recursive wrapping of nested stores (recursive_wrap) + a flat KvReader/KvPersister view (flat_store), sharing one (path,key,value) descent frontier and reusing the #16 path_set_writeback engine. Load-bearing fix: the recursion read-surface and the write-back boundary must be different objects (naive boundary=self infinite-loops). Model-2 read + write-into-existing in scope; persistent creation deferred to P3. Design-only (no code yet).
frontend_dol_ideas.md zoddal design: TypeScript KV interface, adapters, Zod bridge, zod-collection-ui integration
dol_content_metadata_bifurcation.md The content/metadata split-store problem (feeds issue #80)
code-quality-improvements.md Tech-debt tracker: dead code, coverage gaps (feeds issue #94)

The fleet usage ledger (#91) lives in gitignored misc/data/ — start at its README. python -m fleet_ledger scan records one immutable, timestamped scan: dol's 89 direct dependents pinned at full 40-char commits, their imports, AST-verified subclasses and call-site kwargs (all file:line), a generated zero-usage report, and the pre-PR test-gate order. baseline captures every dependent's suite result so a candidate's damage can be told apart from pre-existing red; pin restore freezes the fleet at a scan and pin undo releases it. Scans accumulate — nothing is overwritten. Figures quoted in these docs are from the 2026-08-21 scan.

Agent Skills & Commands (.claude/)

Dev skills (.claude/skills/, for working on dol):

  • dol-dev-wrap-kvs — the wrap_kvs/store_decorator/Store.wrap machinery: the signature-conditioning rule, FirstArgIsMapping, the delegation architecture + self/ signature traps (#18/#6), and the mandatory dependents test-gate. Read before touching trans.py/base.py.
  • dol-dev-portability — Windows/POSIX landmines for path/key code.

Consumer skills (.claude/skills/, for using dol):

  • dol-store-building — wrap any backend behind a dict interface: wrap_kvs, codecs, the ready-made file stores, filt_iter, caching, and self-aware transforms.

Commands (.claude/commands/): /new-store, /add-codec, /explain-store — interactive scaffolds.

Known Limitations / Gotchas

  • wrap_kvs + self inside methods: When a wrap_kvs-decorated class uses self[k] in its own methods, self is the unwrapped inner store, so transforms are bypassed (Issue #18 — delegation architecture). Blessed fix: from dol import wrapped_self and write wrapped_self(self)[k] to reach the outer, transform-applying store (climbs to the outermost wrapper for stacked/Pipe wraps; a safe no-op on direct Store subclasses). The older sq(self)[k] re-wrap still works. A structural fix (is-a wrapping) is proposed in misc/docs/dol_issue18_design.md.
  • clear() is disabled on KvPersister. Call ensure_clear_to_kv_store(store) to re-enable.
  • No async support in core. Use synchronous wrappers for async backends (thread pool, etc.).
  • Transforms wanting the store: a transform is called f(self, data) only if its first param is named self/store/mapping and it has ≥2 required params; otherwise f(data). Mark explicitly with FirstArgIsMapping(f). (bytes.decode as obj_of_data now works — Issue #9 fixed.)
  • Windows paths: cross-platform fixes landed (Issues #40/#52/#58 resolved, CI green). See the dol-dev-portability skill before touching path/key code.
  • Key-path write-through / autovivification (Issue #16): by default, writing a deep path whose intermediates are missing raises KeyError (unchanged), and — a pre-existing gotcha — writing an existing deep path on a copy-semantics/persistent store (Files, any wrap_kvs with a non-identity obj_of_data) silently mutates a detached copy and is lost. Opt in with create_missing=True on add_path_access/KeyPath (or the autoviv(store) alias): missing intermediates are created via a contextual mk_missing(ctx) factory and every write is persisted through a single boundary write-back — which also closes the silent-loss. Creation is announced via warnings.warn (pass on_create=None to silence); unsafe cases raise PathCreationError/PathWritebackError. Engine lives in the dependency-free leaf dol/_paths_core.py (path_set_writeback); path_set there also fixes the old factory-not-propagated bug. Persistent store-of-stores (nested Files) is out of scope → deferred to #10. Full design: misc/docs/dol_issue16_design.md.