fix(sql_base,base): make the Mapping contract truthful (contains, len, missing-key policy) - #4
Merged
Merged
Conversation
`__len__` returned a SELECT's `rowcount`, which is -1 on drivers that don't pre-buffer results (SQLite among them). CPython's `__len__` guard then raised `ValueError: __len__() should return >= 0`, so `len(store)` and `list(store)` were both unusable even though `iter(store)` worked. It now issues a `SELECT count(*)`. Two smaller changes in the same area: - `__getitem__`'s bare `except:` is narrowed to `except Exception:`, so KeyboardInterrupt and SystemExit are no longer swallowed. Behaviour is otherwise unchanged. - A new keyword-only `missing_key_policy` makes the Mapping contract opt-in fixable. It defaults to `'empty'`, i.e. exactly today's behaviour (an absent key yields an empty result, so the inherited `__contains__` reports True for every key and `.get(key, default)` never returns its default). Passing `'raise'` raises KeyError instead, which is what `collections.abc.Mapping` requires and what makes `in` and `.get` truthful. The default is left alone because existing consumers branch on the empty/None result. Also fixes the `_first_value` doctest, which called a name (`_get_first`) that does not exist. Adds sqldol/tests/test_base_mapping_contract.py (in-memory SQLite, no external service), covering both the count fix and both policies -- including regression guards pinning the default's legacy per-class behaviour. Claude-Session: https://claude.ai/code/session_01L1aQPB34n7PU7jmbztSjBe
Addresses the first half of issue #2 (deliberately does not close it -- see below). `SQLAlchemyPersister` defined no `__contains__`, so it inherited the brute-force one from `dol.base.Collection`, which scans `iter(self)` looking for a key equal to `k`. But the persister's `__iter__` yields ORM row objects, never keys, so the comparison never matched and `k in persister` was False for every key -- including keys `persister[k]` resolves happily. `SQLAlchemyStore` and `SQLAlchemyTupleStore` inherited the same wrong answer, since `Store.__contains__` delegates through `_id_of_key`. Adds a getitem-based `__contains__` on the persister, which fixes all three classes at once and costs one query instead of a full scan. A key that cannot name a row at all still answers False rather than raising, matching what the inherited scan did. Deliberately out of scope: the other half of #2, making `__iter__` yield keys instead of ORM rows. That one is not backwards compatible -- `SQLAlchemyTupleStore._key_of_id` does `getattr(obj, field)` on whatever is yielded, and at least one known consumer is written against the row-yielding behaviour. A test pins the current iteration behaviour so the split is explicit, and the issue stays open for that half. Adds sqldol/tests/test_sqlalchemy_store_contract.py (in-memory SQLite, no external service). Claude-Session: https://claude.ai/code/session_01L1aQPB34n7PU7jmbztSjBe
Member
Author
|
Post-merge review notes (design, not fixed here):
|
This was referenced Sep 22, 2026
Closed
thorwhalen
added a commit
that referenced
this pull request
Sep 22, 2026
) The same rowcount trap #4 fixed in SqlBaseKvReader.__len__ and #5 fixed in SqlBaseKvStore.__setitem__ was still in TableRows.__len__, so len() of a TableRows over SQLite raised "ValueError: __len__() should return >= 0". Ask the database for COUNT(*), honouring the row filter. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits fixing sqldol's Mapping-contract violations, both scoped to leave
default behavior unchanged:
fix(base): make SqlBaseKvReader.__len__ return a real count—__len__returned a SELECT's
rowcount, which is-1on drivers that don't pre-buffer(SQLite among them), so
len(store)/list(store)raisedValueErroron thosebackends. Now issues
SELECT count(*). Also narrows__getitem__'s bareexcept:toexcept Exception:(no longer swallows KeyboardInterrupt/SystemExit) and adds an opt-in keyword-only
missing_key_policy('empty'default = today's behavior,
'raise'makesin/.get()truthful percollections.abc.Mapping).fix(sql_base): make \k in store` agree with `store[k]`—SQLAlchemyPersisterhad nocontains, so it inherited the brute-force scan fromdol.base.Collection, which compares againstiter's yield — but this persister'siteryields ORM row objects, never keys, sok in persisterwasFalsefor every key, including keyspersister[k]resolves happily. Adds a getitem-basedcontains`.Refs #2 (fixes the
__contains__half only; making__iter__yield keys insteadof ORM rows is deliberately out of scope — not backward compatible, a known
consumer is written against the row-yielding behavior — and is pinned by a new
regression test). Does not close #2.
Dependents check (fleet_dependents.json lists
raglab-appas the soledependent of
sqldol):raglab_appimports onlySqlDictReader/SqlDictStore(both
SqlBaseKvReader/SqlBaseKvStoresubclasses, commit 1's class) — neverSQLAlchemyPersister/SQLAlchemyStore/SQLAlchemyTupleStore(commit 2'sclasses), so the
__contains__change cannot reach it. Commit 1 keepsmissing_key_policydefaulted to'empty'(byte-for-byte prior behavior,pinned by
test_missing_key_policy_defaults_to_the_legacy_empty_behavior), andthe
__len__/except Exceptionchanges are strictly corrective.raglab_app'sown test suite is intentionally disabled in its CI (
[tool.wads.ci.testing] enabled = false— "private Streamlit app with no test suite; its modules usescript-style imports and run code at import time") and its full install needs a
pg_config/postgres toolchain not present on this box, so I could not execute itdirectly; the source-level compatibility read above is the substitute. Recorded
in DECISIONS.md.
Branch sat pushed with green CI and no PR for two weeks (thorwhalen/fleet_stuff
cleanup). Verified: master has not moved since the branch was cut (no rebase
needed).
sqldolhas nopyproject.toml/wads CI config (setup.py/setup.cfgonly), so gated with a plain venv + pytest instead of
wads ci-local: 20/20tests pass (
sqldol/tests/test_base_mapping_contract.py,sqldol/tests/test_sqlalchemy_store_contract.py). Hosted CI on the branch wasalready green from the original 2026-09-08 push.
Refs #2
🤖 Generated with Claude Code