Skip to content

feat: paginate scan/query; batched items()/values() through wrappers (#4) - #8

Merged
thorwhalen merged 2 commits into
masterfrom
feat/pagination-bulk-views-4
Oct 13, 2026
Merged

thorwhalen merged 2 commits into
masterfrom
feat/pagination-bulk-views-4

Conversation

@thorwhalen

@thorwhalen thorwhalen commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Fixes #4 (the parts that need no breaking change; see the two follow-ups below).

  • Pagination (data loss fix): __iter__, __len__, scan_items/scan_values and the query readers now follow LastEvaluatedKey instead of reading only the first ~1 MB page.
  • Key hook: _primary_key(k) is the one place a store key becomes the item's primary key (base, partition and prefix readers override it); __getitem__ is shared.
  • Batched .items()/.values(): dol's ItemsView/ValuesView seam (copied onto wrappers by Store): keys come lazily from the caller's store in batches of 100, are resolved with dol.inner_most_key, read with one BatchGetItem, and store[key] is answered from them only during that lookup, in the calling thread only (same design as perf: SQLAlchemyPersister items()/values() use one query (#11) sqldol#12), so codecs, filt_iter scopes and prefix/partition scoping still apply. Keys the batch did not return (UnprocessedKeys...) fall back to GetItem; a ValidationException falls back; other backend errors propagate. Snapshot per batch.
  • Tests with a paginating, call-counting fake table (+ batch_get_item).

Adversarial review (Opus; also checked the request against botocore's service model with a Stubber): request shape, reserved-word keys, UnprocessedKeys fallback, unordered responses, Decimal keys, empty-page pagination, error classes of the partition module, dataclass attrs all fine. Confirmed and fixed: _chunks listed the whole store first (a full COUNT scan plus all keys before the first pair) -> now lazy; the batch was instance state shared across threads (could be left behind) -> thread-local. Known/minor: a batch DynamoDB rejects outright (e.g. a data field that is also a key field) silently degrades to single GETs.

Follow-ups filed: #6 (relax scan_items refusal), #7 (DynamoDbStore decision, recommendation: no).

🤖 Generated with Claude Code

thorwhalen and others added 2 commits October 13, 2026 14:12
…primary_key hook (#4)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@thorwhalen
thorwhalen merged commit c4b314b into master Oct 13, 2026
12 checks passed
@thorwhalen
thorwhalen deleted the feat/pagination-bulk-views-4 branch October 13, 2026 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move key/value transforms into dol hooks (_id_of_key/_obj_of_data) and land DynamoDbStore

1 participant