Repository navigation
Fast update and synching #56
Description
See this recipe to copy data over from src to targ:
src = ... # make source store
targ = ... # make target store
targ.update(src)Note that unless a specialized mechanism is implemented this update will be performed by looping through src.items(), and write all items in targ, regardless of the existence of the key. With many items this can be quite inefficient.
targ could implement a specialized efficient mechanism for the update method, but this would only affect the "writing to targ" part in the general case (if our update method remains insensitive to the particulars of src). Part of the inefficiencies come from iterating through src, so (since targ has control here) we could implement targ.update in such a way that it uses some special features of src to coordinate an efficient copying of data.
In sshdol we have a ssh_files.sync_to method to be able to perform an efficient synching of remote (ssh files) to local files (where the ssh_files instance is made). This is a way to circumvent the fact that to make local_files.update(ssh_files) we'd need to implement a specialized local_files class that knows how ssh_files work. A clean design is not clear here, and may not be worth it just to maintain the update interface, avoiding an extra method name. But a design could be envisioned whereby all base dol Stores have a mechanism that enables coordination between both sides of the update to be implemented.
Activity
thorwhalen commentedon Jul 2, 2026
Triage (2026-07):
priority: low, Wave 3.targ.update(src)currently falls back to the genericMutableMappingloop (iteratesrc, write each item) — slow for large/remote stores. A clean solution needs a duck-typed "fast-sync" protocol:targ.updatedetects whethersrc(and itself) expose an optimized transfer (e.g. rsync-backedsshdol) and negotiates it, while preserving the plain Mapping interface for the general case. Design still open — no source change yet. Seemisc/docs/dol_issues_report.md(#71).thorwhalen commentedon Sep 22, 2026
Update: #93 (2026-08-21) did the concrete verification this issue was asking for, under its "The
update()case (#56) — what we verified" section. Summary for anyone landing here first:Storenever overridesupdate; theMutableMappingmixin runs per-itemself[k] = vthrough the wrapped__setitem__-- transform-correct, but never bulk.self.store.update(...)delegation inbase.pywould bypass all transforms -- correctly left disabled.updateis found by MRO before__getattr__delegation).cached_keys' generatedupdate(forwards per item, batch-maintains its cache once) is the in-tree proof that bulk ops need layer cooperation, not a single override.misc/docs/dol_roadmap.mdalongside Beyond codecs: layer kinds for filters/caches in the flat model, and fast-op hook placement #93's layer-kind work.No action needed here beyond this pointer -- #93 is now the live design home for this.