proposal: bounded wallet sync scheduling, concurrent reads, and portable snapshot performance

[ty-everett]

Problem and users

Sister issue to https://github.com/p2ppsr/metanet-client-desktop/pull/42. Desktop is shipping local/provider backups and BRC-38/39 file UX against the published Toolbox line. The operator accepts the current scheduling/concurrency and performance limitations for that PR; this issue stages the subsequent TS Stack work and a later consumer PR. It is not a reason to hold Desktop's functional release.

Today a safe copy can hold WalletStorageManager.runAsSync across the entire network transfer. Desktop shares this access queue with readers/writers to prevent unsafe overlap during session replacement and conflict reconciliation. Large copies can consequently delay foreground wallet operations for minutes. Simply removing the client lock would trade latency for inconsistent primary selection, conflicting writes and checkpoint races.

The current portable helpers also materialize complete datasets/JSON/byte arrays, have no streaming progress/cancellation contract, and read tables sequentially without owning a consistent database snapshot. StorageClient does not expose the lower-level StorageProvider interface or a complete authenticated BRC-38 snapshot API. Desktop's agreed initial export source is its consistently captured, synchronized local copy; it must label that source and checkpoint honestly. It does not claim to export the remote provider's complete operational sync-state history.

Proposed end state

1. Make safe yielding a Toolbox capability

• Refactor WalletStorageManager.syncToWriter, syncFromReader, updateBackups and primary reconciliation into bounded, resumable steps. Expose an async iterator or equivalent coordinator API with a durable acknowledged checkpoint, explicit completion state, byte/row progress and cancellation signal.
• Limit exclusive ownership to each atomic destination merge/checkpoint commit and the short primary-selection fence. Do not hold a global writer lock while waiting for remote I/O, serialization, encryption or retry backoff.
• Allow snapshot-isolated reads while background copying progresses where the provider supports it. Serialize conflicting wallet writes and proof-monitor writes; schedule foreground work before the next background page, with aging/quotas so the background copy also makes progress.
• Fence sessions/generations across manager replacement and primary changes. Drain or reject stale queued work; never let a queued operation resume against an obsolete writable primary. Make reentrancy explicit rather than allowing accidental nested-lock deadlocks.
• Define a bounded safe cancellation point and distinguish cancel requested, acknowledged page committed, and stopped. On a lost acknowledgement, recover from the destination's durable checkpoint; never advance based only on the sender's progress log.
• Expose provider capabilities for isolation, pagination and concurrency. Preserve the serialized fallback for older providers; consumers should no longer need an application-wide mutex to maintain correctness.

2. Reduce storage and transport work

• Use stable keyset cursors, a fixed snapshot/high-water mark and indexed identity/entity/update-key predicates. Avoid OFFSET scans, repeated full-table counts, and full sync-map serialization on every page. Include tie-breakers for same-timestamp records and explicit tombstone coverage.
• Project metadata separately from raw transaction/proof blobs. Fetch large payloads only when needed; reuse content-addressed, verified blobs instead of repeatedly serializing/copying them. Bound every queue and in-flight byte budget.
• Negotiate raw binary transfer only when both peers support it, retain base64-JSON interoperability, and preserve authenticated transport and integrity checks. Avoid binary-to-number-array-to-JSON-to-base64 round trips. Evaluate bounded read-ahead/double buffering while keeping ordered atomic destination commits and memory backpressure.
• Desktop real-wallet testing exposed multi-gigabyte heap amplification from aggregate exports retaining expanded number[] binary payloads. Keep IDB binary payloads as packed typed buffers, encode/decode one row or bounded page at a time, avoid whole-file JavaScript strings, and bound native IPC buffering. Desktop now uses a schema-pinned packed adapter and bounded staging merge pages; upstream should provide this as a supported versioned API.
• Adapt chunk size from encoded bytes and observed latency, not row count alone. Cache totals per snapshot; coalesce duplicate reads; use capped retry/backoff without repeatedly reloading unchanged successful pages.
• Prevent adaptive page sizing from collapsing into persistently tiny pages when fixed query, authentication, or network overhead dominates. The current SyncPageBudget uses floor(records * 5000 / elapsedMs) across the whole read/commit cycle; Desktop observed long proof transfers shrink to single-record pages. Separate fixed overhead from per-byte/row cost, use smoothed measurements and hysteresis, and allow bounded upward probes under the negotiated byte/memory cap. Benchmark high fixed latency, large proof rows, growing OFFSET cost and transitions between entity tables; a smaller page must not multiply total work without improving foreground latency or cancellation.
• Optimize IDB/SQLite adapters for the actual queries: read-only snapshot transactions, bulk operations, necessary compound indexes, and appropriate SQLite WAL/busy handling. Benchmark before choosing parallelism or tuning caches; do not multiply writers to an unsupported database.

3. Complete snapshot and portable archive APIs

• Add a capability-negotiated, authenticated single-user snapshot/export API to StorageServer/StorageClient. Bind immutable snapshot handles to authenticated identity, network, schema/version and source storage, with expiry, bounded cursor reads and cleanup.
• Include the full BRC-38 closure and original source metadata/sync-state rows from one coherent snapshot, including concurrent monitor/other-device writes. Do not substitute a replica's new checkpoints for source records. Refuse unsupported requests rather than return a partial archive presented as complete.
• Provide streaming BRC-38 canonical output and bounded-memory BRC-39 encryption/decryption/file processing while preserving exact formats, NFC password rules, Argon2id strength and authenticated validation. Support worker/native backends, transferable buffers, progress and cancellation without adding an incompatible wallet-specific envelope.
• Add an import plan/preview API, isolated staging, semantic verification, ID remapping, repeat-import behavior and a durable commit/recovery journal. Preserve original provenance and operationally sensitive sync/primary values without automatically activating them.
• Preserve legacy managed-change policy values in the imported archive; document any later operational policy migration separately. BRC-38 v1 intentionally excludes pending action batches, auxiliary runtime tables and storage-global logs. Export/import must clearly state that scope and must not require those batches to settle or silently include them. Any future format extension is separate standards work.
• Keep one identity/network/profile per export, import and closure. A wallet with several profiles exports each separately; single-profile consumers such as Peacock handle one profile. Never combine another profile's records, local database binding, primary selection or synchronization checkpoints. Consumers may share device-level backup destination/automatic-sync preferences, but each participating profile must authenticate, synchronize and retain recovery state independently. Keep main storage and restore activation profile-specific.
• Handle nullable optional properties in legacy structured histories consistently: canonical BRC-38 omits absent optional object properties. Never silently drop array entries or alter meaningful values. Add conformance vectors for deployed histories and exact, non-mutating source capture.

Affected packages, services, and implementations

packages/wallet/wallet-toolbox: WalletStorageManager, StorageProvider/StorageIdb/StorageKnex, StorageClientBase/StorageClient, StorageServer, sync entities/checkpoints and storage/portable. SDK storage contracts/capability declarations; browser/mobile entrypoints and @bsv/expo-wallet-toolbox adapters where applicable. Consumers: Metanet Desktop first, then Peacock and Metanet Explorer Mobile. BSV Desktop's native storage/monitor separation and BSV Browser's restore-before-background-backup lifecycle are useful reference patterns, not evidence of file-format conformance.

Compatibility and migration

Stage additive capability negotiation and contract/conformance tests first. Old client/new server and new client/old server must keep existing correctness, cancellation and restart behavior. Do not make the unpublished raw transport/schema candidate a prerequisite for the already accepted Desktop release. If indexes or persistence need migration, document schema versions, resumable upgrade, mixed-version behavior and downgrade constraints; never reset existing data. Keep BRC-38/39/40 wire semantics stable unless a separately reviewed standard revision is necessary.

Security and operational impact

Snapshot/import handles must be authorized per identity and network, bounded by resource quotas/expiry, and never reveal another user's data. Validate before allocation/activation, bound file/KDF resource costs, retain authenticated encryption and transport, and keep passwords/archives/identity/payloads out of telemetry. No new CORS bypass or credential broadening. Roll out source-owned client/server releases with provider compatibility probes and the host infrastructure's normal availability gates. Retain serialized fallback and a forward rollback path.

Validation plan and staged delivery

1. Publish the coordinator/snapshot invariants and baseline traces before changing locks. Specify lock ownership, commit boundaries, session fencing, cancellation and checkpoint recovery as executable contracts.
2. Add provider isolation and indexed paging tests; implement the scheduler with fault injection (disconnect, timeout after commit, late acknowledgement, corrupted part, monitor write, same-timestamp boundary, primary change, process kill/restart). Verify no lost/duplicated logical data and no two writable primary sessions.
3. Benchmark each improvement separately, then together: IDB, SQLite and remote; small/large wallets and large individual records; cold full copy, incremental copy, no-change repeat, export/import; slow/high-latency/disconnected networks. Record wall time, wire bytes, CPU, peak memory, query counts, queue wait, time holding locks, cancellation latency and foreground p50/p95/p99 latency under load. Publish reproducible fixtures/hardware and baseline comparisons, not unsupported overall-speed claims from one component benchmark.
4. Establish budgets from those baselines. Acceptance must show foreground requests finish between background commit windows, bounded memory/in-flight bytes independent of total wallet size, fair continued background progress, quiescent unchanged copies, and a documented cancellation bound excluding a separately bounded in-flight I/O deadline.
5. Run BRC-38/39 independent reader/writer vectors and complete-table/relationship/tombstone/binary verification, repeat merge, interrupted commit, wrong identity/network/password, corrupted/truncated archives, and an explicitly authorized funded restore/spend. Include native macOS/Windows/Linux plus mobile suspension/disk pressure. Preserve key-recovery versus data-recovery distinctions.
6. Publish released packages and migration/compatibility notes, validate deployed providers, then open the follow-on Desktop PR to replace its coarse queue and local-snapshot workaround. Port the shared behavior to Peacock/Mobile with explicit network identity handling.

Related existing issues: #542 (sync result error handling) and #543 (output basket ID remapping) should be covered by the import/sync correctness gates, not hidden by performance changes.

• Searched open issues and the current portable/sync implementation for overlap.
• No breaking API/wire/persistence change is assumed acceptable merely for performance.