Sync both stores, with the exclusion lists they actually need

[mspinola]

ADR-0007 split the data across two stores. The sync scripts still mirrored one, so bars written to $MARKETDATA_STORE reached neither the Mac nor the dash — a producer running green every night while half its output stayed on the Windows box.

Why a second pass with the same flags would have been worse than nothing

The two stores disagree about the one file that matters:

	COT store	bar store
live bookkeeping	manifests/<half>.json	manifest.json (root)
root manifest.json	dead legacy aggregate — exclude	the only index — must be carried

Both robocopy /XF and rsync --exclude match by name at any depth, so copying the COT store's exclusion list onto the bar store strips its whole index in transit and delivers a replica full of parquet it cannot enumerate. Every file is present, so disk usage and a directory listing both look right; only a read notices.

This repo already documents that exact trap one directory over, for vintage/snapshots.json. So: two passes, two lists, per transport.

• sync-store.cmd — two robocopy /MIR passes. Pass 2 runs even when pass 1 fails (the stores are independent; aborting early lets a COT hiccup silently stop bars), and the exit is the worse of the two codes.
• push-to-server.cmd — two rsync pairs: data with --delete, then the manifest on its own without --delete, so the mirror cannot remove the replica's copy in the window before the new one lands.
• pull-store.sh — the same split on the consumer-pull side.

The freshness check needed two signals, not one

verify-replicas.sh goes from two checks to four. This is the part that would have been wrong if copied across:

• COT. cotdata rewrites status.json on every run, new data or not → "mtime is today" is a clean test.
• Bars. marketdata has no status.json and rewrites manifest.json only when a bar is actually written. A weekend, a holiday, or a deferred --require-final run legitimately writes nothing, so demanding "today" there fails every Saturday — and a weekly false alarm stops being read by the second month. Bars get a staleness window instead (BAR_MAX_AGE_DAYS, default 4: Friday's write is still fresh on Tuesday).

Both transports preserve timestamps, so a replica's mtime is the producer's write time. That is what makes either test mean anything — and it buys a sharper one for free: both replicas should carry identical mtimes, so a mismatch means one push is behind even while both sit inside the window and neither looks wrong alone.

sync_preflight.py half-worked on a bar store, which is worse than failing

It did not error. The manifest loaded through the legacy per-domain fallback, so the summary looked plausible — while the on-disk orphan check globbed a flat prices/ and never descended into bars/<domain>/<source>/, reporting zero orphans no matter what a mirror would delete.

Layout is now detected per store. Being handed one of each exits 2 with CANNOT JUDGE rather than guessing: that pairing is far likelier to be two swapped paths than an intention, and the mirror it would green-light deletes the entire destination.

Verification

These are the files whose whole job is to notice, so they now have tests:

• tests/test_sync_preflight.py — 12 tests: layout detection, the nested-bars/ orphan regression, cross-layout refusal, and the original checks unchanged.
• tests/test_verify_replicas.py — 8 tests running the real bash script with ssh and both --check binaries stubbed on PATH. Covers the case that motivated this: a current COT store beside a stale bar store, green under the old two-check version and red under this one.

219 passed, ruff clean. The .cmd files could not be executed — there is no cmd in this sandbox — so their control flow is reasoned from cmd's parse-time expansion rules, same caveat as #109. What I checked rather than remembered: marketdata's store layout, raw_root(), and that manifest.json is written by _touch_manifest only on an actual write, all read from the marketdata source.

Also in here

• Markers renamed so none is a prefix of another — REPLACE_WITH_LOCAL_COT was a prefix of REPLACE_WITH_LOCAL_COTDATA_STORE, which quietly corrupts a find-and-replace.
• verify-replicas.sh epoch formatting made portable: date -r reads a number on BSD and a filename on GNU.

---

Generated by Claude Code