From 636636989e8be016cdbe7f0bf2a3150ea353e0d9 Mon Sep 17 00:00:00 2001 From: "Matt S." Date: Fri, 21 Aug 2026 00:18:57 +0000 Subject: [PATCH] Sync both stores, with the exclusion lists they actually need MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-0007 split the data across two stores. The sync scripts still mirrored one, so bars written to $MARKETDATA_STORE reached neither the Mac nor the dash — a producer running green every night while half its output stayed on the Windows box. The fix is not "add a second pass with the same flags". The two stores disagree about the one file that matters: COT store manifests/.json is live; root manifest.json is a dead legacy aggregate, and excluding it is correct. bar store root manifest.json is the ONLY index; there is no manifests/. Both robocopy /XF and rsync --exclude match by NAME AT ANY DEPTH, so carrying the COT list over to the bar store strips its whole index in transit and delivers a replica full of parquet it cannot enumerate. Every file is present, so size and a directory listing both look right; only a read notices. That is the same trap this repo already documents one directory over, for vintage/snapshots.json. So: two passes, two lists, per transport. sync-store.cmd two robocopy /MIR passes; pass 2 runs even when pass 1 fails, and the exit is the worse of the two codes push-to-server.cmd two rsync pairs (data --delete, then the manifest on its own without --delete, so the mirror cannot remove the replica's copy before the new one lands) pull-store.sh the same split on the consumer-pull side verify-replicas.sh goes from two checks to four, and the two stores need two different freshness signals — this is the part that would have been wrong if copied across. cotdata rewrites status.json on EVERY run, so "mtime is today" is a clean test. marketdata has no status.json and rewrites manifest.json only when a bar is actually written, so a weekend, a holiday or a deferred --require-final run legitimately writes nothing: demanding "today" there would fail every Saturday, and a weekly false alarm stops being read by the second month. Bars get a staleness window instead (BAR_MAX_AGE_DAYS, default 4). Both transports preserve timestamps, so a replica's mtime is the PRODUCER's write time. That is what makes either test mean anything, and it buys a sharper one for free: both replicas should carry identical mtimes, so a mismatch means one push is behind even while both sit inside the window and neither looks wrong alone. sync_preflight.py learns both layouts. It did not fail loudly on a bar store — it half-worked, which is worse: the manifest loaded through the legacy fallback so the summary looked plausible, while the on-disk check globbed a flat prices/ and never descended into bars///, reporting zero orphans no matter what a mirror would delete. Layout is now detected per store, and being handed one of each exits 2 rather than guessing — that pairing is far likelier to be two swapped paths than an intention, and the mirror it would green-light deletes the destination. Tests, because these are the files whose whole job is to notice: test_sync_preflight.py 12 tests — layout detection, the nested-bars orphan regression, cross-layout refusal test_verify_replicas.py 8 tests — runs the real bash script with ssh and both --check binaries stubbed on PATH Also renamed the markers so none is a prefix of another (REPLACE_WITH_LOCAL_COT was a prefix of REPLACE_WITH_LOCAL_COTDATA_STORE, which quietly breaks a find-and-replace), and made verify-replicas.sh's epoch formatting portable — `date -r` reads a number on BSD and a filename on GNU. 219 passed, ruff clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2 --- docs/SYNCING.md | 158 ++++++++++++++------ docs/WINDOWS_SCHEDULING.md | 2 +- docs/examples/mac/pull-store.sh | 87 ++++++++--- docs/examples/mac/verify-replicas.sh | 127 ++++++++++++---- docs/examples/sync_preflight.py | 159 +++++++++++++++----- docs/examples/windows/push-to-server.cmd | 116 ++++++++++++--- docs/examples/windows/sync-store.cmd | 94 +++++++++--- tests/test_sync_preflight.py | 179 +++++++++++++++++++++++ tests/test_verify_replicas.py | 156 ++++++++++++++++++++ 9 files changed, 916 insertions(+), 162 deletions(-) create mode 100644 tests/test_sync_preflight.py create mode 100644 tests/test_verify_replicas.py diff --git a/docs/SYNCING.md b/docs/SYNCING.md index c456e98..66c2572 100644 --- a/docs/SYNCING.md +++ b/docs/SYNCING.md @@ -6,19 +6,37 @@ the producer and the consumers are different machines, something has to move the This page is about **what** to move and what to leave behind. The transport is the easy part and comes last. -> [!NOTE] -> **Two stores since ADR-0007.** Every bar — Norgate, databento and Yahoo alike — moved to +> [!IMPORTANT] +> **Two stores since ADR-0007, and they do not share an exclusion list.** Every bar — +> Norgate, databento and Yahoo alike — moved to > [`crucible-marketdata`](https://pypi.org/project/crucible-marketdata/) and its own -> `$MARKETDATA_STORE`. The Windows box still produces both and still pushes both to the same -> replicas, so the topology, exclusions, auth gotchas and preflight advice below apply -> unchanged — there are now **two source directories to mirror instead of one**, and the -> `bars/`, `metadata/` and `_raw/` rows describe the bar store rather than this one. Where a -> command names `cotdata-prices --prices`, read +> `$MARKETDATA_STORE`. The Windows box produces both and pushes both to the same replicas, +> so the topology, transports and auth gotchas below are unchanged. What is **not** +> unchanged is what you exclude: +> +> | | COT store | bar store | +> |---|---|---| +> | live bookkeeping | `manifests/.json` | `manifest.json` (root) | +> | root `manifest.json` | dead legacy aggregate — **exclude** | the only index — **must be carried** | +> | `_cache/` | cotdata's CFTC zip cache — exclude | does not exist | +> | `citpy/` | consumer-owned — exclude | does not exist | +> | `_raw/` | pre-ADR-0007 leftover — exclude | databento's PAID bronze — exclude | +> | `vintage/` | irreplaceable, carry to the Mac | does not exist | +> +> The `manifest.json` row is the trap. Both `robocopy /XF` and `rsync --exclude` match by +> **name at any depth**, so carrying the COT store's exclusion list over to the bar store +> strips the bar store's whole index in transit and delivers a replica full of parquet it +> cannot enumerate. That is the same failure this page already documents for +> `vintage/snapshots.json`, one directory over. Mirror the two stores in **separate passes**; +> do not point one `robocopy /MIR` at a shared parent folder. +> +> Where a command names `cotdata-prices --prices`, read > `marketdata-update --bars --domain futures --require-final`. > > A cotdata store built before the move still has `prices/`, `metadata/` and `_raw/` sitting > in it. Nothing writes them any more; leaving them is harmless and deleting them is safe > once the bar store is confirmed synced. + ## This deployment: one Norgate producer, two replicas A single Windows server is the only producer (Norgate prices, CFTC COT). It feeds two @@ -37,10 +55,13 @@ folder was deliberately not carried over, so a real network sync replaced it. - **Producer:** the Windows server (Norgate prices, CFTC COT), one-directional. - **Consumer:** the Mac, read-only. -- **Transport:** `robocopy /MIR` ([`examples/windows/sync-store.cmd`](examples/windows/sync-store.cmd)) - pushing to an SMB share the Mac exports for `~/code/cotdata_store`, reached from the - server as `\\\cotdata_store` (use the Mac's LAN IP if its name will not resolve - from a headless server). +- **Transport:** two `robocopy /MIR` passes, one per store + ([`examples/windows/sync-store.cmd`](examples/windows/sync-store.cmd)), pushing to SMB + shares the Mac exports for `~/code/cotdata_store` and `~/code/marketdata_store`, reached + from the server as `\\\cotdata_store` and `\\\marketdata_store` (use the Mac's + LAN IP if its name will not resolve from a headless server). The second pass runs even + when the first fails: the stores are independent, so aborting early would let a COT + hiccup silently stop bars reaching the Mac for as long as it lasted. - **Trigger:** chained onto the end of the producer task behind an `errorlevel` guard, so it fires only after a successful run rather than on a timer (a deferred `--require-final` bar run exits non-zero and is skipped). @@ -63,10 +84,12 @@ maintenance than a per-symbol roll-rule table. - **Transport:** an `rsync` push over SSH, chained onto the producer task, using a packaged rsync on Windows (cwRsync or WSL). robocopy cannot speak SSH, and SMB must never be exposed over the internet, so the Mac's SMB path does not carry here. See - [`examples/windows/push-to-server.cmd`](examples/windows/push-to-server.cmd). The - exclusions match the Mac push (`_cache/`, `_raw/`, `citpy/`, `manifest.json`, plus - `*.tmp` for partial-write temps), so the producer-internal databento bronze under - `_raw/databento/` never leaves the Windows box. + [`examples/windows/push-to-server.cmd`](examples/windows/push-to-server.cmd), which + pushes both stores to two remote paths. Per store the exclusions match the Mac push — + `_cache/`, `_raw/`, `citpy/`, `manifest.json`, `*.tmp` for the COT store; `_raw/` and + `*.tmp` only for the bar store, whose root `manifest.json` is pushed last on its own + line rather than excluded. Either way the PAID databento bronze under `_raw/databento/` + never leaves the Windows box. - **Auth:** key-based SSH only. A scheduled task cannot type a passphrase, so use a dedicated key with `ssh -o BatchMode=yes`, never a password prompt. - **cwRsync gotcha:** a Cygwin rsync (what `choco install rsync` gives you) must drive the @@ -121,31 +144,42 @@ about two producers writing the *same* files. This is the part that matters, and on a real store it is most of the bytes. -Per store, since there are now two. `bars/`, `metadata/` and `_raw/` live in -`$MARKETDATA_STORE`; everything else here is `$COTDATA_STORE`. +**One table per store.** They are not interchangeable — see the `manifest.json` rows. -| Directory | Sync? | Why | +`$COTDATA_STORE`: + +| Entry | Sync? | Why | |---|---|---| -| `bars/` (marketdata) | **yes** | the data | -| `metadata/` (marketdata) | **yes** | contract specs | | `cot_legacy/`, `cot_disagg/`, `cot_tff/` | **yes** | the data | -| `prices/` | **NO** (legacy) | pre-ADR-0007 leftover; bars live in the marketdata store now | -| `manifests/` | **yes** | per-half bookkeeping | -| `status.json` | yes | the producer's own view, useful on the replica | +| `manifests/` | **yes** | per-half bookkeeping, disjoint and mergeable | +| `vintage/` | **yes**, to the Mac | irreplaceable; see below for why the dash skips it | +| `status.json` | yes | the producer's own view, and the freshness signal a replica check reads | +| `manifest.json` | **NO** | legacy aggregate, nothing writes it (see below) | | `_cache/` | **NO** | cotdata's own download cache of CFTC source zips, producer-internal, free to rebuild | -| `_raw/` | **NO** | databento's append-only PAID raw store, producer-internal (marketdata's now) | +| `prices/`, `metadata/` | **NO** (legacy) | pre-ADR-0007 leftovers; bars and specs live in the bar store now | +| `_raw/` | **NO** (legacy) | pre-ADR-0007 leftover, databento's PAID raw store (marketdata's now) | | anything a consumer added by hand | **NO** | no producer creates it, so a mirror deletes it (see below) | -| `manifest.json` | **NO** | legacy aggregate, nothing writes it (see below) | + +`$MARKETDATA_STORE`: + +| Entry | Sync? | Why | +|---|---|---| +| `bars/` | **yes** | the data — `bars///_.parquet` | +| `metadata/` | **yes** | contract specs | +| `manifest.json` | **YES** | the bar store's ONLY index. Not the COT store's legacy file — carry it, and carry it LAST | +| `_raw/` | **NO** | databento's append-only PAID raw store, producer-internal | +| `_cache/`, `citpy/`, `vintage/` | n/a | the bar store has none of these | On one real store the `_cache/` and `_raw/` exclusions dropped the payload from 270 MB to -about 82 MB. The other two are correctness issues rather than savings. +about 82 MB. The `manifest.json` rows are correctness rather than savings, and they point +opposite ways: excluding it is right for one store and destroys the other. ### `_cache/` holds source archives, not derived data cotdata's own CFTC providers write it: `_cache/cot_legacy`, `_cache/cot_disagg` and `_cache/cot_tff` hold the downloaded year zips (`dea_fut_xls_2004.zip` and so on) that -`--cot-*` HEAD-checks to decide whether anything changed. `_cache/databento` is the -equivalent for that provider. +`--cot-*` HEAD-checks to decide whether anything changed. (A store built before ADR-0007 +also has a `_cache/databento` from when that provider lived here; nothing writes it now.) It is producer-internal and **free** to rebuild, since the CFTC download costs nothing. Do not confuse it with `_raw/`, which is the *paid* databento raw store. Both are @@ -173,15 +207,23 @@ directory left the store. If you have anything similar, exclude it today and move it out of the store. Then no future transport, and no colleague configuring one, can reach it. -### `manifest.json` is legacy +### `manifest.json` is legacy — **in the COT store only** -Nothing writes it any more (see ADR-0007). It held both producer halves in one file, -which is exactly the shape a file sync resolves last-writer-wins: a producer pushing an -aggregate containing only its own half would silently drop the other half's entries on -arrival. The per-half files under `manifests/` are disjoint and merge correctly. +Nothing writes cotdata's aggregate any more (see ADR-0007). It held both producer halves +in one file, which is exactly the shape a file sync resolves last-writer-wins: a producer +pushing an aggregate containing only its own half would silently drop the other half's +entries on arrival. The per-half files under `manifests/` are disjoint and merge correctly. Run `cotdata-update --migrate-manifests` once per store, then delete `manifest.json`. +**The bar store's `manifest.json` is the opposite of legacy.** marketdata keeps one live +manifest at its root and has no `manifests/` directory at all. Excluding it — by copying +the COT store's exclusion list, which is the obvious thing to do — delivers a replica +holding every parquet and no index. It is a quiet failure: the files are all there, so +disk usage and a directory listing both look right, and only a read notices. Both +transports match exclusions by **name at any depth**, so there is no `/manifests/` prefix +to make the rule safe. Two stores, two passes, two lists. + ### `vintage/` is irreplaceable, so where it is WRITTEN matters The vintage tree (`vintage/raw/`, `observations/`, `revisions/`, `snapshots.json`) records @@ -203,7 +245,7 @@ Per replica in this deployment: | Target | Carries `vintage/`? | Why | |---|---|---| | Mac (research) | **Yes, in full** | Natural second copy of irreplaceable bytes, ~1 GB/yr, and research may query revisions | -| Linux dash VPS | **No** | cot-analyzer reads prices and COT only; it would carry ~1 GB/yr of archives it never opens | +| Linux dash VPS | **No** | cot-analyzer reads bars and COT only; it would carry ~1 GB/yr of archives it never opens | **Naming gotcha, already handled:** the vintage provenance index is `snapshots.json`, not `manifest.json`. Both sync scripts exclude `manifest.json` *unanchored* (robocopy `/XF` @@ -225,11 +267,20 @@ python docs/examples/sync_preflight.py SRC_STORE DEST_STORE Exit 0 means DEST holds nothing SRC does not produce. Exit 1 lists what a mirror would remove and refuses. It reads only. +Run it **once per pair** — `$COTDATA_STORE` against its target, then `$MARKETDATA_STORE` +against its own. It detects which layout each store is (a bar store has `bars/` and a live +root `manifest.json`; a COT store has `manifests/`) and prints the verdict. Handing it one +of each exits 2 with `CANNOT JUDGE` rather than guessing: that pairing is far more likely +to be two paths swapped than an intention, and the mirror it would otherwise green-light +deletes the entire destination. + It checks two things the eye does not. **Entries only on DEST**, which a mirror deletes. -And **the same key produced by different sources on each side**: cotdata's price path is +And **the same key produced by different sources on each side**: cotdata's price path was `prices/_.parquet` with no source component, so a Norgate `ES_backadj` and a -databento `ES_backadj` are the same file, and a sync resolves them last-writer-wins with -nothing in the output to say so. +databento `ES_backadj` were the same file, and a sync resolved them last-writer-wins with +nothing in the output to say so. In the bar store that particular collision cannot happen +— the vendor is a directory — but the single-table domains (`metadata/contract_specs`) +still carry no source in their path, so the check runs there too. On a real pair on 2026-07-26 that second check found **94 collisions** between a Norgate-sourced research store and a databento-sourced server store. Neither store was @@ -258,6 +309,11 @@ If a manifest arrives before the parquet it describes, a consumer briefly sees a pointing at data that has not landed. The other order is harmless: data present but not yet announced. +Both example transports do this per store: the COT push holds back `manifests/`, and the +bar push holds back the root `manifest.json`, each sent on a final pass **without** +`--delete` so the mirror cannot remove the replica's copy in the window before the new one +lands. + In practice this is a nicety rather than a hazard, because `get_cot` read parquet directly and the manifest is status only. But if your transport lets you control ordering, sync the data directories first and `manifests/` last. @@ -312,10 +368,11 @@ unattended, at the cost of a daemon on both machines. ## Verifying a sync worked -On the replica: +On the replica, once per store: ```bash -cotdata-update --check +cotdata-update --check # reads $COTDATA_STORE +marketdata-update --check # reads $MARKETDATA_STORE; exits 1 on an empty store ``` Compare `newest data` and `last write (UTC)` against the producer's output. The lag @@ -327,7 +384,24 @@ mirroring and stale files are accumulating. If it shows *fewer*, the sync has no completed or an exclusion is too broad. To automate this across replicas, [`examples/mac/verify-replicas.sh`](examples/mac/verify-replicas.sh) -checks a local store and a remote one (over SSH) in one pass and exits non-zero if either -did not receive today's push. `status.json` is rewritten on every producer push, so its -mtime date is the "did this replica update today" signal. Wire it to launchd/cron a little +checks a local store and a remote one (over SSH) in one pass — **four checks, two stores +per replica** — and exits non-zero naming every laggard. Wire it to launchd/cron a little after the producer's run. + +**The two stores need two different freshness signals**, and the difference is not a +detail you can round off: + +- **COT.** cotdata rewrites `status.json` on *every* run, new data or not, so "mtime is + today" is a clean did-this-replica-update-today test. +- **Bars.** marketdata has no `status.json`, and it rewrites `manifest.json` only when a + bar or a spec is actually written. A weekend, a holiday, or a deferred `--require-final` + run legitimately writes nothing, so demanding "today" here fails every Saturday — and an + alarm that cries wolf weekly stops being read by the second month. The script uses a + staleness **window** instead (`BAR_MAX_AGE_DAYS`, default 4: Friday's write is still + fresh on Tuesday). + +Both transports preserve timestamps, so a replica's mtime is the *producer's* write time +rather than the copy time — which is what makes either test mean anything, and what lets +the script also compare the **two replicas against each other**. A matching producer run +lands the same mtime on both; different mtimes mean one push is behind, and inside the +staleness window neither replica looks wrong on its own. diff --git a/docs/WINDOWS_SCHEDULING.md b/docs/WINDOWS_SCHEDULING.md index 3a0fef2..624ce31 100644 --- a/docs/WINDOWS_SCHEDULING.md +++ b/docs/WINDOWS_SCHEDULING.md @@ -28,7 +28,7 @@ Create **two** wrapper scripts — they run *different* commands from *different > **Ready-made templates:** copy [`docs/examples/windows/run-prices.cmd`](examples/windows/run-prices.cmd) and [`run-cot.cmd`](examples/windows/run-cot.cmd) out of the repo into your `` (e.g. `C:\Users\you\cotdata\scheduler\`) rather than retyping them — then just fill in the placeholders. Keep them outside the repo so a `git pull` never clobbers your edited paths. -> **Fill in your real paths.** Inside the `.cmd` files, overwrite the plain-text markers `REPLACE_WITH_STORE_PATH` (your synced store, e.g. `\\Mac\code\cotdata_store`) and `REPLACE_WITH_VENV_PATH` (your virtualenv, e.g. `C:\Users\you\code\cotdata\.venv`). **Don't use angle-bracket placeholders like `` inside a `.cmd`** — cmd reads `<` and `>` as redirection and the script fails with "The syntax of the command is incorrect," even on `REM` comment lines. The `` notation in the *task commands* further down is fine to substitute since those are quoted or typed at the prompt. +> **Fill in your real paths.** Inside the `.cmd` files, overwrite the plain-text markers — `REPLACE_WITH_STORE_PATH` (your synced store, e.g. `\\Mac\code\cotdata_store`) and `REPLACE_WITH_VENV_PATH` (your virtualenv, e.g. `C:\Users\you\code\cotdata\.venv`) in the producer wrappers, and the per-store `REPLACE_WITH_COTDATA_*` / `REPLACE_WITH_MARKETDATA_*` pairs in [`sync-store.cmd`](examples/windows/sync-store.cmd) and [`push-to-server.cmd`](examples/windows/push-to-server.cmd), which mirror **two** stores since ADR-0007 (see [SYNCING.md](SYNCING.md)). No marker is a prefix of another, so a find-and-replace is safe. **Don't use angle-bracket placeholders like `` inside a `.cmd`** — cmd reads `<` and `>` as redirection and the script fails with "The syntax of the command is incorrect," even on `REM` comment lines. The `` notation in the *task commands* further down is fine to substitute since those are quoted or typed at the prompt. `run-prices.cmd` — bars (with `--require-final`, so it runs only once Norgate's **Final** prices are in, not interim bars). Note `MARKETDATA_STORE`: a *different* directory from `COTDATA_STORE`, not an alias for it. diff --git a/docs/examples/mac/pull-store.sh b/docs/examples/mac/pull-store.sh index 6aa1da1..606f51d 100755 --- a/docs/examples/mac/pull-store.sh +++ b/docs/examples/mac/pull-store.sh @@ -1,33 +1,55 @@ #!/usr/bin/env bash -# Pull the cotdata store from the producer machine onto a read-only replica. +# Pull BOTH stores from the producer machine onto a read-only replica. # The consumer decides when to fetch, so this is the alternative to the producer # pushing (see examples/windows/sync-store.cmd). # +# Since ADR-0007 the producer writes two stores and this script pulls both: +# COT store CFTC positioning (cotdata-cot --cot-all) +# bar store bars + specs (marketdata-update --bars / --metadata) +# Separate passes with separate exclusions. They disagree about manifest.json -- +# legacy in one, the only index in the other -- so one list cannot serve both. +# # Copy this next to your launchd/cron config and overwrite the markers below: -# REPLACE_WITH_PRODUCER_HOST = ssh target e.g. matt@windows-box -# REPLACE_WITH_REMOTE_STORE = store ON that host e.g. /c/Users/matt/cotdata_store -# REPLACE_WITH_LOCAL_STORE = store on THIS mac e.g. /Users/you/code/cotdata_store +# REPLACE_WITH_PRODUCER_HOST = ssh target e.g. matt@windows-box +# REPLACE_WITH_REMOTE_COTDATA_STORE = COT store ON that host e.g. /c/Users/matt/cotdata_store +# REPLACE_WITH_LOCAL_COTDATA_STORE = COT store on THIS mac e.g. /Users/you/code/cotdata_store +# REPLACE_WITH_REMOTE_MARKETDATA_STORE = bar store ON that host e.g. /c/Users/matt/code/marketdata_store +# REPLACE_WITH_LOCAL_MARKETDATA_STORE = bar store on THIS mac e.g. /Users/you/code/marketdata_store +# The marker names match verify-replicas.sh deliberately, and none is a prefix of +# another: a find-and-replace over the shorter name would otherwise mangle the longer. # (Plain-text markers, not angle-bracket placeholders: an unedited <...> would be # read as a shell redirection.) # # rsync must exist on BOTH ends. On Windows that means WSL or a packaged rsync. With # only OpenSSH, `scp -r` works and the payload is small, but you lose deletion # handling. See docs/SYNCING.md. -set -euo pipefail +# +# Note there is no `set -e`. Both stores are pulled even when the first fails: they +# are independent, and aborting early would let a COT hiccup silently stop bars +# reaching this Mac for as long as it lasted. The exit code below still reports it. +set -uo pipefail + +HOST="REPLACE_WITH_PRODUCER_HOST" +COT_SRC="$HOST:REPLACE_WITH_REMOTE_COTDATA_STORE" +COT_DEST="REPLACE_WITH_LOCAL_COTDATA_STORE" +BAR_SRC="$HOST:REPLACE_WITH_REMOTE_MARKETDATA_STORE" +BAR_DEST="REPLACE_WITH_LOCAL_MARKETDATA_STORE" -SRC="REPLACE_WITH_PRODUCER_HOST:REPLACE_WITH_REMOTE_STORE" -DEST="REPLACE_WITH_LOCAL_STORE" +rc=0 +# ── COT store ─────────────────────────────────────────────────────────────── # What each exclusion is for: -# _cache, _raw producer-internal, most of the bytes. _cache is cotdata's cache -# of downloaded CFTC zips. _raw is a pre-ADR-0007 leftover (databento's -# paid bronze store, now marketdata's) — still excluded. +# _cache cotdata's cache of downloaded CFTC zips: producer-internal and +# free to rebuild, and most of the bytes. +# _raw pre-ADR-0007 leftover (databento's paid bronze store, now +# marketdata's). Kept so a store built before the move still +# excludes it. # citpy consumer-owned, not written by any producer, so --delete removes # it and no producer run brings it back. Kept as a backstop: such # files belong outside the store. See docs/SYNCING.md. # manifest.json legacy aggregate, nothing writes it, and it is the one file a # sync resolves last-writer-wins across both producer halves -EXCLUDES=( +COT_EXCLUDES=( --exclude '_cache/' --exclude '_raw/' --exclude 'citpy/' @@ -36,11 +58,38 @@ EXCLUDES=( # Two passes so a manifest never arrives before the data it describes. Harmless if # reversed (readers open parquet directly), but free to get right. -rsync -az --delete "${EXCLUDES[@]}" --exclude 'manifests/' "$SRC/" "$DEST/" -rsync -az "$SRC/manifests/" "$DEST/manifests/" - -# Confirm what landed. Compare against the producer's own --check output: the lag -# column is measured on WRITE time, so an entry the producer skipped shows as behind -# even when its data looks fine. -export COTDATA_STORE="$DEST" -cotdata-update --check +rsync -az --delete "${COT_EXCLUDES[@]}" --exclude 'manifests/' "$COT_SRC/" "$COT_DEST/" || rc=$? +rsync -az "$COT_SRC/manifests/" "$COT_DEST/manifests/" || rc=$? + +# ── bar store ─────────────────────────────────────────────────────────────── +# A DIFFERENT list, and the difference is load-bearing: +# +# manifest.json is the bar store's ONLY index -- marketdata keeps one file at the +# store root, not a manifests/ directory. It is held out of the --delete pass so +# the local copy is not removed before the new one lands, then pulled on its own +# line. Reusing COT_EXCLUDES here would have excluded it outright, leaving a bar +# store this Mac cannot enumerate. rsync --exclude matches by NAME AT ANY DEPTH, +# the same trap docs/SYNCING.md documents for vintage/snapshots.json. +# +# _cache and citpy are absent from the bar store: no marketdata provider writes a +# download cache, and citpy is a cotdata-store consumer artefact. _raw IS +# excluded -- databento's append-only PAID raw store +# ($MARKETDATA_DATABENTO_RAW, else _raw/databento under the bar store) -- because +# a replica has no use for it and re-fetching costs money (ADR-0006). +BAR_EXCLUDES=( + --exclude '_raw/' +) + +rsync -az --delete "${BAR_EXCLUDES[@]}" --exclude 'manifest.json' "$BAR_SRC/" "$BAR_DEST/" || rc=$? +rsync -az "$BAR_SRC/manifest.json" "$BAR_DEST/manifest.json" || rc=$? + +# Confirm what landed, per store. Compare against the producer's own --check +# output: cotdata's lag column is measured on WRITE time, so an entry the producer +# skipped shows as behind even when its data looks fine. +COTDATA_STORE="$COT_DEST" cotdata-update --check || rc=$? +echo +# `marketdata-update --check` exits 1 on an EMPTY store, which is the failure this +# whole change exists to catch: bars that never arrive at all. +MARKETDATA_STORE="$BAR_DEST" marketdata-update --check || rc=$? + +exit "$rc" diff --git a/docs/examples/mac/verify-replicas.sh b/docs/examples/mac/verify-replicas.sh index d9fdea6..63836af 100755 --- a/docs/examples/mac/verify-replicas.sh +++ b/docs/examples/mac/verify-replicas.sh @@ -1,54 +1,127 @@ #!/usr/bin/env bash -# verify-replicas.sh — confirm the latest producer run reached BOTH store replicas -# TODAY: one local store and one remote store read over SSH. +# verify-replicas.sh — confirm the latest producer run reached BOTH replicas and +# BOTH stores: one local (Mac, over SMB) and one remote (dash VPS, over SSH), each +# holding a COT store and a bar store since ADR-0007. # -# How it works: the producer rewrites `status.json` on every push, and the sync -# carries it, so if a replica's status.json mtime date == the day this runs, that -# replica received today's push. Run it AFTER the producer's scheduled run, e.g. -# from launchd/cron. Exits 0 on PASS (both current), 1 on FAIL (names the laggard). +# Four checks, not two. Bars moved to their own store with their own sync pass, so +# a COT-only check would pass green while no bar has reached a replica in weeks -- +# which is exactly the failure this file was extended to catch. +# +# Run it AFTER the producer's scheduled run, e.g. from launchd/cron. +# Exits 0 on PASS, 1 on FAIL (naming every laggard, not just the first). +# +# ── The two stores need two different freshness signals ───────────────────── +# COT store: cotdata rewrites `status.json` on EVERY run, new data or not. So +# "status.json mtime is today" is a clean did-this-replica-update-today signal. +# Bar store: marketdata has no status.json, and it rewrites `manifest.json` only +# when a bar or a spec is actually WRITTEN. A weekend, a holiday, or a deferred +# `--require-final` run legitimately writes nothing, so demanding "today" here +# would fail every Saturday. The check is a staleness WINDOW instead +# (BAR_MAX_AGE_DAYS, default 4 — Friday's write is still fresh on Tuesday). +# Both syncs preserve timestamps (rsync -a, robocopy), so a replica's mtime is the +# PRODUCER's write time, not the copy time. That is what makes either check mean +# anything, and it is what makes the cross-check at the end possible. # # Configure by editing the block below or exporting the vars before calling. # Markers are plain text (not ) so an unedited copy still parses. set -uo pipefail # ── config ────────────────────────────────────────────────────────────────── -LOCAL_STORE="${LOCAL_STORE:-REPLACE_WITH_LOCAL_STORE}" # e.g. $HOME/code/cotdata_store -LOCAL_CHECK="${LOCAL_CHECK:-REPLACE_WITH_LOCAL_COTDATA_UPDATE}" # cotdata-update path, e.g. .../.venv/bin/cotdata-update -REMOTE="${REMOTE:-REPLACE_WITH_REMOTE}" # ssh target, e.g. deploy@dash.example.com -REMOTE_STORE="${REMOTE_STORE:-REPLACE_WITH_REMOTE_STORE}" # e.g. /srv/cotdata_store -REMOTE_CHECK="${REMOTE_CHECK:-REPLACE_WITH_REMOTE_COTDATA_UPDATE}" # cotdata-update path ON the remote -SSH_KEY="${SSH_KEY:-}" # optional: private key; empty = default key/agent +LOCAL_COT="${LOCAL_COT:-REPLACE_WITH_LOCAL_COTDATA_STORE}" # e.g. $HOME/code/cotdata_store +LOCAL_BARS="${LOCAL_BARS:-REPLACE_WITH_LOCAL_MARKETDATA_STORE}" # e.g. $HOME/code/marketdata_store +LOCAL_COT_CHECK="${LOCAL_COT_CHECK:-REPLACE_WITH_LOCAL_COTDATA_UPDATE}" # .../.venv/bin/cotdata-update +LOCAL_BAR_CHECK="${LOCAL_BAR_CHECK:-REPLACE_WITH_LOCAL_MARKETDATA_UPDATE}" # .../.venv/bin/marketdata-update +REMOTE="${REMOTE:-REPLACE_WITH_REMOTE_SSH_TARGET}" # ssh target, e.g. deploy@dash.example.com +REMOTE_COT="${REMOTE_COT:-REPLACE_WITH_REMOTE_COTDATA_STORE}" # e.g. /srv/cotdata_store +REMOTE_BARS="${REMOTE_BARS:-REPLACE_WITH_REMOTE_MARKETDATA_STORE}" # e.g. /srv/marketdata_store +REMOTE_COT_CHECK="${REMOTE_COT_CHECK:-REPLACE_WITH_REMOTE_COTDATA_UPDATE}" # cotdata-update ON the remote +REMOTE_BAR_CHECK="${REMOTE_BAR_CHECK:-REPLACE_WITH_REMOTE_MARKETDATA_UPDATE}" # marketdata-update ON the remote +BAR_MAX_AGE_DAYS="${BAR_MAX_AGE_DAYS:-4}" # see the note above +SSH_KEY="${SSH_KEY:-}" # optional: private key; empty = default key/agent # ──────────────────────────────────────────────────────────────────────────── TODAY=$(date +%F) +NOW=$(date +%s) SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8) [ -n "$SSH_KEY" ] && SSH_OPTS+=(-i "$SSH_KEY") pass=0; fail=0 -check_date() { # $1 = status.json mtime date (YYYY-MM-DD), $2 = label - if [ "$1" = "$TODAY" ]; then - echo " PASS: $2 status.json written today ($1)"; pass=$((pass + 1)) + +# date -r prints the file's mtime on both macOS (BSD) and Linux (GNU). +mtime_local() { date -r "$1" +%s 2>/dev/null; } +mtime_remote() { ssh "${SSH_OPTS[@]}" "$REMOTE" "date -r '$1' +%s" 2>/dev/null; } + +# Format an EPOCH, and note this is a different flag on each platform: BSD reads +# `date -r` as either a file or a seconds count, GNU reads it as a file only and +# wants `date -d @seconds` for the number. This file lives under mac/ but the +# fallback costs one line and stops it breaking the day someone runs it on Linux. +fmt_epoch() { date -r "$1" "${2:-+%F}" 2>/dev/null || date -d "@$1" "${2:-+%F}" 2>/dev/null; } + +check_today() { # $1 = mtime epoch (may be empty), $2 = label + local when; when=$([ -n "$1" ] && fmt_epoch "$1" || echo "") + if [ "$when" = "$TODAY" ]; then + echo " PASS: $2 status.json written today ($when)"; pass=$((pass + 1)) + else + echo " FAIL: $2 status.json last written '${when:-unknown}', not today ($TODAY)"; fail=$((fail + 1)) + fi +} + +check_age() { # $1 = mtime epoch (may be empty), $2 = label + if [ -z "$1" ]; then + echo " FAIL: $2 manifest.json missing or unreadable — no bars have reached this replica" + fail=$((fail + 1)); return + fi + local age_days when + age_days=$(( (NOW - $1) / 86400 )) + when=$(fmt_epoch "$1") + if [ "$age_days" -le "$BAR_MAX_AGE_DAYS" ]; then + echo " PASS: $2 manifest.json written $when (${age_days}d old, window ${BAR_MAX_AGE_DAYS}d)" + pass=$((pass + 1)) else - echo " FAIL: $2 status.json last written '${1:-unknown}', not today ($TODAY)"; fail=$((fail + 1)) + echo " FAIL: $2 manifest.json written $when — ${age_days}d old, past the ${BAR_MAX_AGE_DAYS}d window" + fail=$((fail + 1)) fi } -echo "=== cotdata replica verification — $TODAY ===" +echo "=== store replica verification — $TODAY ===" echo -echo "[local] $LOCAL_STORE" -# date -r prints the file's mtime on both macOS (BSD) and Linux (GNU). -check_date "$(date -r "$LOCAL_STORE/status.json" +%F 2>/dev/null)" "local" -COTDATA_STORE="$LOCAL_STORE" "$LOCAL_CHECK" --check 2>&1 | sed 's/^/ /' +echo "[local COT] $LOCAL_COT" +check_today "$(mtime_local "$LOCAL_COT/status.json")" "local COT" +COTDATA_STORE="$LOCAL_COT" "$LOCAL_COT_CHECK" --check 2>&1 | sed 's/^/ /' echo -echo "[remote] $REMOTE:$REMOTE_STORE" -check_date "$(ssh "${SSH_OPTS[@]}" "$REMOTE" "date -r '$REMOTE_STORE/status.json' +%F" 2>/dev/null)" "remote" -ssh "${SSH_OPTS[@]}" "$REMOTE" "COTDATA_STORE=$REMOTE_STORE $REMOTE_CHECK --check" 2>&1 | sed 's/^/ /' +echo "[local bars] $LOCAL_BARS" +L_BAR_MTIME=$(mtime_local "$LOCAL_BARS/manifest.json") +check_age "$L_BAR_MTIME" "local bar" +# --check exits 1 on an EMPTY bar store, the failure this file exists to catch. +MARKETDATA_STORE="$LOCAL_BARS" "$LOCAL_BAR_CHECK" --check 2>&1 | sed 's/^/ /' echo +echo "[remote COT] $REMOTE:$REMOTE_COT" +check_today "$(mtime_remote "$REMOTE_COT/status.json")" "remote COT" +ssh "${SSH_OPTS[@]}" "$REMOTE" "COTDATA_STORE=$REMOTE_COT $REMOTE_COT_CHECK --check" 2>&1 | sed 's/^/ /' +echo +echo "[remote bars] $REMOTE:$REMOTE_BARS" +R_BAR_MTIME=$(mtime_remote "$REMOTE_BARS/manifest.json") +check_age "$R_BAR_MTIME" "remote bar" +ssh "${SSH_OPTS[@]}" "$REMOTE" "MARKETDATA_STORE=$REMOTE_BARS $REMOTE_BAR_CHECK --check" 2>&1 | sed 's/^/ /' +echo + +# Cross-check, free and strictly sharper than either window on its own. Both syncs +# preserve timestamps, so BOTH replicas should carry the producer's own mtime, +# identical to the second. If they differ, one push is behind the other — and that +# stays true inside the staleness window, where neither replica looks wrong alone. +if [ -n "$L_BAR_MTIME" ] && [ -n "$R_BAR_MTIME" ] && [ "$L_BAR_MTIME" != "$R_BAR_MTIME" ]; then + echo " WARN: the two bar replicas hold DIFFERENT manifest mtimes" + echo " local $(fmt_epoch "$L_BAR_MTIME" '+%F %T')" + echo " remote $(fmt_epoch "$R_BAR_MTIME" '+%F %T')" + echo " One of the two bar pushes is behind. Both syncs preserve timestamps," + echo " so a matching producer run should land the same mtime on both." + echo +fi -if [ "$fail" -eq 0 ] && [ "$pass" -eq 2 ]; then - echo "RESULT: PASS — both replicas received today's producer push." +if [ "$fail" -eq 0 ] && [ "$pass" -eq 4 ]; then + echo "RESULT: PASS — both replicas hold a current COT store and a current bar store." exit 0 else - echo "RESULT: FAIL — $fail replica(s) did not update today. Check the producer task and its chained sync." + echo "RESULT: FAIL — $fail of 4 checks did not pass. Check the producer tasks and their chained syncs." exit 1 fi diff --git a/docs/examples/sync_preflight.py b/docs/examples/sync_preflight.py index a3195dc..b4895b7 100644 --- a/docs/examples/sync_preflight.py +++ b/docs/examples/sync_preflight.py @@ -5,7 +5,23 @@ `SRC` is the producer, `DEST` the replica about to be mirrored onto. Answers one question: **would a `--delete` mirror destroy something DEST owns?** -Exit 0 = safe. Exit 1 = do not sync, with the reason. +Exit 0 = safe. Exit 1 = do not sync, with the reason. Exit 2 = could not judge. + +Works on either store, and tells them apart +------------------------------------------- +Since ADR-0007 this deployment mirrors two stores with two different shapes, and +the difference is not cosmetic: + +* a **cotdata** store keeps per-half bookkeeping in ``manifests/.json`` and + its root ``manifest.json`` is a dead legacy aggregate; +* a **marketdata** store keeps one live ``manifest.json`` at the root, and its + data lives under ``bars///``. + +Reading a marketdata store with cotdata's rules half-works, which is the bad kind +of wrong: the manifest loads (via the legacy fallback) so the summary looks +plausible, while the on-disk check never descends into ``bars/`` and therefore +reports zero orphaned files no matter how many there are. So the layout is +detected per store, and comparing one of each is refused rather than guessed at. Why this exists --------------- @@ -16,12 +32,18 @@ The case this was written for, found on a live pair on 2026-07-26: a Mac store holding 94 Norgate-sourced price entries, while a 25-hour databento ingest ran -against the same store. cotdata's price path is `prices/_.parquet` with +against the same store. cotdata's price path was `prices/_.parquet` with **no source component**, so once Stage 2 built those 41 databento symbols they would have landed on the same paths Norgate already owned, and all 41 were a subset of Norgate's 47. Mirroring from the Norgate producer would have deleted or overwritten them. Neither store was wrong; the topology was. +That exact collision cannot recur in a marketdata store — the vendor is a +directory (`bars/futures/norgate/` beside `bars/futures/databento/`), so two +vendors cannot contend for one path. The check still runs there, because the +single-table domains (`metadata/contract_specs.parquet`) have no source in their +path and can still collide. + Reads only. Never writes, never deletes, never syncs. """ from __future__ import annotations @@ -30,33 +52,82 @@ import sys from pathlib import Path -HALVES = ("cot", "prices") -# Directories a producer owns. Anything else under the root is either excluded by -# the sync (see SYNCING.md) or consumer-owned and out of scope here. -DATA_DIRS = ("prices", "metadata", "cot_legacy", "cot_disagg", "cot_tff") +COTDATA, MARKETDATA = "cotdata", "marketdata" + +COT_HALVES = ("cot", "prices") +# Directories a cotdata producer owns. Anything else under the root is either +# excluded by the sync (see SYNCING.md) or consumer-owned and out of scope here. +# `prices` and `metadata` are pre-ADR-0007 leftovers, still listed so a store that +# has not been cleaned up yet is still checked rather than silently skipped. +COT_DATA_DIRS = ("prices", "metadata", "cot_legacy", "cot_disagg", "cot_tff") -def load_manifest(root: Path) -> dict: - """Merge the per-half manifests, falling back to the legacy aggregate per DOMAIN. +def _read_json(path: Path) -> dict: + try: + data = json.loads(path.read_text()) + except (OSError, ValueError): + return {} + return data if isinstance(data, dict) else {} - Per domain, not per half: a store can be part migrated, and a half file existing - does not mean every domain in it is present. + +def detect_layout(root: Path) -> str: + """Which package produced this store. + + Structural evidence first (a directory that only one layout has), manifest + keys second, and cotdata as the default so a store that predates the split + keeps its old reading. """ + if (root / "bars").is_dir(): + return MARKETDATA + if (root / "manifests").is_dir(): + return COTDATA + return MARKETDATA if "bars" in _read_json(root / "manifest.json") else COTDATA + + +def load_manifest(root: Path, layout: str) -> dict: + """Domain -> {entry: record}, with the non-dict store-level flags dropped. + + marketdata keeps one manifest and no halves. cotdata merges the per-half files + and falls back to the legacy aggregate per DOMAIN, not per half: a store can be + part migrated, and a half file existing does not mean every domain in it is + present. + """ + root_json = _read_json(root / "manifest.json") + if layout == MARKETDATA: + # schema_version / universe_is_point_in_time sit beside the domains. + return {d: e for d, e in root_json.items() if isinstance(e, dict)} + merged: dict = {} - legacy_path = root / "manifest.json" - legacy = json.loads(legacy_path.read_text()) if legacy_path.exists() else {} - for half in HALVES: - p = root / "manifests" / f"{half}.json" - if p.exists(): - for domain, entries in json.loads(p.read_text()).items(): - if isinstance(entries, dict): - merged.setdefault(domain, {}).update(entries) - for domain, entries in legacy.items(): + for half in COT_HALVES: + for domain, entries in _read_json(root / "manifests" / f"{half}.json").items(): + if isinstance(entries, dict): + merged.setdefault(domain, {}).update(entries) + for domain, entries in root_json.items(): if isinstance(entries, dict) and domain not in merged: merged[domain] = entries return merged +def data_files(root: Path, layout: str) -> dict: + """{directory label: {file key}} for the parquet actually on disk. + + Keys are relative to the directory, so a marketdata bar file is + ``futures/norgate/ES_backadj.parquet`` — the vendor stays in the key, which is + what makes a same-name-different-vendor pair two entries here rather than one. + """ + if layout == MARKETDATA: + out = {} + bars = root / "bars" + if bars.is_dir(): + out["bars"] = {str(f.relative_to(bars)) for f in bars.rglob("*.parquet")} + meta = root / "metadata" + if meta.is_dir(): + out["metadata"] = {f.name for f in meta.glob("*.parquet")} + return out + return {d: {f.name for f in (root / d).glob("*.parquet")} + for d in COT_DATA_DIRS if (root / d).is_dir()} + + def sources(manifest: dict) -> dict: """{(domain, key): source} across every domain.""" return {(d, k): (e or {}).get("source") @@ -67,7 +138,7 @@ def sources(manifest: dict) -> dict: def main(argv=None) -> int: argv = argv if argv is not None else sys.argv[1:] if len(argv) != 2: - print(__doc__.strip().splitlines()[2].strip()) + print("usage: python sync_preflight.py SRC_STORE DEST_STORE") return 2 src, dest = Path(argv[0]).expanduser(), Path(argv[1]).expanduser() for label, root in (("SRC", src), ("DEST", dest)): @@ -75,19 +146,33 @@ def main(argv=None) -> int: print(f"{label} is not a directory: {root}") return 2 - s_man, d_man = load_manifest(src), load_manifest(dest) + s_layout, d_layout = detect_layout(src), detect_layout(dest) + print(f"SRC {src} [{s_layout}]") + print(f"DEST {dest} [{d_layout}]\n") + if s_layout != d_layout: + # Not a judgement call to make on the user's behalf: mirroring a COT store + # onto a bar store would delete the whole of the other one, and it is far + # more likely to be two swapped paths than an intention. + print(f"CANNOT JUDGE: SRC is a {s_layout} store and DEST is a {d_layout} " + f"store. These are different layouts holding different data, and a " + f"mirror between them would delete everything on DEST.\n" + f"Since ADR-0007 each store syncs to its OWN target — check the two " + f"paths, then run this once per pair.") + return 2 + + layout = s_layout + s_man, d_man = load_manifest(src, layout), load_manifest(dest, layout) s_src, d_src = sources(s_man), sources(d_man) problems = [] - print(f"SRC {src}") - print(f"DEST {dest}\n") - print(f"{'domain':<12} {'src':>6} {'dest':>6} {'dest-only':>9} sources") + width = max([len("domain")] + [len(d) for d in set(s_man) | set(d_man)]) + print(f"{'domain':<{width}} {'src':>6} {'dest':>6} {'dest-only':>9} sources") for domain in sorted(set(s_man) | set(d_man)): s_e = s_man.get(domain, {}) if isinstance(s_man.get(domain), dict) else {} d_e = d_man.get(domain, {}) if isinstance(d_man.get(domain), dict) else {} only = set(d_e) - set(s_e) src_names = {v for (dom, _), v in d_src.items() if dom == domain and v} - print(f"{domain:<12} {len(s_e):>6} {len(d_e):>6} {len(only):>9} " + print(f"{domain:<{width}} {len(s_e):>6} {len(d_e):>6} {len(only):>9} " f"{', '.join(sorted(src_names)) or '-'}") if only: problems.append( @@ -99,22 +184,22 @@ def main(argv=None) -> int: clashes = sorted(k for k in set(s_src) & set(d_src) if s_src[k] and d_src[k] and s_src[k] != d_src[k]) if clashes: + where = ("cotdata's price path carries no source component" + if layout == COTDATA else + "a marketdata single-table domain carries no source component") problems.append( (f"{len(clashes)} entries are produced by DIFFERENT sources on each side. " - f"cotdata's price path carries no source component, so these collide on " - f"the same file and the sync resolves them last-writer-wins", + f"{where}, so these collide on the same file and the sync resolves them " + f"last-writer-wins", [f"{d}/{k}: src={s_src[(d, k)]} dest={d_src[(d, k)]}" for d, k in clashes[:8]])) - # Files on disk with no manifest entry: a mirror deletes these too. - for d in DATA_DIRS: - sp, dp = src / d, dest / d - if dp.is_dir(): - s_files = {f.name for f in sp.glob("*.parquet")} if sp.is_dir() else set() - d_files = {f.name for f in dp.glob("*.parquet")} - only = d_files - s_files - if only: - problems.append((f"{d}/: {len(only)} parquet files exist only on DEST", - sorted(only)[:8])) + # Files on disk with no counterpart on SRC: a mirror deletes these too. + s_files, d_files = data_files(src, layout), data_files(dest, layout) + for d, names in sorted(d_files.items()): + only = names - s_files.get(d, set()) + if only: + problems.append((f"{d}/: {len(only)} parquet files exist only on DEST", + sorted(only)[:8])) print() if not problems: diff --git a/docs/examples/windows/push-to-server.cmd b/docs/examples/windows/push-to-server.cmd index c49a6ba..17ec2eb 100644 --- a/docs/examples/windows/push-to-server.cmd +++ b/docs/examples/windows/push-to-server.cmd @@ -1,8 +1,14 @@ @echo off -REM cotdata store push to the REMOTE Linux dash server, over rsync + SSH. -REM Chained after the producer task (like sync-store.cmd) so it fires at a +REM Store push to the REMOTE Linux dash server, over rsync + SSH — BOTH stores. +REM Chained after the producer tasks (like sync-store.cmd) so it fires at a REM known-consistent moment rather than on a timer that might land mid-run. REM +REM Since ADR-0007 this box produces two stores and this script pushes both: +REM $COTDATA_STORE CFTC positioning (cotdata-cot --cot-all) +REM $MARKETDATA_STORE bars + specs (marketdata-update --bars / --metadata) +REM Separate passes, separate exclusions -- the two stores disagree about +REM manifest.json (see the bar-store block below), so one list cannot serve both. +REM REM Why not robocopy here: robocopy cannot speak SSH, and the dash server is a REM remote VPS, so SMB is off the table (never expose SMB over the internet). REM This uses rsync, which needs a packaged rsync ON WINDOWS. cwRsync (a Cygwin @@ -32,10 +38,12 @@ REM REPLACE_WITH_SSH_KEY_CYG = batch SSH private key, cygdrive form REM e.g. /cygdrive/c/Users/you/.ssh/cotdata_push REM REPLACE_WITH_KNOWN_HOSTS_CYG= a writable known_hosts, cygdrive form REM e.g. /cygdrive/c/Users/you/.ssh/known_hosts -REM REPLACE_WITH_STORE_PATH_CYG = source store, cygdrive form +REM REPLACE_WITH_COTDATA_STORE_CYG = source COT store, cygdrive form REM e.g. /cygdrive/c/Users/you/cotdata_store -REM REPLACE_WITH_REMOTE = user@host:/path/to/store (no trailing slash) -REM e.g. deploy@dash.example.com:/srv/cotdata_store +REM REPLACE_WITH_MARKETDATA_STORE_CYG = source bar store, cygdrive form +REM e.g. /cygdrive/c/Users/you/code/marketdata_store +REM REPLACE_WITH_COT_REMOTE = user@host:/path/to/cotdata_store (no trailing slash) +REM REPLACE_WITH_BAR_REMOTE = user@host:/path/to/marketdata_store (no trailing slash) REM See docs/SYNCING.md ("Dash store") for the exclusions and the one-time cutover. setlocal @@ -43,22 +51,44 @@ set "RSYNC=C:\ProgramData\chocolatey\bin\rsync.exe" set "SSH_EXE=REPLACE_WITH_SSH_EXE_CYG" set "KEY=REPLACE_WITH_SSH_KEY_CYG" set "KNOWN=REPLACE_WITH_KNOWN_HOSTS_CYG" -set "SRC=REPLACE_WITH_STORE_PATH_CYG" -set "DEST=REPLACE_WITH_REMOTE" +set "COT_SRC=REPLACE_WITH_COTDATA_STORE_CYG" +set "COT_DEST=REPLACE_WITH_COT_REMOTE" +set "BAR_SRC=REPLACE_WITH_MARKETDATA_STORE_CYG" +set "BAR_DEST=REPLACE_WITH_BAR_REMOTE" set "SSH=%SSH_EXE% -i %KEY% -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=%KNOWN%" +set "RC=0" +call :push_cot +call :push_bars + +REM Both stores are attempted even when the first fails, deliberately: they are +REM independent, and aborting early would let a COT hiccup silently stop bars +REM reaching the dash for as long as it lasted -- one broken thing presenting as +REM two. rsync exits 0 on success and non-zero on error, which Task Scheduler +REM already treats as failure, so no robocopy-style normalisation is needed. +REM If both fail, RC carries the bar push's code -- the echo lines above name +REM each store separately, so the log is not ambiguous about which broke. +if not "%RC%"=="0" exit /b %RC% +echo push ok ^(cot + bars^) +exit /b 0 + + +REM =========================================================================== +:push_cot REM Data first, manifests last, so a manifest never announces parquet that has not REM landed (harmless if reversed; readers open parquet directly). --delete makes REM this a true mirror. The exclusions match the Mac push: -REM _cache, _raw producer-internal. _raw/databento (the paid databento bronze) is a -REM pre-ADR-0007 leftover here, owned by marketdata now -REM rides under _raw and so is excluded, per ADR-0006. +REM _cache cotdata's download cache of CFTC source zips, producer-internal +REM and free to rebuild. +REM _raw pre-ADR-0007 leftover -- databento's paid bronze store, owned by +REM marketdata now. Kept so a store built before the move still +REM excludes it. REM citpy consumer-owned on the server; excluding it from --delete is REM what stops the mirror from wiping it. REM manifest.json legacy aggregate, resolved last-writer-wins across halves. REM *.tmp, *.part a producer's partial-write temps (atomic write via os.replace, REM and in-flight raw downloads); never propagate a half-written file. -REM vintage/ NOT pushed here, unlike the Mac sync. cot-analyzer reads prices +REM vintage/ NOT pushed here, unlike the Mac sync. cot-analyzer reads bars REM and COT, never the vintage tree, so the dash would carry roughly REM 1 GB/year of raw CFTC archives it never opens. The Mac keeps the REM second copy instead. Drop this exclusion if something on the dash @@ -68,14 +98,62 @@ REM "vintage/raw/" so the small derived tables still ride along --exclude "_cache/" --exclude "_raw/" --exclude "citpy/" ^ --exclude "manifest.json" --exclude "*.tmp" --exclude "*.part" ^ --exclude "manifests/" --exclude "vintage/" ^ - -e "%SSH%" "%SRC%/" "%DEST%/" -if %ERRORLEVEL% NEQ 0 ( echo push FAILED, rsync code %ERRORLEVEL% & exit /b %ERRORLEVEL% ) + -e "%SSH%" "%COT_SRC%/" "%COT_DEST%/" +if errorlevel 1 goto :cot_failed REM manifests/ last, without --delete: the per-half files are disjoint and merge. -"%RSYNC%" -az -e "%SSH%" "%SRC%/manifests/" "%DEST%/manifests/" -if %ERRORLEVEL% NEQ 0 ( echo manifests push FAILED, rsync code %ERRORLEVEL% & exit /b %ERRORLEVEL% ) +"%RSYNC%" -az -e "%SSH%" "%COT_SRC%/manifests/" "%COT_DEST%/manifests/" +if errorlevel 1 goto :cot_failed +echo cot push ok +goto :eof -REM rsync exits 0 on success and non-zero on error, which Task Scheduler already -REM treats as failure, so no robocopy-style exit-code normalisation is needed. -echo push ok -exit /b 0 +:cot_failed +REM Capture the code BEFORE echoing it: an intervening command can clear +REM ERRORLEVEL, and the point of this line is to carry rsync's own code out. +set "RC=%ERRORLEVEL%" +echo COT push FAILED, rsync code %RC% +goto :eof + + +REM =========================================================================== +:push_bars +REM The bar store, with its OWN exclusions. Two differences from the COT push, +REM both load-bearing: +REM +REM manifest.json is the bar store's ONLY index -- marketdata keeps one file at +REM the store root, not a manifests/ directory. It is excluded from the --delete +REM pass so the server's copy is not removed before the new one lands, then +REM pushed on its own line afterwards. Reusing the COT push's exclusion list +REM here would have excluded it outright and delivered the dash a bar store it +REM cannot enumerate. rsync --exclude matches by name at any depth, the same +REM trap docs/SYNCING.md documents for vintage/snapshots.json. +REM +REM _cache and citpy are absent from the bar store -- no marketdata provider +REM writes a download cache, and citpy is a cotdata-store consumer artefact -- +REM so listing them would be misleading rather than merely redundant. _raw IS +REM excluded: it is databento's append-only PAID raw store +REM ($MARKETDATA_DATABENTO_RAW, else _raw/databento under the bar store), which +REM a replica has no use for and which costs money to re-fetch (ADR-0006). +REM +REM Note there is no vendor collision to guard against here: marketdata puts the +REM source in the path (bars/futures/norgate/ beside bars/futures/databento/), so +REM two vendors cannot contend for one file the way they could under cotdata's old +REM prices/_.parquet. Push whichever vendors the dash should read. +"%RSYNC%" -az --delete ^ + --exclude "_raw/" --exclude "*.tmp" --exclude "*.part" ^ + --exclude "manifest.json" ^ + -e "%SSH%" "%BAR_SRC%/" "%BAR_DEST%/" +if errorlevel 1 goto :bar_failed + +REM The manifest last, on its own, for the same reason manifests/ goes last above. +"%RSYNC%" -az -e "%SSH%" "%BAR_SRC%/manifest.json" "%BAR_DEST%/manifest.json" +if errorlevel 1 goto :bar_failed +echo bar push ok +goto :eof + +:bar_failed +REM Capture the code BEFORE echoing it: an intervening command can clear +REM ERRORLEVEL, and the point of this line is to carry rsync's own code out. +set "RC=%ERRORLEVEL%" +echo bar push FAILED, rsync code %RC% +goto :eof diff --git a/docs/examples/windows/sync-store.cmd b/docs/examples/windows/sync-store.cmd index 4c3be7b..7e8e1bd 100644 --- a/docs/examples/windows/sync-store.cmd +++ b/docs/examples/windows/sync-store.cmd @@ -1,27 +1,47 @@ @echo off -REM cotdata store sync wrapper for Windows Task Scheduler. +REM Store sync wrapper for Windows Task Scheduler — BOTH stores. REM Run this AFTER the producer tasks, not on its own timer, so it fires at a REM known-consistent moment rather than possibly mid-run. REM -REM Copy this file into your scheduler folder and overwrite the two markers below. +REM Since ADR-0007 this box produces two stores and this script mirrors both: +REM $COTDATA_STORE CFTC positioning (cotdata-cot --cot-all) +REM $MARKETDATA_STORE bars + specs (marketdata-update --bars / --metadata) +REM They are mirrored in SEPARATE passes with SEPARATE exclusions. Do not merge +REM them into one robocopy over a shared parent: the two stores disagree about +REM manifest.json (see the pass-2 note below), and one exclusion list cannot be +REM right for both. +REM +REM Copy this file into your scheduler folder and overwrite the four markers below. REM Do NOT put angle brackets in a .cmd file: cmd reads them as redirection and the REM file fails with "The syntax of the command is incorrect" even on comment lines. -REM REPLACE_WITH_STORE_PATH = your local store e.g. C:\Users\you\cotdata_store -REM REPLACE_WITH_DEST_PATH = the sync target e.g. Z:\cotdata_store or -REM \\mac\code\cotdata_store or a Syncthing folder +REM COT_SRC = your local COT store e.g. C:\Users\you\cotdata_store +REM COT_DEST = its sync target e.g. \\mac\code\cotdata_store +REM BAR_SRC = your local bar store e.g. C:\Users\you\code\marketdata_store +REM BAR_DEST = its sync target e.g. \\mac\code\marketdata_store REM See docs/SYNCING.md for what each exclusion is for. setlocal +set "COT_SRC=REPLACE_WITH_COTDATA_STORE_PATH" +set "COT_DEST=REPLACE_WITH_COTDATA_DEST_PATH" +set "BAR_SRC=REPLACE_WITH_MARKETDATA_STORE_PATH" +set "BAR_DEST=REPLACE_WITH_MARKETDATA_DEST_PATH" + +REM --------------------------------------------------------------------------- +REM Pass 1 — the COT store. +REM REM /MIR mirrors (copies new + deletes removed). /XD excludes directories: -REM _cache, _raw producer-internal, ~70%% of the bytes. _cache is cotdata's -REM cache of downloaded CFTC zips. _raw is a pre-ADR-0007 leftover -REM (databento's paid bronze store, now marketdata's) — still excluded. +REM _cache cotdata's cache of downloaded CFTC zips: producer-internal and +REM free to rebuild, and most of the bytes. +REM _raw pre-ADR-0007 leftover (databento's paid bronze store, now +REM marketdata's). Kept so a store built before the move still +REM excludes it; a fresh cotdata store has no such directory. REM citpy consumer-owned, not written by any producer, so /MIR removes it REM and no producer run brings it back. Kept as a backstop: such REM files belong outside the store. See docs/SYNCING.md. REM /XF excludes the legacy aggregate manifest: nothing writes it, and it is the one -REM file a sync would resolve last-writer-wins across two halves. +REM file a sync would resolve last-writer-wins across two halves. cotdata's live +REM bookkeeping is the per-half files under manifests/, which are disjoint. REM vintage/ IS carried here, deliberately and in full (including vintage/raw). Those REM bytes are irreplaceable -- CFTC serves current state only, so a lost vintage cannot REM be re-fetched -- and the Mac is the natural second copy. It costs roughly 1 GB/year. @@ -30,17 +50,57 @@ REM matches by NAME AT ANY DEPTH, so had it been called manifest.json this sync REM have silently delivered raw bytes with no index. REM /XF also drops partial-write temps (*.tmp from parquet/JSON writes, *.part from REM in-flight raw downloads) so a sync mid-capture never lands a truncated file. -robocopy "REPLACE_WITH_STORE_PATH" "REPLACE_WITH_DEST_PATH" /MIR /R:2 /W:5 /NFL /NDL /NP ^ +robocopy "%COT_SRC%" "%COT_DEST%" /MIR /R:2 /W:5 /NFL /NDL /NP ^ /XD _cache _raw citpy ^ /XF manifest.json *.tmp *.part +set "COT_RC=%ERRORLEVEL%" + +REM --------------------------------------------------------------------------- +REM Pass 2 — the bar store. A DIFFERENT exclusion list, and the difference is +REM load-bearing. +REM +REM manifest.json is NOT excluded here. In the COT store it is a dead legacy +REM aggregate; in the bar store it is the ONLY manifest -- marketdata keeps one +REM file at the store root, not a manifests\ directory. Carrying pass 1's /XF +REM over to this pass would strip the bar store's whole index in transit and +REM deliver parquet the replica cannot enumerate. This is the same name-at-any- +REM depth trap docs/SYNCING.md documents for vintage\snapshots.json, and it is +REM why the two stores get two passes rather than one shared exclusion list. +REM +REM _cache and citpy are not excluded because the bar store has neither: no +REM marketdata provider writes a download cache, and citpy is a cotdata-store +REM consumer artefact. Listing them would be harmless but misleading. +REM +REM _raw IS excluded: it holds databento's append-only PAID raw store +REM ($MARKETDATA_DATABENTO_RAW, else _raw\databento under the bar store). A +REM replica has no use for it and re-fetching it costs money. +REM +REM Both stores commit parquet with an atomic replace, so *.tmp / *.part are the +REM same partial-write guard as above. +robocopy "%BAR_SRC%" "%BAR_DEST%" /MIR /R:2 /W:5 /NFL /NDL /NP ^ + /XD _raw ^ + /XF *.tmp *.part +set "BAR_RC=%ERRORLEVEL%" + +REM Pass 2 runs even when pass 1 failed, deliberately. They mirror independent +REM stores, so aborting on the first failure would mean a COT hiccup silently +REM stops bars reaching the replica for as long as it lasts -- one broken thing +REM presenting as two. Each pass reports its own code and the exit is the worst. REM robocopy uses exit codes 0-7 for SUCCESS (1 = files copied, 2 = extras present, REM 3 = both, and so on) and 8+ for failure. Task Scheduler treats any non-zero as a -REM failure, so without this every successful sync would be reported as an error and -REM "restart on failure" would loop. -if %ERRORLEVEL% GEQ 8 ( - echo sync FAILED with robocopy code %ERRORLEVEL% - exit /b %ERRORLEVEL% -) -echo sync ok ^(robocopy code %ERRORLEVEL%^) +REM failure, so without this normalisation every successful sync would be reported +REM as an error and "restart on failure" would loop. +if %COT_RC% GEQ 8 goto :cot_failed +if %BAR_RC% GEQ 8 goto :bar_failed +echo sync ok ^(cot robocopy %COT_RC%, bars robocopy %BAR_RC%^) exit /b 0 + +:cot_failed +echo COT sync FAILED with robocopy code %COT_RC% +if %BAR_RC% GEQ 8 echo bar sync ALSO FAILED with robocopy code %BAR_RC% +exit /b %COT_RC% + +:bar_failed +echo bar sync FAILED with robocopy code %BAR_RC% ^(cot sync ok, code %COT_RC%^) +exit /b %BAR_RC% diff --git a/tests/test_sync_preflight.py b/tests/test_sync_preflight.py new file mode 100644 index 0000000..777303d --- /dev/null +++ b/tests/test_sync_preflight.py @@ -0,0 +1,179 @@ +"""docs/examples/sync_preflight.py — the guard that refuses a destructive mirror. + +It is a docs example rather than shipped code, but it is the one thing standing +between a `robocopy /MIR` and a store, and since ADR-0007 it has to read TWO store +layouts. A preflight that reads a bar store with cotdata's rules does not fail +loudly — it prints a plausible summary and reports zero orphans forever — so the +layout handling is worth pinning down. +""" +import importlib.util +import json +from pathlib import Path + +import pytest + +_SRC = Path(__file__).resolve().parents[1] / "docs" / "examples" / "sync_preflight.py" + + +@pytest.fixture(scope="module") +def pf(): + spec = importlib.util.spec_from_file_location("sync_preflight", _SRC) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def _entry(source="norgate", rows=10): + return {"last_date": "2026-08-20", "n_rows": rows, "source": source, "updated_at": "x"} + + +def make_cot_store(root: Path, entries=("ES_13874A",), source="cftc"): + (root / "manifests").mkdir(parents=True, exist_ok=True) + (root / "cot_legacy").mkdir(exist_ok=True) + (root / "manifests" / "cot.json").write_text(json.dumps( + {"cot_legacy": {name: _entry(source) for name in entries}})) + for name in entries: + (root / "cot_legacy" / f"{name}.parquet").write_bytes(b"") + return root + + +def make_bar_store(root: Path, entries=(("futures", "norgate", "ES_backadj"),), + source=None): + (root / "bars").mkdir(parents=True, exist_ok=True) + man = {"schema_version": 2, "universe_is_point_in_time": False, "bars": {}} + for domain, vendor, name in entries: + d = root / "bars" / domain / vendor + d.mkdir(parents=True, exist_ok=True) + (d / f"{name}.parquet").write_bytes(b"") + man["bars"][f"{domain}/{vendor}/{name}"] = _entry(source or vendor) + (root / "manifest.json").write_text(json.dumps(man)) + return root + + +# ── layout detection ──────────────────────────────────────────────────────── + +def test_detects_each_layout_from_structure(pf, tmp_path): + assert pf.detect_layout(make_cot_store(tmp_path / "cot")) == pf.COTDATA + assert pf.detect_layout(make_bar_store(tmp_path / "bars")) == pf.MARKETDATA + + +def test_detects_marketdata_from_the_manifest_when_bars_dir_is_absent(pf, tmp_path): + """A store whose manifest exists but whose bars/ has not been created yet — + e.g. a replica mid-first-sync, manifest last. Structure is missing, so the + manifest key has to carry the decision.""" + root = tmp_path / "empty-ish" + root.mkdir() + (root / "manifest.json").write_text(json.dumps({"schema_version": 2, "bars": {}})) + assert pf.detect_layout(root) == pf.MARKETDATA + + +def test_unrecognisable_store_reads_as_cotdata(pf, tmp_path): + """The default matters: a pre-split store has neither manifests/ nor bars/, + and it must keep its old reading rather than be judged by the new rules.""" + root = tmp_path / "legacy" + root.mkdir() + (root / "manifest.json").write_text(json.dumps({"cot_legacy": {"ES": _entry()}})) + assert pf.detect_layout(root) == pf.COTDATA + + +# ── the marketdata manifest is live, not legacy ───────────────────────────── + +def test_marketdata_manifest_is_read_without_the_legacy_fallback(pf, tmp_path): + """cotdata's loader treats root manifest.json as a per-domain FALLBACK. Applied + to a bar store it happens to work, so this pins the intent: the bar store's + manifest is the real thing, and the store-level flags beside the domains are + dropped rather than mistaken for a domain.""" + root = make_bar_store(tmp_path / "bars") + m = pf.load_manifest(root, pf.MARKETDATA) + assert set(m) == {"bars"} # not schema_version / the pit flag + assert "futures/norgate/ES_backadj" in m["bars"] + + +def test_cotdata_manifest_merge_still_prefers_halves_over_legacy(pf, tmp_path): + root = make_cot_store(tmp_path / "cot", entries=("ES_13874A",)) + (root / "manifest.json").write_text(json.dumps( + {"cot_legacy": {"STALE": _entry()}, "cot_tff": {"ZN_043602": _entry()}})) + m = pf.load_manifest(root, pf.COTDATA) + assert set(m["cot_legacy"]) == {"ES_13874A"} # half file wins for its domain + assert "cot_tff" in m # legacy fills a domain it lacks + + +# ── on-disk orphans, the check that silently did nothing for bar stores ───── + +def test_orphaned_bar_parquet_is_found_under_the_nested_layout(pf, tmp_path, capsys): + """The regression this rewrite exists for: bars live at + bars///, so a flat glob finds nothing and every bar store + reports clean no matter what a mirror would delete.""" + src = make_bar_store(tmp_path / "src") + dest = make_bar_store(tmp_path / "dest") + d = dest / "bars" / "futures" / "norgate" + (d / "CL_backadj.parquet").write_bytes(b"") # on disk, not in either manifest + + assert pf.main([str(src), str(dest)]) == 1 + out = capsys.readouterr().out + assert "bars/: 1 parquet files exist only on DEST" in out + assert "futures/norgate/CL_backadj.parquet" in out + + +def test_same_name_under_different_vendors_is_not_an_orphan(pf, tmp_path): + """The vendor is part of the key, so norgate/ES_backadj and databento/ES_backadj + are two files — which is exactly why the 2026-07-26 collision cannot recur in + this layout. If the key were the bare filename they would cancel out and a real + orphan would be hidden.""" + src = make_bar_store(tmp_path / "src", + entries=(("futures", "norgate", "ES_backadj"),)) + dest = make_bar_store(tmp_path / "dest", + entries=(("futures", "databento", "ES_backadj"),)) + files = pf.data_files(dest, pf.MARKETDATA) + assert files["bars"] == {"futures/databento/ES_backadj.parquet"} + assert pf.main([str(src), str(dest)]) == 1 # genuinely dest-only, refused + + +def test_matching_bar_stores_are_safe(pf, tmp_path, capsys): + src = make_bar_store(tmp_path / "src") + dest = make_bar_store(tmp_path / "dest") + assert pf.main([str(src), str(dest)]) == 0 + assert "SAFE" in capsys.readouterr().out + + +# ── cross-layout comparison is refused, not guessed ───────────────────────── + +def test_comparing_a_cot_store_with_a_bar_store_refuses(pf, tmp_path, capsys): + """Two swapped paths is far likelier than an intention, and the mirror it would + green-light deletes the whole destination.""" + cot, bars = make_cot_store(tmp_path / "cot"), make_bar_store(tmp_path / "bars") + assert pf.main([str(cot), str(bars)]) == 2 # not 0 (safe) and not 1 (refused) + out = capsys.readouterr().out + assert "CANNOT JUDGE" in out + assert "cotdata" in out and "marketdata" in out + + +# ── the original checks still hold ────────────────────────────────────────── + +def test_dest_only_manifest_entries_are_refused(pf, tmp_path, capsys): + src = make_cot_store(tmp_path / "src", entries=("ES_13874A",)) + dest = make_cot_store(tmp_path / "dest", entries=("ES_13874A", "CL_067651")) + assert pf.main([str(src), str(dest)]) == 1 + assert "exist only on DEST" in capsys.readouterr().out + + +def test_same_key_different_source_is_flagged(pf, tmp_path, capsys): + """The collision with no visible symptom. In a bar store only the single-table + domains can still do this — the vendor is in the path for bars.""" + src = tmp_path / "src" + dest = tmp_path / "dest" + for root, vendor in ((src, "norgate"), (dest, "databento")): + make_bar_store(root) + m = json.loads((root / "manifest.json").read_text()) + m["metadata"] = {"contract_specs": _entry(vendor)} + (root / "manifest.json").write_text(json.dumps(m)) + + assert pf.main([str(src), str(dest)]) == 1 + out = capsys.readouterr().out + assert "DIFFERENT sources on each side" in out + assert "metadata/contract_specs: src=norgate dest=databento" in out + + +def test_bad_arguments_and_missing_directories_exit_2(pf, tmp_path): + assert pf.main([]) == 2 + assert pf.main([str(tmp_path), str(tmp_path / "nope")]) == 2 diff --git a/tests/test_verify_replicas.py b/tests/test_verify_replicas.py new file mode 100644 index 0000000..a46231d --- /dev/null +++ b/tests/test_verify_replicas.py @@ -0,0 +1,156 @@ +"""docs/examples/mac/verify-replicas.sh — the alarm that says a sync stopped. + +Runs the REAL script, with `ssh` and the two `--check` binaries stubbed on PATH. +Worth testing rather than eyeballing for the same reason the script exists: its +whole job is to be the thing that notices, and a verifier that silently passes is +indistinguishable from a working sync until someone reads a stale chart. + +ADR-0007 split the data across two stores, so the case that matters here is a +CURRENT COT store beside a STALE (or absent) bar store — green under the old +two-check version, red under this one. +""" +import os +import subprocess +import time +from pathlib import Path + +import pytest + +_SCRIPT = Path(__file__).resolve().parents[1] / "docs" / "examples" / "mac" / "verify-replicas.sh" +DAY = 86400 + + +@pytest.fixture() +def rig(tmp_path): + """A local pair of stores plus a stubbed `ssh` standing in for the remote pair. + + The stub answers the two things the script asks a remote: an mtime probe + (`date -r FILE +%s`) and a `--check` run. Remote mtimes come from files in + `remote/`, so a test moves them the same way it moves the local ones. + """ + local_cot = tmp_path / "local_cot" + local_bars = tmp_path / "local_bars" + remote = tmp_path / "remote" + for p in (local_cot, local_bars, remote / "cot", remote / "bars"): + p.mkdir(parents=True) + (local_cot / "status.json").write_text("{}") + (local_bars / "manifest.json").write_text("{}") + (remote / "cot" / "status.json").write_text("{}") + (remote / "bars" / "manifest.json").write_text("{}") + + bin_dir = tmp_path / "bin" + bin_dir.mkdir() + # The stub resolves a remote path to its local stand-in under remote/, so the + # script's real `date -r` runs against a real file and the platform handling + # is exercised rather than mocked away. + (bin_dir / "ssh").write_text( + "#!/usr/bin/env bash\n" + 'cmd="${@: -1}"\n' + 'case "$cmd" in\n' + ' *"date -r"*) f=$(printf %s "$cmd" | sed "s/.*date -r .\\(.*\\). +%s.*/\\1/")\n' + ' exec date -r "$f" +%s ;;\n' + ' *) echo "stub --check ok" ;;\n' + 'esac\n') + for name in ("cotdata-update", "marketdata-update"): + (bin_dir / name).write_text("#!/usr/bin/env bash\necho 'stub --check ok'\n") + for f in bin_dir.iterdir(): + f.chmod(0o755) + + env = dict(os.environ) + env.update({ + "PATH": f"{bin_dir}{os.pathsep}{env['PATH']}", + "LOCAL_COT": str(local_cot), "LOCAL_BARS": str(local_bars), + "LOCAL_COT_CHECK": str(bin_dir / "cotdata-update"), + "LOCAL_BAR_CHECK": str(bin_dir / "marketdata-update"), + "REMOTE": "stub@example.invalid", + "REMOTE_COT": str(remote / "cot"), "REMOTE_BARS": str(remote / "bars"), + "REMOTE_COT_CHECK": "cotdata-update", "REMOTE_BAR_CHECK": "marketdata-update", + }) + return {"env": env, "local_cot": local_cot, "local_bars": local_bars, + "remote": remote} + + +def run(rig): + return subprocess.run(["bash", str(_SCRIPT)], env=rig["env"], + capture_output=True, text=True) + + +def age(path: Path, days: float): + when = time.time() - days * DAY + os.utime(path, (when, when)) + + +def test_all_current_passes(rig): + r = run(rig) + assert r.returncode == 0, r.stdout + assert "RESULT: PASS" in r.stdout + + +def test_stale_bar_store_fails_while_cot_is_current(rig): + """The exact failure ADR-0007 introduced and the old two-check version could + not see: COT arriving on schedule while no bar has landed in a fortnight.""" + age(rig["local_bars"] / "manifest.json", 14) + age(rig["remote"] / "bars" / "manifest.json", 14) + r = run(rig) + assert r.returncode == 1 + assert "local bar manifest.json written" in r.stdout + assert "past the 4d window" in r.stdout + assert "PASS: local COT" in r.stdout # the COT half really is fine + assert "RESULT: FAIL" in r.stdout + + +def test_a_weekend_old_bar_store_still_passes(rig): + """Not a bug to be fixed by tightening the window. marketdata writes its + manifest only when a bar is actually written, so a Friday-to-Tuesday gap is + the normal state of a correctly working sync — a `today` test here would cry + wolf every weekend and get ignored by the second month.""" + age(rig["local_bars"] / "manifest.json", 3) + age(rig["remote"] / "bars" / "manifest.json", 3) + r = run(rig) + assert r.returncode == 0, r.stdout + assert "3d old, window 4d" in r.stdout + + +def test_window_is_configurable(rig): + age(rig["local_bars"] / "manifest.json", 6) + age(rig["remote"] / "bars" / "manifest.json", 6) + assert run(rig).returncode == 1 + rig["env"]["BAR_MAX_AGE_DAYS"] = "10" + assert run(rig).returncode == 0 + + +def test_missing_bar_manifest_names_the_real_problem(rig): + """A replica that never received a bar store at all. Distinct from stale, and + the likelier state right after the migration.""" + (rig["local_bars"] / "manifest.json").unlink() + r = run(rig) + assert r.returncode == 1 + assert "no bars have reached this replica" in r.stdout + + +def test_stale_cot_store_still_fails(rig): + """The original check, unchanged: cotdata rewrites status.json every run, so + anything but today means the producer or its sync did not run.""" + age(rig["local_cot"] / "status.json", 2) + r = run(rig) + assert r.returncode == 1 + assert "FAIL: local COT status.json last written" in r.stdout + + +def test_divergent_replicas_warn_even_when_both_are_inside_the_window(rig): + """Both syncs preserve timestamps, so a matching producer run lands the SAME + mtime on both replicas. Different mtimes mean one push is behind — and inside + the staleness window neither replica looks wrong on its own.""" + age(rig["remote"] / "bars" / "manifest.json", 2) + r = run(rig) + assert r.returncode == 0 # both still inside the window + assert "WARN: the two bar replicas hold DIFFERENT manifest mtimes" in r.stdout + + +def test_matching_replicas_do_not_warn(rig): + same = time.time() - DAY + for p in (rig["local_bars"] / "manifest.json", rig["remote"] / "bars" / "manifest.json"): + os.utime(p, (same, same)) + r = run(rig) + assert r.returncode == 0 + assert "WARN" not in r.stdout