Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,7 @@ dist/

# roll-rule investigation cache (scripts/investigate_databento_roll_rule.py)
.rollrule_cache/

# local scheduler backups (same machine-specific paths as /scheduler/)
/scheduler.bak*/
/scheduler_backup*/
32 changes: 24 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,13 +240,27 @@ cotdata-update --reconcile # prune stale manifest entries (see below)
`--check` reports per-domain row counts, newest data date, last write, and any entries lagging behind their peers (a partial-run signal):

```
domain entries rows newest data last write (UTC) behind
prices 84 829,096 2026-07-14 2026-07-15T10:15:24Z 1d
cot_legacy 44 70,201 2026-07-07 2026-07-14T04:26:55Z 8d
domain entries rows newest data last write (UTC) behind
prices 98 954,524 2026-08-07 2026-08-08T15:40:20Z 14d ← RETIRED
cot_legacy 53 81,821 2026-08-11 2026-08-21T12:10:42Z 10d
cot_disagg 29 29,187 2026-08-11 2026-08-21T12:10:51Z 10d
...
✓ all entries current (none lag behind their domain's newest).

⚠ prices, metadata: RETIRED — this store still holds the files, but cotdata stopped writing
them at ADR-0007. Their dates are frozen and will never advance again.
Daily bars and contract specs now live in the marketdata store ($MARKETDATA_STORE);
read them with `marketdata-update --check`.

✓ every entry in the live domains was written by the latest producer pass.
```

**A `RETIRED` domain is not a warning you can wave away.** ADR-0007 moved bars and contract
specs to marketdata; a store created before that move still holds `prices/` and `metadata/`,
and nothing advances them. They earn a label rather than silence because lag is measured
*within* a domain — a uniformly frozen tree is perfectly self-consistent, so it scores
`lagging: 0` and reads exactly like a domain that ran cleanly minutes ago. See
[SYNCING.md](docs/SYNCING.md#retiring-the-moved-domains-prices-and-metadata) for removing them.

### `status.json` — new-data signal for downstream tools

Every producer run writes `$COTDATA_STORE/status.json` (atomically, beside the data), so tools that trigger on fresh data poll one small structured file instead of scanning the store:
Expand All @@ -255,14 +269,16 @@ Every producer run writes `$COTDATA_STORE/status.json` (atomically, beside the d
{
"generated_at": "2026-07-15T10:15:24Z",
"schema_version": 2,
"newest_data": { "prices": "2026-07-14", "cot_legacy": "2026-07-07", "cot_disagg": "2026-07-07", "cot_tff": "2026-07-07", "cot_supplemental": "2026-07-07" },
"domains": { "prices": { "newest_data": "2026-07-14", "last_write": "2026-07-15T10:15:24Z", "entries": 84, "rows": 829096, "lagging": 0 }, "...": {} },
"last_run": { "kinds": ["prices"], "ok": ["ES", "..."], "symbols_failed": [], "rows": 1658000, "seconds": 88, "at": "2026-07-15T10:15:24Z" }
"newest_data": { "cot_legacy": "2026-08-11", "cot_disagg": "2026-08-11", "cot_tff": "2026-08-11", "cot_supplemental": "2026-08-11", "prices": "2026-08-07" },
"domains": { "cot_legacy": { "newest_data": "2026-08-11", "last_write": "2026-08-21T12:10:42Z", "entries": 53, "rows": 81821, "lagging": 0, "retired": false },
"prices": { "newest_data": "2026-08-07", "last_write": "2026-08-08T15:40:20Z", "entries": 98, "rows": 954524, "lagging": 0, "retired": true }, "...": {} },
"last_run": { "kinds": ["cot_legacy", "cot_disagg"], "ok": ["001602", "..."], "symbols_failed": [], "rows": 111008, "seconds": 22, "at": "2026-08-21T12:10:58Z" }
}
```

**Polling contract:**
- To detect **new data**, compare `newest_data.<domain>` (e.g. `newest_data.prices`, `newest_data.cot_legacy`) against your last-seen value. It advances **only when genuinely new daily data arrives** — a no-op run leaves it unchanged.
- To detect **new data**, compare `newest_data.<domain>` (e.g. `newest_data.cot_legacy`) against your last-seen value. It advances **only when genuinely new daily data arrives** — a no-op run leaves it unchanged.
- **Check `domains.<domain>.retired` first.** A retired domain's date is frozen, so a poller keyed on it waits forever and never errors — it just quietly reports data that stopped moving. This bullet used to give `newest_data.prices` as its example, and on 2026-08-21 a consumer following it read a two-week-old date from a store whose bars had moved to marketdata. **Bars are not in this store; poll `marketdata-update --check` instead.**
- To detect that **a run happened at all** (new data or not), use `generated_at`.
- `last_run` carries the most recent run's outcome (which domains, per-symbol failures) for alerting.

Expand Down
37 changes: 37 additions & 0 deletions docs/SYNCING.md
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,43 @@ disk usage and a directory listing both look right, and only a read notices. Bot
transports match exclusions by **name at any depth**, so there is no `/manifests/` prefix
to make the rule safe. Two stores, two passes, two lists.

### Retiring the moved domains: `prices/` and `metadata/`

ADR-0007 moved daily bars and contract specs to marketdata. On a store created before that
move the old directories are still on disk, still full, and **still mirrored** — the pass-1
exclusion list covers `_cache`, `_raw` and `citpy`, not these. So every sync faithfully
carries a frozen tree to every replica, indefinitely.

**Why that is worse than it sounds.** `cotdata-update --check` measures lag *within* a
domain: each entry against its own domain's newest write. A tree where nothing has been
written since the cutover is perfectly self-consistent, so it scores `lagging: 0` and sits
in the report beside genuinely-current COT domains looking exactly as healthy. `status.json`
carried the same claim to machine consumers through `newest_data.prices`.

On 2026-08-21 that is precisely what happened: a consumer read `newest_data.prices`, got
`2026-08-07` from a store whose bars had moved two weeks earlier, and had no signal it was
reading an abandoned tree. The bar store was current the whole time and its replica was
current too — nothing was broken except which directory the reader was pointed at.

`--check` now labels both domains `RETIRED` and names the store the data moved to, so the
report no longer implies health it cannot verify. The files themselves still need removing:

1. **Repoint the consumers first.** Bars come from `$MARKETDATA_STORE` via
`marketdata-update --check`. Deleting before repointing turns a silently-wrong reader
into a hard-failing one, which is better but still an outage.
2. **Delete on the producer, then let the mirror propagate.** `robocopy /MIR` and
`rsync --delete` both remove what the source no longer has, so one producer-side
deletion clears every replica on the next sync. Deleting on a replica instead
accomplishes nothing: the next sync restores it from the producer.
3. **Do not add `prices` to the pass-1 `/XD` list as a substitute.** Excluded directories
are neither copied *nor deleted*, so an exclusion freezes the stale copy on every
replica permanently instead of clearing it — the opposite of what you want. Exclude it
only *after* the deletion has propagated, and only if you want belt and braces.

`metadata/` is small enough that leaving it costs nothing; `prices/` is typically ~46 MB per
replica. Neither is recoverable from cotdata once deleted, but both are fully reproducible
from marketdata, which is where they now live.

### `vintage/` is irreplaceable, so where it is WRITTEN matters

The vintage tree (`vintage/raw/`, `observations/`, `revisions/`, `snapshots.json`) records
Expand Down
Loading
Loading