Skip to content

Label the retired prices/metadata domains, and make the scheduling doc match the box - #112

Merged
mspinola merged 4 commits into
mainfrom
claude/retire-prices-domain-and-fix-scheduling-docs
Aug 21, 2026
Merged

Label the retired prices/metadata domains, and make the scheduling doc match the box#112
mspinola merged 4 commits into
mainfrom
claude/retire-prices-domain-and-fix-scheduling-docs

Conversation

@mspinola

Copy link
Copy Markdown
Owner

Why

A consumer read newest_data.prices from a store whose bars moved to marketdata at ADR-0007, got 2026-08-07 with lagging: 0, and had no signal it was reading an abandoned tree. The bar store was current the whole time and its replica was current too — the only thing wrong was which directory the reader was pointed at.

The silence was structural, not a one-off. Lag is measured within a domain (each entry against its own domain's newest write), so a uniformly frozen tree is perfectly self-consistent, scores lagging: 0, and sits in the report beside genuinely-current COT domains looking exactly as healthy. store.py already documented prices and metadata as having no writer left; only the status report had never been told.

What --check says now

domain             entries         rows   newest data      last write (UTC)  behind
prices                  98      954,524    2026-08-07  2026-08-08T15:40:20Z     14d  ← RETIRED
metadata                 1           49          None  2026-08-09T13:47:50Z       —  ← RETIRED
cot_legacy              53       81,821    2026-08-11  2026-08-21T12:10:42Z     10d
...

⚠ prices, metadata: RETIRED — this store still holds the files, but cotdata stopped writing
                    them at ADR-0007. Their dates are frozen and will never advance again.
  Daily bars and contract specs now live in the marketdata store ($MARKETDATA_STORE);
  read them with `marketdata-update --check`.

✓ every entry in the live domains was written by the latest producer pass.

Real output against the production store, not a mock-up.

Design notes

  • Retired domains stay visible rather than being dropped from _DOMAINS. A pre-move store still holds the files; hiding them would make --check disagree with the operator's own disk. The goal is to stop them reading as healthy, not to stop them being seen.
  • The all-clear is now scoped — "every entry in the live domains". Unqualified, it was vouching for the dead tree too.
  • retired is carried into status.json, and the build_status_doc contract docstring no longer uses prices as its worked example of what to poll. Neither does the README's polling contract, which is where the advice that caused this actually came from.
  • SYNCING.md explains why /XD prices is the wrong fix. Excluded directories are neither copied nor deleted, so an exclusion freezes the stale copy on every replica permanently instead of clearing it. Delete on the producer and let /MIR propagate.

Scheduling doc corrections

Independent of the above, the doc had drifted from the box it describes:

  • It called the bars task "marketdata bars" in six pasteable commands. The task is named cotdata prices — created before ADR-0007, never renamed because renaming loses the run history. Every one of those commands targeted a task that does not exist.
  • The equities producer had no documentation anywhere: the task count read "three tasks (plus an optional fourth)" against five real tasks, and the wrapper it runs existed only as a gitignored local file. Adds docs/examples/windows/run-equities.cmd and the schtasks line, with the reasoning for why it is a separate task rather than a step in run-prices.cmd.

Tests

The fixture's lagging exemplar lived in prices, which is now deliberately silent, so it moves to a live domain — the mechanic those tests are named for still gets exercised. Five new tests cover the retired flag, the suppression, the report text, the status.json contract, and a guard that live domains still report lag.

222 passed. The 3 failures in test_sync_preflight.py / test_vintage_ingest.py are pre-existing and unchanged — verified against main.

Also

Widened the domain column: cot_supplemental is 16 chars and was overflowing a 12-wide field, shunting the rest of that row right. Visible in the output above.

🤖 Generated with Claude Code

mspinola and others added 4 commits August 21, 2026 08:44
…c match the box

A consumer read `newest_data.prices` from a store whose bars moved to marketdata
at ADR-0007, got 2026-08-07 with `lagging: 0`, and had no signal it was reading an
abandoned tree. The bar store was current the whole time; only the directory the
reader was pointed at was dead.

The silence was structural. Lag is measured WITHIN a domain -- each entry against
its own domain's newest write -- so a uniformly frozen tree is perfectly
self-consistent, scores zero, and sits in the report beside genuinely-current COT
domains looking exactly as healthy. `store.py` already documented both domains as
having no writer left; only the status report hadn't been told.

status.py:
  - `_RETIRED_DOMAINS = {"prices", "metadata"}`. They stay VISIBLE rather than
    being dropped from `_DOMAINS`: a pre-move store still holds the files, and
    hiding them would make `--check` disagree with the operator's own disk.
  - `--check` tags those rows `<- RETIRED` and names the store the data moved to.
  - The all-clear now reads "every entry in the live domains", because the
    unqualified version was vouching for the dead tree too.
  - `retired` is carried into `status.json` so machine pollers can see it.
  - The `build_status_doc` contract docstring used `prices` as its worked example
    of what to poll. It now uses a live domain and says to check `retired` first.
  - Widen the domain column: `cot_supplemental` is 16 chars and was overflowing a
    12-wide field, shunting the rest of that row right.

Docs:
  - README's polling contract also gave `newest_data.prices` as its example. Same
    fix, plus the sample `--check` and `status.json` output refreshed to show a
    retired domain rather than a fictional healthy one.
  - SYNCING.md gains "Retiring the moved domains", including why adding `prices`
    to the pass-1 `/XD` list is the wrong fix: excluded directories are neither
    copied nor deleted, so it freezes the stale copy on every replica forever
    instead of clearing it. Delete on the producer and let `/MIR` propagate.
  - WINDOWS_SCHEDULING.md called the bars task "marketdata bars" in six pasteable
    commands. The task is named `cotdata prices` on the box -- created before
    ADR-0007, never renamed because that loses the run history. Every command
    targeted a task that does not exist.
  - Document the equities task, which had no coverage anywhere: task count was
    "three plus an optional fourth" against five real tasks, and the wrapper it
    runs existed only as a gitignored local file. Adds
    docs/examples/windows/run-equities.cmd.

Tests: the fixture's lagging exemplar lived in `prices`, which is now deliberately
silent, so it moves to a live domain -- the mechanic those tests are named for
still gets exercised. Five new tests cover the retired flag, the suppression, the
report text, the status.json contract, and that live domains still report lag.

Pre-existing failures in test_sync_preflight.py and test_vintage_ingest.py are
unchanged by this branch (verified against main).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
crowdmon's scheduling page tells you to run
`schtasks /Change /TN "cotdata prices" /TR "...\run-nightly.cmd"` so a panel
publish can be chained behind the bars. crowdmon was deprecated 2026-08-07 and
that chain was never installed: there is no run-nightly.cmd and no run-publish.cmd
for it to call. Running it points this box's futures producer at a file that does
not exist, and it fails quietly -- the store just stops advancing.

The warning belongs here rather than only there, for two reasons. It is where
someone administering the live task is actually reading, and mspinola/crowdmon is
archived and read-only on GitHub, so its own page cannot currently be amended.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…omain

"Task shows success but wrote nothing" told you to confirm a bar run by watching
`newest_data.prices` advance in cotdata's status.json. That is the retired domain
-- frozen at the ADR-0007 cutover, and in the wrong store besides -- so it can
never confirm anything about a bar run.

Worse than useless, given where it sits: this is the section someone reads when a
bar run already looks wrong, and the check it prescribes shows a permanent stall
regardless of what actually happened. Points at `marketdata-update --check`
instead.

Found because a consumer on the research Mac hit the same trap from the README's
polling contract on 2026-08-21.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The repeating-trigger section covered the trigger and stopped there. A repetition
multiplies everything the wrapper does, and on this deployment the wrapper ends by
mirroring both stores -- including the vintage tree, which SYNCING.md puts at
roughly 1 GB/year. The Friday COT poller fires 23 times in 45 minutes, so a quiet
Friday scanned a gigabyte-plus over SMB and again over SSH, 23 times, for nothing.

The bars task gets this for free: --require-final defers non-zero, the wrapper
carries the code out, and the chained syncs are never reached. COT has no such
signal, so it needs an explicit guard -- and the obvious guard is the broken one.

Measured on the reference box 2026-08-21 across a real no-op repeat:
manifests/cot.json's MD5 CHANGED while status.json's newest_data stayed
byte-identical. Every manifest entry is rewritten each pass so it carries a current
updated_at, which is what the lag check depends on. A guard keyed on that file
therefore never fires -- worse than no guard, since it looks like a fix. `%~tF` is
independently wrong: one-minute resolution, so a capture landing in the same minute
as the "before" reading compares equal and skips the sync that should have run.

Also records the trap on the other side, which costs bars rather than time: do NOT
guard the daily catch-up. Both sync scripts mirror BOTH stores since ADR-0007, so
the 08:10 COT run is what carries bar data to the replicas after a night when the
bars task captured but its own sync failed. COT is weekly, so newest_data does not
move on ~4 days in 5 -- guarding the catch-up would skip the sync on exactly the
days that safety net exists for. Give the guard to the poller only, via an argument
the catch-up does not pass, and set it with PowerShell rather than schtasks /TR,
which would fold the argument into the executable path.

And: make the guard fail toward syncing. A redundant mirror is slow; a skipped one
is invisible.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mspinola
mspinola force-pushed the claude/retire-prices-domain-and-fix-scheduling-docs branch from 13bebab to 9f12180 Compare August 21, 2026 21:03
@mspinola
mspinola merged commit 218c694 into main Aug 21, 2026
5 checks passed
@mspinola
mspinola deleted the claude/retire-prices-domain-and-fix-scheduling-docs branch August 21, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant