Label the retired prices/metadata domains, and make the scheduling doc match the box - #112
Merged
mspinola merged 4 commits intoAug 21, 2026
Merged
Conversation
…c match the box
A consumer read `newest_data.prices` from a store whose bars moved to marketdata
at ADR-0007, got 2026-08-07 with `lagging: 0`, and had no signal it was reading an
abandoned tree. The bar store was current the whole time; only the directory the
reader was pointed at was dead.
The silence was structural. Lag is measured WITHIN a domain -- each entry against
its own domain's newest write -- so a uniformly frozen tree is perfectly
self-consistent, scores zero, and sits in the report beside genuinely-current COT
domains looking exactly as healthy. `store.py` already documented both domains as
having no writer left; only the status report hadn't been told.
status.py:
- `_RETIRED_DOMAINS = {"prices", "metadata"}`. They stay VISIBLE rather than
being dropped from `_DOMAINS`: a pre-move store still holds the files, and
hiding them would make `--check` disagree with the operator's own disk.
- `--check` tags those rows `<- RETIRED` and names the store the data moved to.
- The all-clear now reads "every entry in the live domains", because the
unqualified version was vouching for the dead tree too.
- `retired` is carried into `status.json` so machine pollers can see it.
- The `build_status_doc` contract docstring used `prices` as its worked example
of what to poll. It now uses a live domain and says to check `retired` first.
- Widen the domain column: `cot_supplemental` is 16 chars and was overflowing a
12-wide field, shunting the rest of that row right.
Docs:
- README's polling contract also gave `newest_data.prices` as its example. Same
fix, plus the sample `--check` and `status.json` output refreshed to show a
retired domain rather than a fictional healthy one.
- SYNCING.md gains "Retiring the moved domains", including why adding `prices`
to the pass-1 `/XD` list is the wrong fix: excluded directories are neither
copied nor deleted, so it freezes the stale copy on every replica forever
instead of clearing it. Delete on the producer and let `/MIR` propagate.
- WINDOWS_SCHEDULING.md called the bars task "marketdata bars" in six pasteable
commands. The task is named `cotdata prices` on the box -- created before
ADR-0007, never renamed because that loses the run history. Every command
targeted a task that does not exist.
- Document the equities task, which had no coverage anywhere: task count was
"three plus an optional fourth" against five real tasks, and the wrapper it
runs existed only as a gitignored local file. Adds
docs/examples/windows/run-equities.cmd.
Tests: the fixture's lagging exemplar lived in `prices`, which is now deliberately
silent, so it moves to a live domain -- the mechanic those tests are named for
still gets exercised. Five new tests cover the retired flag, the suppression, the
report text, the status.json contract, and that live domains still report lag.
Pre-existing failures in test_sync_preflight.py and test_vintage_ingest.py are
unchanged by this branch (verified against main).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
crowdmon's scheduling page tells you to run `schtasks /Change /TN "cotdata prices" /TR "...\run-nightly.cmd"` so a panel publish can be chained behind the bars. crowdmon was deprecated 2026-08-07 and that chain was never installed: there is no run-nightly.cmd and no run-publish.cmd for it to call. Running it points this box's futures producer at a file that does not exist, and it fails quietly -- the store just stops advancing. The warning belongs here rather than only there, for two reasons. It is where someone administering the live task is actually reading, and mspinola/crowdmon is archived and read-only on GitHub, so its own page cannot currently be amended. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…omain "Task shows success but wrote nothing" told you to confirm a bar run by watching `newest_data.prices` advance in cotdata's status.json. That is the retired domain -- frozen at the ADR-0007 cutover, and in the wrong store besides -- so it can never confirm anything about a bar run. Worse than useless, given where it sits: this is the section someone reads when a bar run already looks wrong, and the check it prescribes shows a permanent stall regardless of what actually happened. Points at `marketdata-update --check` instead. Found because a consumer on the research Mac hit the same trap from the README's polling contract on 2026-08-21. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The repeating-trigger section covered the trigger and stopped there. A repetition multiplies everything the wrapper does, and on this deployment the wrapper ends by mirroring both stores -- including the vintage tree, which SYNCING.md puts at roughly 1 GB/year. The Friday COT poller fires 23 times in 45 minutes, so a quiet Friday scanned a gigabyte-plus over SMB and again over SSH, 23 times, for nothing. The bars task gets this for free: --require-final defers non-zero, the wrapper carries the code out, and the chained syncs are never reached. COT has no such signal, so it needs an explicit guard -- and the obvious guard is the broken one. Measured on the reference box 2026-08-21 across a real no-op repeat: manifests/cot.json's MD5 CHANGED while status.json's newest_data stayed byte-identical. Every manifest entry is rewritten each pass so it carries a current updated_at, which is what the lag check depends on. A guard keyed on that file therefore never fires -- worse than no guard, since it looks like a fix. `%~tF` is independently wrong: one-minute resolution, so a capture landing in the same minute as the "before" reading compares equal and skips the sync that should have run. Also records the trap on the other side, which costs bars rather than time: do NOT guard the daily catch-up. Both sync scripts mirror BOTH stores since ADR-0007, so the 08:10 COT run is what carries bar data to the replicas after a night when the bars task captured but its own sync failed. COT is weekly, so newest_data does not move on ~4 days in 5 -- guarding the catch-up would skip the sync on exactly the days that safety net exists for. Give the guard to the poller only, via an argument the catch-up does not pass, and set it with PowerShell rather than schtasks /TR, which would fold the argument into the executable path. And: make the guard fail toward syncing. A redundant mirror is slow; a skipped one is invisible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mspinola
force-pushed
the
claude/retire-prices-domain-and-fix-scheduling-docs
branch
from
August 21, 2026 21:03
13bebab to
9f12180
Compare
mspinola
deleted the
claude/retire-prices-domain-and-fix-scheduling-docs
branch
August 21, 2026 21:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A consumer read
newest_data.pricesfrom a store whose bars moved to marketdata at ADR-0007, got2026-08-07withlagging: 0, and had no signal it was reading an abandoned tree. The bar store was current the whole time and its replica was current too — the only thing wrong was which directory the reader was pointed at.The silence was structural, not a one-off. Lag is measured within a domain (each entry against its own domain's newest write), so a uniformly frozen tree is perfectly self-consistent, scores
lagging: 0, and sits in the report beside genuinely-current COT domains looking exactly as healthy.store.pyalready documentedpricesandmetadataas having no writer left; only the status report had never been told.What
--checksays nowReal output against the production store, not a mock-up.
Design notes
_DOMAINS. A pre-move store still holds the files; hiding them would make--checkdisagree with the operator's own disk. The goal is to stop them reading as healthy, not to stop them being seen.retiredis carried intostatus.json, and thebuild_status_doccontract docstring no longer usespricesas its worked example of what to poll. Neither does the README's polling contract, which is where the advice that caused this actually came from.SYNCING.mdexplains why/XD pricesis the wrong fix. Excluded directories are neither copied nor deleted, so an exclusion freezes the stale copy on every replica permanently instead of clearing it. Delete on the producer and let/MIRpropagate.Scheduling doc corrections
Independent of the above, the doc had drifted from the box it describes:
"marketdata bars"in six pasteable commands. The task is namedcotdata prices— created before ADR-0007, never renamed because renaming loses the run history. Every one of those commands targeted a task that does not exist.docs/examples/windows/run-equities.cmdand theschtasksline, with the reasoning for why it is a separate task rather than a step inrun-prices.cmd.Tests
The fixture's lagging exemplar lived in
prices, which is now deliberately silent, so it moves to a live domain — the mechanic those tests are named for still gets exercised. Five new tests cover the retired flag, the suppression, the report text, thestatus.jsoncontract, and a guard that live domains still report lag.222 passed. The 3 failures intest_sync_preflight.py/test_vintage_ingest.pyare pre-existing and unchanged — verified againstmain.Also
Widened the domain column:
cot_supplementalis 16 chars and was overflowing a 12-wide field, shunting the rest of that row right. Visible in the output above.🤖 Generated with Claude Code