Skip to content

Read bars from marketdata, not cotdata (ADR-0007 step 4) - #11

Merged
mspinola merged 2 commits into
mainfrom
claude/adr7-step4-repoint-prices
Aug 9, 2026
Merged

Read bars from marketdata, not cotdata (ADR-0007 step 4)#11
mspinola merged 2 commits into
mainfrom
claude/adr7-step4-repoint-prices

Conversation

@mspinola

@mspinola mspinola commented Aug 9, 2026

Copy link
Copy Markdown
Owner

ADR-0007 makes cotdata CFTC positioning only and moves every bar to marketdata. The provider shipped and was verified against cotdata's own output on the Windows box (mspinola/marketdata#7, mspinola/marketdata#8). This is the consumer side.

COT positioning still comes from cotdata. Only bars moved.

Companion: mspinola/cot-analyzer#26 — these two land together, and this is the one that changes reads.

The three call sites

All adjustment="backadj", which is what ADR-0007 measured, so the reads themselves are a rename:

file before after
signals.py:1028 cotdata.get_prices(symbol, adjustment="backadj") marketdata.get_bars(symbol, "backadj")
options_data.py:366 same same
CotIndexer.py:719 …, start=start_date …, start=start_date

Tier names and the returned frame are unchanged.

The part that is not a rename, and would have failed silently

CotIndexer busts its parquet caches when the upstream store schema moves, because the per-symbol guards key on column presence and cannot see a value-only change. The example that guard was written for is named in its own comment: reconstructed volume being promoted — a price schema bump.

Prices are no longer in cotdata. So after this repoint, watching cotdata.schema_version() alone leaves uncovered the exact case the guard exists for: stale cached metrics computed against a superseded bar schema, with no error anywhere.

The marker now records both store versions, as separate keys rather than one combined number — they are independent counters, and a max would hide a bump in whichever store happens to sit lower. A marker written before the split has no marketdata version, reads as 0, and busts once. That is correct rather than merely tolerated: the price source moved underneath those caches.

Two tests cover it, including the pre-split marker case.

Verified rather than assumed

marketdata's futures path returns the stored columns without cotdata's keep-list filter, so the frame is wider — it carries FirstVolume, SecondVolume, FirstContract, SecondContract, Volume_Reconstructed, Volume_Source. All three consumers select columns explicitly (price_data[['Open','High','Low','Close']], df['Close']), so the extra columns pass through untouched. Checked before repointing rather than after a failure.

Testing

238 pass, ruff clean, CI green on 3.10–3.13. Run locally against real cotdata and marketdata installs rather than mocks, so the import swap and the new schema call are actually exercised. The signals test doubles were repointed too (signals.marketdata.get_bars), since a stale monkeypatch target would have silently stopped patching and let the tests hit the real store.

CI needed a marketdata sibling checkout: it is not on PyPI, so the declared dependency resolves only from the workspace, and without the checkout pip install -e . failed before a single test ran. A dummy MARKETDATA_STORE joins the dummy COTDATA_STORE for the same reason the first exists — neither package defaults a missing root, both raise by name, and there are two roots now.

Deployment note

The Linux server needs MARKETDATA_STORE set and a second synced store — wired up in mspinola/cot-analyzer#26. Both stores are synced, not produced there.

Three price call sites move from cotdata.get_prices to marketdata.get_bars, all
of them adjustment="backadj", which is what ADR-0007 measured. Tier names and
the returned frame are unchanged, so the reads themselves are a rename. COT
positioning still comes from cotdata; only bars moved.

The part that is not a rename is the cache marker, and it would have failed
silently. CotIndexer busts its parquet caches when the upstream store schema
moves, because the per-symbol guards key on column presence and cannot see a
value-only change. The example that guard was written for is named in its own
comment: reconstructed volume being promoted, which was a PRICE schema bump. So
after this repoint, watching cotdata alone leaves the exact case the guard
exists for uncovered — stale metrics computed against a superseded bar schema,
with no error anywhere.

The marker now records both store versions, as separate keys rather than a
combined number: they are independent counters, and a max would hide a bump in
whichever store happens to sit lower. A marker written before the split has no
marketdata version, reads as 0, and busts once — correct rather than merely
tolerated, since the price source moved underneath those caches.

Verified the wider frame is safe rather than assuming it: marketdata's futures
path returns the stored columns without cotdata's keep-list filter, and all
three consumers select columns explicitly, so the extra reconstruction columns
pass through untouched.

238 tests pass against real cotdata and marketdata installs, ruff clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2
Every job went red at install. This repo resolves its internal siblings by
checking them out and installing editable, and marketdata has no PyPI release,
so declaring the dependency without adding the checkout makes `pip install -e .`
fail before a single test runs.

A dummy MARKETDATA_STORE joins the dummy COTDATA_STORE for the same reason the
first one exists: neither package defaults a missing store root to somewhere
plausible, both raise by name, and there are two roots now.

The local run passed because the sibling was already installed in the venv. CI
builds the environment from scratch, which is the only place a missing
distribution shows up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2
@mspinola
mspinola marked this pull request as ready for review August 9, 2026 02:30
@mspinola
mspinola merged commit 95dad5a into main Aug 9, 2026
4 checks passed
@mspinola
mspinola deleted the claude/adr7-step4-repoint-prices branch August 9, 2026 03:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant