Read bars from marketdata, not cotdata (ADR-0007 step 4) - #11
Merged
Conversation
Three price call sites move from cotdata.get_prices to marketdata.get_bars, all of them adjustment="backadj", which is what ADR-0007 measured. Tier names and the returned frame are unchanged, so the reads themselves are a rename. COT positioning still comes from cotdata; only bars moved. The part that is not a rename is the cache marker, and it would have failed silently. CotIndexer busts its parquet caches when the upstream store schema moves, because the per-symbol guards key on column presence and cannot see a value-only change. The example that guard was written for is named in its own comment: reconstructed volume being promoted, which was a PRICE schema bump. So after this repoint, watching cotdata alone leaves the exact case the guard exists for uncovered — stale metrics computed against a superseded bar schema, with no error anywhere. The marker now records both store versions, as separate keys rather than a combined number: they are independent counters, and a max would hide a bump in whichever store happens to sit lower. A marker written before the split has no marketdata version, reads as 0, and busts once — correct rather than merely tolerated, since the price source moved underneath those caches. Verified the wider frame is safe rather than assuming it: marketdata's futures path returns the stored columns without cotdata's keep-list filter, and all three consumers select columns explicitly, so the extra reconstruction columns pass through untouched. 238 tests pass against real cotdata and marketdata installs, ruff clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2
Every job went red at install. This repo resolves its internal siblings by checking them out and installing editable, and marketdata has no PyPI release, so declaring the dependency without adding the checkout makes `pip install -e .` fail before a single test runs. A dummy MARKETDATA_STORE joins the dummy COTDATA_STORE for the same reason the first one exists: neither package defaults a missing store root to somewhere plausible, both raise by name, and there are two roots now. The local run passed because the sibling was already installed in the venv. CI builds the environment from scratch, which is the only place a missing distribution shows up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ADR-0007 makes
cotdataCFTC positioning only and moves every bar tomarketdata. The provider shipped and was verified againstcotdata's own output on the Windows box (mspinola/marketdata#7, mspinola/marketdata#8). This is the consumer side.COT positioning still comes from
cotdata. Only bars moved.Companion: mspinola/cot-analyzer#26 — these two land together, and this is the one that changes reads.
The three call sites
All
adjustment="backadj", which is what ADR-0007 measured, so the reads themselves are a rename:signals.py:1028cotdata.get_prices(symbol, adjustment="backadj")marketdata.get_bars(symbol, "backadj")options_data.py:366CotIndexer.py:719…, start=start_date…, start=start_dateTier names and the returned frame are unchanged.
The part that is not a rename, and would have failed silently
CotIndexerbusts its parquet caches when the upstream store schema moves, because the per-symbol guards key on column presence and cannot see a value-only change. The example that guard was written for is named in its own comment: reconstructed volume being promoted — a price schema bump.Prices are no longer in
cotdata. So after this repoint, watchingcotdata.schema_version()alone leaves uncovered the exact case the guard exists for: stale cached metrics computed against a superseded bar schema, with no error anywhere.The marker now records both store versions, as separate keys rather than one combined number — they are independent counters, and a max would hide a bump in whichever store happens to sit lower. A marker written before the split has no marketdata version, reads as 0, and busts once. That is correct rather than merely tolerated: the price source moved underneath those caches.
Two tests cover it, including the pre-split marker case.
Verified rather than assumed
marketdata's futures path returns the stored columns withoutcotdata's keep-list filter, so the frame is wider — it carriesFirstVolume,SecondVolume,FirstContract,SecondContract,Volume_Reconstructed,Volume_Source. All three consumers select columns explicitly (price_data[['Open','High','Low','Close']],df['Close']), so the extra columns pass through untouched. Checked before repointing rather than after a failure.Testing
238 pass, ruff clean, CI green on 3.10–3.13. Run locally against real
cotdataandmarketdatainstalls rather than mocks, so the import swap and the new schema call are actually exercised. Thesignalstest doubles were repointed too (signals.marketdata.get_bars), since a stale monkeypatch target would have silently stopped patching and let the tests hit the real store.CI needed a
marketdatasibling checkout: it is not on PyPI, so the declared dependency resolves only from the workspace, and without the checkoutpip install -e .failed before a single test ran. A dummyMARKETDATA_STOREjoins the dummyCOTDATA_STOREfor the same reason the first exists — neither package defaults a missing root, both raise by name, and there are two roots now.Deployment note
The Linux server needs
MARKETDATA_STOREset and a second synced store — wired up in mspinola/cot-analyzer#26. Both stores are synced, not produced there.