ADR-0007 step 2: record that the marketdata futures provider shipped - #103
Merged
Conversation
Step 1 of the 2026-08-04 work order's suggested order is done: marketdata now has a Norgate futures provider, and contract specs (step 2) came with it. No cotdata code changes — deletion is step 5 and comes after the consumers are repointed, so every consumer's only working path is still this package. The handoff records one finding the work order did not anticipate. Its §4 established that crowdmon needs both stored tiers because propadj is derived from the pair, read as a statement about what the producer must fetch. It is also a statement about what the store must hold, and marketdata's store could not: one path per symbol meant the two tiers collided and the second write would have silently replaced the first. Fixing that is a schema change, not a provider detail, so it belongs in the record next to the work order that scoped this. Also notes what deliberately did not ship (MME/MFS have no Norgate series and are absent rather than present-and-broken) and that the provider has not yet run against a live Norgate — it could not, on Linux, for the same reason step 5.3 concludes a synced store is permanent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2
…upling
The previous commit flagged MME/MFS as a gap to confirm before repointing
crowdmon. Confirmed, and it is not a gap: both already fail coverage() for the
same reason they are unported. norgate: null means no stored tiers, so they
report missing: specs,unadj_price,backadj_price and joinable: False today, a
live test pins them as the only two out, the spec inventory measured them on
neither vintage panel, and both are heldout in the deployed params. The counts
agree — 49 of 51 joinable against 49 ported — so the futures registry holds
exactly crowdmon's joinable set.
The rows survive the repoint too: symbols keep coming from cotdata's registry
because COT identity stays here, so coverage() still lists them and still calls
them non-joinable.
Checking that surfaced something the work order's call-site table does not
capture. ContractMaster.load() parses the SHAPE of cotdata's manifest, not just
its functions: rpartition("_") over the "prices" keys. marketdata's keys live
under "bars" and carry a domain and vendor prefix, so that parse yields
"futures/norgate/ES" and matches nothing, and every symbol goes non-joinable.
It fails loudly rather than silently, but swapping the import will not be
enough and the coupling is a format rather than a call.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2
…e to move Checked because cotdata's CLAUDE.md mentions it in passing. It is not a passing matter: the 2026-08-04 work order's §4 is the section it says it exists to record, and it rests entirely on crowdmon. crowdmon/DEPRECATED.md decides it on 2026-08-07, one day after that work order's own crowdmon companion. Its §3 was resolved on 2026-08-08, the day the provider was written: both launchd jobs unloaded, cot-analyzer's /damage page removed, the open work order closed unstarted, and the package left with no consumers at all. The both-tiers rule survives and is not relaxed. It just cannot be argued from crowdmon any more, and it does not need to be. npf/books/treasury_seasonal.py is a live book that sets RETURN_TIER = propadj after its first run came back void: on the verdict window ZB's backadj close is negative on 454 days and 15 of ZB's 100 trades had their sign inverted. It also reads backadj and unadj together for roll detection, so it needs both stored frames independently of propadj. Fifteen sign-inverted trades in a live book is a sharper argument than a raise in a job that no longer runs. What does change is §7.3. Repointing a frozen package with zero consumers is waste, and DEPRECATED.md §2 asks for the opposite — it wants the live pins neutralised because a frozen repo should not depend on data collected after the freeze. So crowdmon's ten call sites leave the step, and with them the "hard one first" ordering that was meant to de-risk everything after it. What remains is the backadj-only repoint the work order calls easy, plus npf, which ADR-0007 defers with livebook. Nothing exercises propadj until that deferred pass runs, which makes the Windows-box comparison the main evidence the port preserved the numbers, and an argument for running it sooner. Whether npf's deferral still makes sense now that it is the only propadj consumer is ADR-0007's call, not this handoff's. Flagged because the deferral was decided when it was one of two. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2
The register convention is to preserve a work order verbatim and append an outcome, because amending the body destroys the record of what was actually asked — which is the only thing that makes "the data contradicted the brief" a checkable claim. So §4 and §7.3 keep their prose and gain pointers, and the corrections live in an appended §8. That is also what the convention asks for directly: where a handoff was wrong, the outcome says so and the body carries the pointer, so a reader starting from the work order is not left with the premises it opened with. §8.1 — §4's constraint stands and the provider enforces it three ways. What fails is its argument: it reasons entirely from crowdmon, deprecated 2026-08-07 and consumer-less by 2026-08-08, so its closing warning describes a job that no longer runs. Re-anchored on npf's treasury seasonal book, which sets propadj after a void run in which 15 of ZB's 100 verdict-window trades had their sign inverted, and which reads both stored frames for roll detection regardless. §4's which-box table is untouched: it turns on Norgate supplying all the tiers, not on who consumes them. §8.2 — §7.3 is void. Repointing a frozen package with no consumers is waste, and DEPRECATED.md §2 asks for its live pins to be neutralised rather than given a new post-freeze dependency. Records what that costs: crowdmon was to go first precisely because it fails loudest, so dropping it drops the check that would have proved the provider before the easy consumers leaned on it. §8.3 and §8.4 carry the manifest-format coupling and the MME/MFS answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Starts ADR-0007 step 2 — the market-data extraction out of
cotdata. Nocotdatacode changes here. The code landed in mspinola/marketdata#7. This PR is thecotdata-side record: a new handoff, plus the close-out of the 2026-08-04 work order that claimed the step.Deletion of
cotdata's price code is §7.5 of that work order and comes after the consumers are repointed. Until thencotdatais still every consumer's only working path, so it is untouched on purpose.What shipped in
marketdataproviders/norgate.py(both tiers, volume reconstruction, data-driven finals gate, NDU-down guard, contract specs), a store tier axis,adjust.ratio_adjustforpropadj, the futures path inget_bars, contract-spec read/write/upsert, 49 futures registry symbols, and--domain/--metadata/--fullon the producer CLI. 89 tests, ruff clean.The store could not hold the result
The work order's §4 established that
crowdmonneeds both stored tiers becausepropadjis derived from the pair — read as a statement about what the producer must fetch. It is also a statement about what the store must hold, andmarketdata's store could not:bars/<domain>/<source>/<symbol>.parquetgave both tiers the same path, so the second write would have silently replaced the first.The one-frame-per-symbol layout rests on a property equities never violate — corporate actions are dated events the vendor hands over with the bars. Norgate's back-adjustment is roll splicing the vendor performed, and the stitched spread appears in no other series, so
backadjcannot be derived fromunadjor the reverse. That is a schema change (v1 → v2), not a provider detail.Measured before writing anything: on v1,
get_bars("ES", "backadj")raisedtier must be one of ('split', 'raw', 'total').check_tieraccepted the futures tier andadjust()then rejected it, so the declared futures domain was a promise about error messages rather than a working path.crowdmonwas deprecated three days after the work order was writtenChecked because
CLAUDE.mdmentions it in passing. It is not a passing matter — the work order's §4 is the section it says it exists to record, and it reasons entirely fromcrowdmon.crowdmon/DEPRECATED.mddecides it 2026-08-07, one day after that work order's owncrowdmoncompanion. Its §3 was resolved 2026-08-08: both launchd jobs unloaded,cot-analyzer's/damagepage removed (that repo's PR #22), the open work order closed unstarted, and the package left with no consumers at all.The both-tiers rule is not relaxed — the provider enforces it three ways. Its argument moves to
npf/books/treasury_seasonal.py, a live book that setsRETURN_TIER = "propadj"after a void first run in which 15 of ZB's 100 verdict-window trades had their sign inverted, and which readsbackadjandunadjtogether for roll detection regardless.§7.3 is void. Repointing a frozen package with no consumers is waste, and
DEPRECATED.md§2 wants its live pins neutralised rather than given a post-freeze dependency. That costs something worth naming:crowdmonwas to go first because it fails loudest, so dropping it drops the check that would have proved the provider before the easy consumers leaned on it. Nothing exercisespropadjuntil the deferrednpfpass, which makes the Windows-box comparison the main evidence the port preserved the numbers.Convention note
The work order body is preserved verbatim and the corrections are appended as its §8, per the handoff register's rule — amending a body after execution destroys the record of what was asked. §4 and §7.3 carry pointers into §8, which is what that convention asks for directly.
Explicitly not done
§7.4 repoint
cotmetrics/cot-analyzer(the only repoint left standing), §7.5 delete fromcotdata.MME/MFSare not ported — Norgate carries no continuous series for either. Confirmed this costscrowdmonnothing: both already failcoverage()for the same reason, pinned by its live test as the only two non-joinable, absent from both vintage panels, andheldoutin the deployed params. Counts agree — 49 of 51 joinable, 49 ported.The provider has not run against a live Norgate. It could not, on Linux, for the same reason the work order's §5.3 concludes a synced store is permanent.
One open question I did not resolve: ADR-0007 defers
npfon live-book grounds, decided whennpfwas one of twopropadjconsumers. It is now the only one. That is ADR-0007's call, and crucible-stack is outside this session's scope, so §8.2 records it rather than answering it.Fixes # (issue) — n/a, this executes crucible-stack ADR-0007 via
docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md.Type of change
Checklist:
pytest) and they pass locally — 89 pass inmarketdata; nocotdatacode changed