Skip to content

ADR-0007 step 2: record that the marketdata futures provider shipped - #103

Merged
mspinola merged 4 commits into
mainfrom
claude/market-data-cot-extraction-y7gydq
Aug 9, 2026
Merged

ADR-0007 step 2: record that the marketdata futures provider shipped#103
mspinola merged 4 commits into
mainfrom
claude/market-data-cot-extraction-y7gydq

Conversation

@mspinola

@mspinola mspinola commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Description

Starts ADR-0007 step 2 — the market-data extraction out of cotdata. No cotdata code changes here. The code landed in mspinola/marketdata#7. This PR is the cotdata-side record: a new handoff, plus the close-out of the 2026-08-04 work order that claimed the step.

Deletion of cotdata's price code is §7.5 of that work order and comes after the consumers are repointed. Until then cotdata is still every consumer's only working path, so it is untouched on purpose.

What shipped in marketdata

providers/norgate.py (both tiers, volume reconstruction, data-driven finals gate, NDU-down guard, contract specs), a store tier axis, adjust.ratio_adjust for propadj, the futures path in get_bars, contract-spec read/write/upsert, 49 futures registry symbols, and --domain / --metadata / --full on the producer CLI. 89 tests, ruff clean.

The store could not hold the result

The work order's §4 established that crowdmon needs both stored tiers because propadj is derived from the pair — read as a statement about what the producer must fetch. It is also a statement about what the store must hold, and marketdata's store could not: bars/<domain>/<source>/<symbol>.parquet gave both tiers the same path, so the second write would have silently replaced the first.

The one-frame-per-symbol layout rests on a property equities never violate — corporate actions are dated events the vendor hands over with the bars. Norgate's back-adjustment is roll splicing the vendor performed, and the stitched spread appears in no other series, so backadj cannot be derived from unadj or the reverse. That is a schema change (v1 → v2), not a provider detail.

Measured before writing anything: on v1, get_bars("ES", "backadj") raised tier must be one of ('split', 'raw', 'total'). check_tier accepted the futures tier and adjust() then rejected it, so the declared futures domain was a promise about error messages rather than a working path.

crowdmon was deprecated three days after the work order was written

Checked because CLAUDE.md mentions it in passing. It is not a passing matter — the work order's §4 is the section it says it exists to record, and it reasons entirely from crowdmon.

crowdmon/DEPRECATED.md decides it 2026-08-07, one day after that work order's own crowdmon companion. Its §3 was resolved 2026-08-08: both launchd jobs unloaded, cot-analyzer's /damage page removed (that repo's PR #22), the open work order closed unstarted, and the package left with no consumers at all.

The both-tiers rule is not relaxed — the provider enforces it three ways. Its argument moves to npf/books/treasury_seasonal.py, a live book that sets RETURN_TIER = "propadj" after a void first run in which 15 of ZB's 100 verdict-window trades had their sign inverted, and which reads backadj and unadj together for roll detection regardless.

§7.3 is void. Repointing a frozen package with no consumers is waste, and DEPRECATED.md §2 wants its live pins neutralised rather than given a post-freeze dependency. That costs something worth naming: crowdmon was to go first because it fails loudest, so dropping it drops the check that would have proved the provider before the easy consumers leaned on it. Nothing exercises propadj until the deferred npf pass, which makes the Windows-box comparison the main evidence the port preserved the numbers.

Convention note

The work order body is preserved verbatim and the corrections are appended as its §8, per the handoff register's rule — amending a body after execution destroys the record of what was asked. §4 and §7.3 carry pointers into §8, which is what that convention asks for directly.

Explicitly not done

§7.4 repoint cotmetrics/cot-analyzer (the only repoint left standing), §7.5 delete from cotdata. MME/MFS are not ported — Norgate carries no continuous series for either. Confirmed this costs crowdmon nothing: both already fail coverage() for the same reason, pinned by its live test as the only two non-joinable, absent from both vintage panels, and heldout in the deployed params. Counts agree — 49 of 51 joinable, 49 ported.

The provider has not run against a live Norgate. It could not, on Linux, for the same reason the work order's §5.3 concludes a synced store is permanent.

One open question I did not resolve: ADR-0007 defers npf on live-book grounds, decided when npf was one of two propadj consumers. It is now the only one. That is ADR-0007's call, and crucible-stack is outside this session's scope, so §8.2 records it rather than answering it.

Fixes # (issue) — n/a, this executes crucible-stack ADR-0007 via docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update — it is the documentation

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have run tests (e.g. pytest) and they pass locally — 89 pass in marketdata; no cotdata code changed
  • Any dependent changes have been merged and published in downstream modules — Norgate futures provider (ADR-0007 step 2) marketdata#7 is open, and no consumer is repointed yet

Step 1 of the 2026-08-04 work order's suggested order is done: marketdata now
has a Norgate futures provider, and contract specs (step 2) came with it. No
cotdata code changes — deletion is step 5 and comes after the consumers are
repointed, so every consumer's only working path is still this package.

The handoff records one finding the work order did not anticipate. Its §4
established that crowdmon needs both stored tiers because propadj is derived
from the pair, read as a statement about what the producer must fetch. It is
also a statement about what the store must hold, and marketdata's store could
not: one path per symbol meant the two tiers collided and the second write
would have silently replaced the first. Fixing that is a schema change, not a
provider detail, so it belongs in the record next to the work order that
scoped this.

Also notes what deliberately did not ship (MME/MFS have no Norgate series and
are absent rather than present-and-broken) and that the provider has not yet
run against a live Norgate — it could not, on Linux, for the same reason step
5.3 concludes a synced store is permanent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2
mspinola and others added 3 commits August 8, 2026 23:53
…upling

The previous commit flagged MME/MFS as a gap to confirm before repointing
crowdmon. Confirmed, and it is not a gap: both already fail coverage() for the
same reason they are unported. norgate: null means no stored tiers, so they
report missing: specs,unadj_price,backadj_price and joinable: False today, a
live test pins them as the only two out, the spec inventory measured them on
neither vintage panel, and both are heldout in the deployed params. The counts
agree — 49 of 51 joinable against 49 ported — so the futures registry holds
exactly crowdmon's joinable set.

The rows survive the repoint too: symbols keep coming from cotdata's registry
because COT identity stays here, so coverage() still lists them and still calls
them non-joinable.

Checking that surfaced something the work order's call-site table does not
capture. ContractMaster.load() parses the SHAPE of cotdata's manifest, not just
its functions: rpartition("_") over the "prices" keys. marketdata's keys live
under "bars" and carry a domain and vendor prefix, so that parse yields
"futures/norgate/ES" and matches nothing, and every symbol goes non-joinable.
It fails loudly rather than silently, but swapping the import will not be
enough and the coupling is a format rather than a call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2
…e to move

Checked because cotdata's CLAUDE.md mentions it in passing. It is not a passing
matter: the 2026-08-04 work order's §4 is the section it says it exists to
record, and it rests entirely on crowdmon.

crowdmon/DEPRECATED.md decides it on 2026-08-07, one day after that work
order's own crowdmon companion. Its §3 was resolved on 2026-08-08, the day the
provider was written: both launchd jobs unloaded, cot-analyzer's /damage page
removed, the open work order closed unstarted, and the package left with no
consumers at all.

The both-tiers rule survives and is not relaxed. It just cannot be argued from
crowdmon any more, and it does not need to be. npf/books/treasury_seasonal.py
is a live book that sets RETURN_TIER = propadj after its first run came back
void: on the verdict window ZB's backadj close is negative on 454 days and 15
of ZB's 100 trades had their sign inverted. It also reads backadj and unadj
together for roll detection, so it needs both stored frames independently of
propadj. Fifteen sign-inverted trades in a live book is a sharper argument than
a raise in a job that no longer runs.

What does change is §7.3. Repointing a frozen package with zero consumers is
waste, and DEPRECATED.md §2 asks for the opposite — it wants the live pins
neutralised because a frozen repo should not depend on data collected after the
freeze. So crowdmon's ten call sites leave the step, and with them the "hard
one first" ordering that was meant to de-risk everything after it. What remains
is the backadj-only repoint the work order calls easy, plus npf, which ADR-0007
defers with livebook. Nothing exercises propadj until that deferred pass runs,
which makes the Windows-box comparison the main evidence the port preserved the
numbers, and an argument for running it sooner.

Whether npf's deferral still makes sense now that it is the only propadj
consumer is ADR-0007's call, not this handoff's. Flagged because the deferral
was decided when it was one of two.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2
The register convention is to preserve a work order verbatim and append an
outcome, because amending the body destroys the record of what was actually
asked — which is the only thing that makes "the data contradicted the brief" a
checkable claim. So §4 and §7.3 keep their prose and gain pointers, and the
corrections live in an appended §8. That is also what the convention asks for
directly: where a handoff was wrong, the outcome says so and the body carries
the pointer, so a reader starting from the work order is not left with the
premises it opened with.

§8.1 — §4's constraint stands and the provider enforces it three ways. What
fails is its argument: it reasons entirely from crowdmon, deprecated 2026-08-07
and consumer-less by 2026-08-08, so its closing warning describes a job that no
longer runs. Re-anchored on npf's treasury seasonal book, which sets propadj
after a void run in which 15 of ZB's 100 verdict-window trades had their sign
inverted, and which reads both stored frames for roll detection regardless. §4's
which-box table is untouched: it turns on Norgate supplying all the tiers, not
on who consumes them.

§8.2 — §7.3 is void. Repointing a frozen package with no consumers is waste,
and DEPRECATED.md §2 asks for its live pins to be neutralised rather than given
a new post-freeze dependency. Records what that costs: crowdmon was to go first
precisely because it fails loudest, so dropping it drops the check that would
have proved the provider before the easy consumers leaned on it.

§8.3 and §8.4 carry the manifest-format coupling and the MME/MFS answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2
@mspinola
mspinola marked this pull request as ready for review August 9, 2026 01:23
@mspinola
mspinola merged commit da4da21 into main Aug 9, 2026
5 checks passed
@mspinola
mspinola deleted the claude/market-data-cot-extraction-y7gydq branch August 9, 2026 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant