Skip to content

Record the Windows verification of the marketdata futures port - #104

Merged
mspinola merged 1 commit into
mainfrom
claude/market-data-cot-extraction-y7gydq
Aug 9, 2026
Merged

Record the Windows verification of the marketdata futures port#104
mspinola merged 1 commit into
mainfrom
claude/market-data-cot-extraction-y7gydq

Conversation

@mspinola

@mspinola mspinola commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Description

Follow-up to #103 (merged), which shipped the marketdata futures provider and left docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md §7 reading "Not yet run against real Norgate" with the first commands for whoever picked it up.

It has now run, on the Windows producer. §7 carries the result instead of the instruction.

Docs only — no cotdata code changes, same as #103.

The verdict

marketdata/scripts/verify_against_cotdata.py against a cotdata store built by the original producer:

symbol rows per tier passthrough reconstruction
ES 7,279 identical identical
CL 10,887 identical identical
GC 12,156 identical identical
ZS 12,271 identical identical
DC 7,299 identical identical

49,892 rows per tier, both tiers, plus contract specs for all five, exit 0. Exact equality, not a tolerance — both producers drive the same Norgate install through two code paths, so any difference would have been a port bug rather than vendor disagreement. Symbols span an index, an energy, a metal, and the two markets whose backadj history goes non-positive.

That is what §7.5 needs before cotdata's price code is deleted, and it was obtainable only while both halves exist. It is on the record now, so the deletion no longer waits on it.

What the real box found that the offline suite could not

Two defects, on the first two contacts, and the pattern matters more than either:

  1. --domain futures stopped at the import guard. The provider was ported without its dependency — cotdata declares norgate = ["norgatedata"] and marketdata had no such extra, so nothing installed it.
  2. get_bars had no volume= parameter. The producer half of volume reconstruction was ported and the consumer half was not, so the columns were written and nothing served them. npf's ml/labels.py:50 passes volume= through, so a repointed call would have raised TypeError.

Neither is visible to a suite that cannot install the vendor or call a parameter that does not exist. §7 states the consequence for the remaining steps: offline green says nothing about the producer box, and each repoint in §7.4/§7.5 should be exercised there before it is called done.

The second was caught only because the harness was changed to report the columns it had not compared. It had been printing reconstruction columns solely when they differed, so "compared and identical" and "never compared" rendered as the same silence.

One expectation this corrected

The reconstruction columns were expected to drift, since each producer reconstructs incrementally over its own store's history and marketdata's was fresh where cotdata's had months of accumulation. They agree exactly: Norgate's historical individual-contract volumes are immutable and the algorithm is identical, so the incremental path converges on what a full recompute produces.

Fixes # (issue) — n/a, this executes crucible-stack ADR-0007 via docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update — it is the documentation

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have run tests (e.g. pytest) and they pass locally — 119 pass in marketdata; no cotdata code changed
  • Any dependent changes have been merged and published in downstream modules — Norgate futures provider (ADR-0007 step 2) marketdata#7 is open; no consumer is repointed yet

Generated by Claude Code

…utput

The handoff said "not yet run against real Norgate" and gave the first commands
for whoever picked it up. It has now run, and §7 carries the result rather than
the instruction: 49,892 rows per tier across ES, CL, GC, ZS and DC, both tiers,
plus contract specs, all identical, exit 0. Exact equality rather than a
tolerance, both producers driving the same Norgate install through two code
paths.

That is what §7.5 needs before cotdata's price code is deleted, and it was
obtainable only while both halves exist. It is on the record now, so the
deletion no longer waits on it.

Two defects surfaced on the first two contacts with the box, and the pattern
matters more than either. The provider was ported without its dependency, so
--domain futures stopped at the import guard with nothing installing
norgatedata. And get_bars had no volume= parameter: the producer half of volume
reconstruction was ported, the consumer half was not, so the columns were
written and nothing served them — npf's ml/labels.py passes volume= through, so
a repoint would have raised TypeError.

Neither is visible to a suite that cannot install the vendor or call a
parameter that does not exist. §7 states the consequence for the remaining
steps: offline green says nothing about the producer box, and each repoint
should be exercised there before it is called done.

One expectation corrected. The reconstruction columns were expected to drift,
each producer reconstructing incrementally over its own store's history. They
agree exactly, because Norgate's historical individual-contract volumes are
immutable and the algorithm is identical, so incremental converges on full.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2
@mspinola
mspinola marked this pull request as ready for review August 9, 2026 02:07
@mspinola
mspinola merged commit 5774938 into main Aug 9, 2026
5 checks passed
@mspinola
mspinola deleted the claude/market-data-cot-extraction-y7gydq branch August 9, 2026 02:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant