Skip to content

Port the Norgate producer tests §7.1 left behind - #13

Merged
mspinola merged 1 commit into
mainfrom
claude/require-final-gate
Aug 9, 2026
Merged

Port the Norgate producer tests §7.1 left behind#13
mspinola merged 1 commit into
mainfrom
claude/require-final-gate

Conversation

@mspinola

@mspinola mspinola commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Rewritten. This PR originally also wired --require-final into the CLI. #12 landed that four minutes before this branch was pushed — written independently, converging on the same design, and #12 is the better version (it also keeps --final-cutoff accepted-and-ignored so a scheduler carrying cotdata's flag does not break, and prints a per-reference defer table). Rebased onto the new main with the duplicate dropped. What remains is the part nothing else covers.

Why

§7.1 brought the Norgate provider across from cotdata but not its tests. The suite here covers what is a pure function — _finals_ready_by_date, _check_roll_gaps, _volume_passthrough, _targets — and, since #12, the finals gate end to end. Nothing covers the code that only runs when update() drives it.

cotdata still had those tests, so the gap was invisible. It surfaced when cotdata#105 (ADR-0007 §7.5) went to delete them: that deletion is the moment these behaviours stop being tested anywhere.

A file-count check would have missed this entirely. The test files existed on both sides. Diffing the test names gave zero overlap across 30 deleted tests — which reads like a naming difference and was, underneath, a real coverage gap.

What is covered now

  • reconstruction attaches its columns without disturbing front-month Volume
  • First/Second are picked by volume rank, not expiry order (the GC/SI case: the nearest serial month is near-empty while a later contract carries the flow)
  • an incremental run keeps rows it did not re-fetch, including ones previously recorded raw
  • full=True really does bypass the trailing window — checked at the fetch, which is what proves it
  • update() writes both stored tiers, from the right suffixed symbols (&ES_CCB then &ES)
  • update() / update_metadata() abort before any fetch when NDU is down
  • an all-null spec row is skipped rather than persisted

Two that go beyond a straight port

Both because a guard that fails open is worse than no guard:

  • A raising NDU probe must read as unreachable. The guard exists because norgatedata fails in unusual ways; a probe that blows up must not be read as a pass.
  • The finals gate must compare dates whatever form it is handed. Its two sides arrive as a datetime (Norgate's bar index) and a string (the store's manifest). Not a duplicate of test_cli_finals.py, which drives the gate with those already normalised.

Notes

Scoped with symbols=["ES"] against the real registry rather than a mocked all_symbols, so the tests drive the same selection path production does.

norgatedata is mocked into sys.modules, so this runs on any OS.

Verification

ruff check src tests scripts clean. MARKETDATA_NO_NETWORK=1 pytest tests/152 passed, 12 skipped (141 on main + the 11 here).

§7.1 brought the provider across from cotdata but not its tests. The suite here
covers what is a pure function -- _finals_ready_by_date, _check_roll_gaps,
_volume_passthrough, _targets -- and, since #12, the finals gate end to end. What
nothing covered is the code that only runs when update() drives it. cotdata still
had those tests, so the gap was invisible until §7.5 went to delete them: that
deletion is the moment the behaviours stop being tested anywhere.

Ported, adapted to the tier-aware store (write_bars(sym, df, domain=, source=,
tier=) where cotdata had write_prices(sym, tier, df, source=)):

  * reconstruction attaches its columns without disturbing front-month Volume
  * First/Second are picked by VOLUME RANK, not expiry order (the GC/SI case)
  * an incremental run keeps rows it did not re-fetch, including 'raw' ones
  * full=True really does bypass the trailing window, checked at the fetch
  * update() writes both stored tiers, from the right suffixed symbols
  * update()/update_metadata() abort BEFORE any fetch when NDU is down
  * an all-null spec row is skipped rather than persisted

Two go beyond a straight port, both because a guard that fails open is worse than
no guard. The NDU probe raising is now asserted to read as unreachable, and the
finals gate is asserted to compare dates whatever form it is handed -- its two
sides arrive as a datetime and a manifest string, which is an input-shape case
test_cli_finals.py does not reach because it drives the gate already normalised.

Scoped with symbols=["ES"] against the real registry rather than a mocked
all_symbols, so the tests drive the same selection path production does.

ruff clean, 152 pass and 12 skip (141 + the 11 here).
@mspinola
mspinola force-pushed the claude/require-final-gate branch from 1654b22 to 84a3417 Compare August 9, 2026 13:41
@mspinola mspinola changed the title Wire --require-final into the producer CLI Port the Norgate producer tests §7.1 left behind Aug 9, 2026
@mspinola
mspinola marked this pull request as ready for review August 9, 2026 13:49
@mspinola
mspinola merged commit fc5f07d into main Aug 9, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants