From 36de91a39c291b70cbd376c49f1b3774f7ee2c64 Mon Sep 17 00:00:00 2001 From: "Matt S." Date: Sat, 8 Aug 2026 23:44:49 +0000 Subject: [PATCH 1/4] ADR-0007 step 2: record that the marketdata futures provider shipped MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 1 of the 2026-08-04 work order's suggested order is done: marketdata now has a Norgate futures provider, and contract specs (step 2) came with it. No cotdata code changes — deletion is step 5 and comes after the consumers are repointed, so every consumer's only working path is still this package. The handoff records one finding the work order did not anticipate. Its §4 established that crowdmon needs both stored tiers because propadj is derived from the pair, read as a statement about what the producer must fetch. It is also a statement about what the store must hold, and marketdata's store could not: one path per symbol meant the two tiers collided and the second write would have silently replaced the first. Fixing that is a schema change, not a provider detail, so it belongs in the record next to the work order that scoped this. Also notes what deliberately did not ship (MME/MFS have no Norgate series and are absent rather than present-and-broken) and that the provider has not yet run against a live Norgate — it could not, on Linux, for the same reason step 5.3 concludes a synced store is permanent. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2 --- ...6-08-04-adr7-step2-price-producer-split.md | 6 +- .../2026-08-08-adr7-step2-provider-shipped.md | 169 ++++++++++++++++++ 2 files changed, 174 insertions(+), 1 deletion(-) create mode 100644 docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md diff --git a/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md b/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md index 571c4a1..de31407 100644 --- a/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md +++ b/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md @@ -1,6 +1,10 @@ # Handoff: ADR-0007 step 2, move price production out of `cotdata` -**Status:** **CLAIMED, NOT STARTED.** This is a work order and a re-measurement, no code +**Status:** **STEP 1 SHIPPED (2026-08-08).** The `marketdata` futures provider exists. +Steps 2–5 of §7 (contract specs are done; consumers not yet repointed; nothing deleted +from `cotdata`) remain. See `docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md`, +which also records a store-layout finding this document did not anticipate. +Originally: **CLAIMED, NOT STARTED** — a work order and a re-measurement, no code **Date:** 2026-08-04 **Lives at:** `cotdata/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md` **Target:** a Claude Code session in a `cotdata` worktree, with `marketdata` beside it diff --git a/docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md b/docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md new file mode 100644 index 0000000..8fead57 --- /dev/null +++ b/docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md @@ -0,0 +1,169 @@ +# Handoff: ADR-0007 step 2, the `marketdata` futures provider is written + +**Status:** **STEP 1 OF §7 SHIPPED.** Contract specs (§7.2) came with it. Steps §7.3–§7.5 +— repoint `crowdmon`, repoint `cotmetrics`/`cot-analyzer`, delete from `cotdata` — are +**not started** +**Date:** 2026-08-08 +**Lives at:** `cotdata/docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md` +**Executes:** [crucible-stack ADR-0007](https://github.com/mspinola/crucible-stack/blob/main/docs/adr/ADR-0007-cotdata-is-cot-only-bars-live-in-marketdata.md), +accepted 2026-07-27, via the work order in +`docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md` +**Code landed in:** `mspinola/marketdata`. **Nothing in `cotdata` changed** — its price +code still works and every consumer still reads it. That is deliberate: deletion is §7.5, +and it comes after the repointing, not before + +--- + +## 0. What this adds to the 2026-08-04 work order + +One finding, and it is structural rather than a detail. + +The work order said step 2 is *write the Norgate futures provider into `marketdata`* +rather than *move a file across a seam*, and priced that difference honestly. It did not +anticipate that **`marketdata`'s store could not hold the result**, and that the fix is a +schema change rather than a provider. + +## 1. The finding: one stored frame per symbol was an equities assumption + +`marketdata`'s store path was `bars///.parquet`. One file per +symbol per vendor, and every adjustment tier derived on read. Its `docs/design.md` states +that posture plainly and credits it to cotdata's `propadj`. + +That design rests on a property nobody had written down because equities never violate +it: **corporate actions are dated events the vendor hands over with the bars.** One +stored frame plus `Dividends` and `Stock Splits` reconstructs any tier. + +Norgate's back-adjustment is not that. It is roll splicing the vendor performed, and the +stitched calendar spread at each roll appears in no other series it publishes. `backadj` +cannot be derived from `unadj`, or the reverse. The work order's §4 says the same thing +from the consumer side — `crowdmon` needs both stored tiers because `propadj` is derived +from the pair — but read as a statement about *what the producer must fetch*. It is also +a statement about **what the store must be able to hold**, and on 2026-08-04 it could not: +both tiers resolved to the same path and the second write would have silently replaced +the first. + +So the store grew a stored-tier component, `_.parquet`, used by futures and +absent for equities. `MARKETDATA_STORE` schema v1 → **v2**. An existing equity store is +extended, not migrated: the equity path is byte-identical and reads through the same code. + +**Measured before writing anything.** On v1, every futures read raised: + +``` +>>> get_bars("ES", "backadj", domain="futures", source="norgate") +ValueError: tier must be one of ('split', 'raw', 'total'), got 'backadj' +``` + +`check_tier` accepted the tier for the futures domain and `adjust()` then rejected it. +The declared `DOMAIN_TIERS["futures"]` entry, which ADR-0007 notes was added so error +messages would be right from day one, was a promise about error messages and not a +working path. Worth stating because "the futures domain is already declared" reads like +partial progress, and the honest measurement is that the consumer path was 0% built. + +## 2. What shipped + +In `marketdata`, all green (89 tests, ruff clean): + +| Piece | Note | +|---|---| +| `providers/norgate.py` | the provider. Both tiers, volume reconstruction, the data-driven finals gate, NDU-down guard, contract specs | +| store tier axis | `config.bars_path`, `store.{write,read,has}_bars(tier=)`, `sources_for(tier=)`. Schema v2 | +| `adjust.ratio_adjust` | `propadj`, ported from `cotdata.prices._ratio_adjust` as a pure two-frame function | +| `adjust.STORED_TIERS` / `DERIVED_TIERS` | which tiers a producer owes against which are computed. A test asserts they partition each domain | +| `bars.get_bars` futures path | stored tiers read through; `propadj` derived; half-stored raises | +| `store.{write,read,upsert}_metadata` | contract specs. The scoped-run upsert is ported too — specs share one table | +| `registry.yaml` | 49 futures symbols | +| `update.py` | `--domain`, `--metadata`, `--full` | +| `provenance` / `pin` | made tier-aware (see §4) | + +`get_prices(symbol, adjustment=...)` becomes `get_bars(symbol, adjustment)`. Tier names +are unchanged, so a repoint is an import change and not a semantic one. + +## 3. The both-tiers rule is enforced in three places + +The work order's §4 warned that shipping a `backadj`-only futures producer would break +`crowdmon` with a `raise` inside `riskunits` on a weekly scheduled job. Three guards, so +the rule cannot be lost by a later edit to any one of them: + +1. **Producer.** Both tiers are fetched and reconstructed before either is written, so a + failure on the second leaves nothing on disk rather than a half-written symbol. +2. **Consumer.** `propadj` against one stored tier raises and names the missing one. + Returning empty would read as "no data for this symbol" when the truth is "the + producer half-finished". +3. **Test.** Parametrised over both halves, asserting the message names the absent tier. + +Loudness is the whole point, and the reason is §4's: additive back-adjusted percent +volatility is ~200x too high for soybeans and **0.47x for gold**, and 0.47x never goes +negative and passes every implausibility screen a spot check would apply. + +## 4. Two things the port broke on the way through, both fixed + +Recorded because both were silent, and neither is in the work order. + +**`provenance()` and `--pin` assumed one series per symbol.** `provenance("ES")` returned +`None` for a futures symbol, and an unscoped `marketdata-update --pin` derived its symbol +list from manifest keys, so futures arrived as the "symbol" `ES_backadj` and then failed +lookup. Both are now tier-aware. Snapshot format v1 → **v2**: an entry is one stored +series, so `--symbols ES` pins `ES_backadj` *and* `ES_unadj`. Pinning one would leave +`propadj` half covered, and a study quoting a volatility figure would verify green against +a store that had moved under it. v1 snapshots still verify — their keys are plain symbols +and an absent `tier` field reads as the domain's default. + +**`marketdata-update --check` ragged every row.** Its symbol column was a fixed 10 +characters, and a futures entry is `futures/norgate/ES_backadj`. Width now comes from the +data. + +## 5. Answers to the work order's §5, as implemented + +**5.1 Does `propadj` stay derived-on-read?** **Yes, implemented that way.** The work order +recommended it on the voided month-end Treasury verdict, and §1 above is now an +independent second reason: it is the only futures tier that *can* be derived, so deriving +it is also the only thing that keeps the store's stated posture true anywhere in this +domain. + +**5.2 Do the two store roots converge?** **Not decided here, and this ships without +needing it.** `MARKETDATA_STORE` stays a separate root with a separate manifest. The +work order is right that three roots is the moment to decide — but that decision binds +launchers and a launchd agent in `crowdmon`, so it belongs with §7.3, not ahead of it. +Nothing here forecloses it. + +**5.3 Is a synced store permanent?** **Yes, and the code now says so rather than +implying it.** `norgatedata` drives a local Norgate Data Updater install and NDU is +Windows-only, so no other machine can produce this half at any Python version. +`--bars` therefore skips futures with a message on a non-Windows box instead of failing +the whole run, and `--domain futures` explains why rather than raising +`ModuleNotFoundError`. Worth writing into ADR-0007's open questions as resolved. + +## 6. Scope left out, deliberately + +- **§7.3 repoint `crowdmon`** (6 modules, 10 call sites) and **§7.4 `cotmetrics` / + `cot-analyzer`**. Not started. The work order says do `crowdmon` first because its tier + requirements are strictest and it fails loudest, and that ordering still holds. +- **§7.5 delete from `cotdata`.** Not started, and it must not be until the repointing + lands. `cotdata`'s price code is still every consumer's only working path. +- **`MME` / `MFS`.** Norgate carries no continuous series for either, so `cotdata` prices + them off the EEM and EFA ETF proxies through yfinance. Serving them in `marketdata` + needs a futures-domain path in the yfinance provider, which is separate work from the + Norgate producer. They are **absent** from the futures registry rather than present and + unserviceable. A consumer repointed at `marketdata` loses them until that is built — + worth confirming against `crowdmon`'s universe before §7.3. +- **Step 3 (`livebook`).** Still out, still a live book. + +## 7. Not yet run against real Norgate + +Every test is offline. The provider has not executed against a live NDU, because this +work happened on Linux and §5.3 is exactly the reason it could not. The pure logic — +`propadj` derivation, the finals-gate cores, the roll-gap check, the tier/store +plumbing — is covered by tests; the `norgatedata` call sites are ported from code that +has run in production in `cotdata` for months, but they are unexercised **here**. + +**First action for whoever picks this up on the Windows box:** + +``` +marketdata-update --bars --domain futures --symbols ES +marketdata-update --check +``` + +then compare `ES_backadj` against `cotdata`'s existing `ES_backadj` for the same dates. +Both producers can run side by side — separate roots, separate manifests, nothing +deleted — so that comparison is available until §7.5, and it is the cheapest possible +check that the port preserved the numbers. From b9b41430e836916a5eca1167425667c091a7e5ef Mon Sep 17 00:00:00 2001 From: "Matt S." Date: Sat, 8 Aug 2026 23:53:40 +0000 Subject: [PATCH 2/4] Answer the MME/MFS question against crowdmon, and find an eleventh coupling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit flagged MME/MFS as a gap to confirm before repointing crowdmon. Confirmed, and it is not a gap: both already fail coverage() for the same reason they are unported. norgate: null means no stored tiers, so they report missing: specs,unadj_price,backadj_price and joinable: False today, a live test pins them as the only two out, the spec inventory measured them on neither vintage panel, and both are heldout in the deployed params. The counts agree — 49 of 51 joinable against 49 ported — so the futures registry holds exactly crowdmon's joinable set. The rows survive the repoint too: symbols keep coming from cotdata's registry because COT identity stays here, so coverage() still lists them and still calls them non-joinable. Checking that surfaced something the work order's call-site table does not capture. ContractMaster.load() parses the SHAPE of cotdata's manifest, not just its functions: rpartition("_") over the "prices" keys. marketdata's keys live under "bars" and carry a domain and vendor prefix, so that parse yields "futures/norgate/ES" and matches nothing, and every symbol goes non-joinable. It fails loudly rather than silently, but swapping the import will not be enough and the coupling is a format rather than a call. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2 --- .../2026-08-08-adr7-step2-provider-shipped.md | 31 +++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md b/docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md index 8fead57..defd7e6 100644 --- a/docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md +++ b/docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md @@ -138,14 +138,41 @@ the whole run, and `--domain futures` explains why rather than raising - **§7.3 repoint `crowdmon`** (6 modules, 10 call sites) and **§7.4 `cotmetrics` / `cot-analyzer`**. Not started. The work order says do `crowdmon` first because its tier requirements are strictest and it fails loudest, and that ordering still holds. + + One repoint task the work order's call-site table does not capture, found while + checking the `MME`/`MFS` question. `ContractMaster.load()` does not only *call* + `cotdata`; it parses the shape of `cotdata`'s manifest: + + ```python + for name in load_manifest().get("prices", {}): + sym, _, adj = str(name).rpartition("_") # "ES_backadj" -> ("ES", "backadj") + ``` + + `marketdata`'s entries live under `"bars"` and read `futures/norgate/ES_backadj`, so + `rpartition("_")` yields `futures/norgate/ES` and matches no registry symbol. Every + symbol would go non-joinable. It fails loudly — `test_every_registry_symbol_but_the_ + uncovered_ones_joins` breaks — but swapping the import is not sufficient, and the + eleventh coupling is a manifest *format* rather than a function call. - **§7.5 delete from `cotdata`.** Not started, and it must not be until the repointing lands. `cotdata`'s price code is still every consumer's only working path. - **`MME` / `MFS`.** Norgate carries no continuous series for either, so `cotdata` prices them off the EEM and EFA ETF proxies through yfinance. Serving them in `marketdata` needs a futures-domain path in the yfinance provider, which is separate work from the Norgate producer. They are **absent** from the futures registry rather than present and - unserviceable. A consumer repointed at `marketdata` loses them until that is built — - worth confirming against `crowdmon`'s universe before §7.3. + unserviceable. + + **Checked against `crowdmon`, and this costs it nothing.** Both already fail + `contract_master.coverage()`, which requires a spec plus both stored tiers: they carry + `norgate: null`, so they report `missing: specs,unadj_price,backadj_price` and + `joinable: False` today. `tests/test_contract_master_live.py` pins exactly that + (`not_joinable <= {"MFS", "MME"}`), the 2026-08-04 spec inventory measured + `joinable-but-unseen []` and `seen-but-unjoinable []`, and both are `Role: heldout` in + the deployed `params.yaml`. `futures/roll.py:110` names them directly as the ETF proxies + with no Delivery Month, which raises rather than returning a wrong answer. The counts + agree: `crowdmon` reports 49 of 51 joinable and the ported futures registry holds 49. + After §7.3 the rows still appear — symbols keep coming from `cotdata`'s registry, since + COT identity stays here — and still read non-joinable, so the live test's assertion + survives the repoint unchanged. - **Step 3 (`livebook`).** Still out, still a live book. ## 7. Not yet run against real Norgate From 74f24bd33fe2ad6c33cbb8432d466dd9229569da Mon Sep 17 00:00:00 2001 From: "Matt S." Date: Sat, 8 Aug 2026 23:56:38 +0000 Subject: [PATCH 3/4] =?UTF-8?q?crowdmon=20is=20deprecated,=20so=20=C2=A74'?= =?UTF-8?q?s=20argument=20and=20=C2=A77.3's=20ordering=20both=20have=20to?= =?UTF-8?q?=20move?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Checked because cotdata's CLAUDE.md mentions it in passing. It is not a passing matter: the 2026-08-04 work order's §4 is the section it says it exists to record, and it rests entirely on crowdmon. crowdmon/DEPRECATED.md decides it on 2026-08-07, one day after that work order's own crowdmon companion. Its §3 was resolved on 2026-08-08, the day the provider was written: both launchd jobs unloaded, cot-analyzer's /damage page removed, the open work order closed unstarted, and the package left with no consumers at all. The both-tiers rule survives and is not relaxed. It just cannot be argued from crowdmon any more, and it does not need to be. npf/books/treasury_seasonal.py is a live book that sets RETURN_TIER = propadj after its first run came back void: on the verdict window ZB's backadj close is negative on 454 days and 15 of ZB's 100 trades had their sign inverted. It also reads backadj and unadj together for roll detection, so it needs both stored frames independently of propadj. Fifteen sign-inverted trades in a live book is a sharper argument than a raise in a job that no longer runs. What does change is §7.3. Repointing a frozen package with zero consumers is waste, and DEPRECATED.md §2 asks for the opposite — it wants the live pins neutralised because a frozen repo should not depend on data collected after the freeze. So crowdmon's ten call sites leave the step, and with them the "hard one first" ordering that was meant to de-risk everything after it. What remains is the backadj-only repoint the work order calls easy, plus npf, which ADR-0007 defers with livebook. Nothing exercises propadj until that deferred pass runs, which makes the Windows-box comparison the main evidence the port preserved the numbers, and an argument for running it sooner. Whether npf's deferral still makes sense now that it is the only propadj consumer is ADR-0007's call, not this handoff's. Flagged because the deferral was decided when it was one of two. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2 --- .../2026-08-08-adr7-step2-provider-shipped.md | 92 +++++++++++++++++-- 1 file changed, 84 insertions(+), 8 deletions(-) diff --git a/docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md b/docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md index defd7e6..83efeb0 100644 --- a/docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md +++ b/docs/handoffs/2026-08-08-adr7-step2-provider-shipped.md @@ -81,8 +81,9 @@ are unchanged, so a repoint is an import change and not a semantic one. ## 3. The both-tiers rule is enforced in three places The work order's §4 warned that shipping a `backadj`-only futures producer would break -`crowdmon` with a `raise` inside `riskunits` on a weekly scheduled job. Three guards, so -the rule cannot be lost by a later edit to any one of them: +`crowdmon` with a `raise` inside `riskunits` on a weekly scheduled job. **That consumer is +gone — see §5a — and the rule still holds, on `npf`'s evidence rather than `crowdmon`'s.** +Three guards, so it cannot be lost by a later edit to any one of them: 1. **Producer.** Both tiers are fetched and reconstructed before either is written, so a failure on the second leaves nothing on disk rather than a half-written symbol. @@ -133,15 +134,90 @@ Windows-only, so no other machine can produce this half at any Python version. the whole run, and `--domain futures` explains why rather than raising `ModuleNotFoundError`. Worth writing into ADR-0007's open questions as resolved. +## 5a. `crowdmon` was deprecated three days after the work order was written + +Checked because `cotdata`'s `CLAUDE.md` mentioned it in passing. It is not a passing +matter: **the work order's §4 — the section it says it exists to record — rests entirely +on `crowdmon`, and `crowdmon` is now inert.** + +`crowdmon/DEPRECATED.md`, decided **2026-08-07**, one day after the work order's own +`crowdmon` companion document. Four pre-registered tests, no positive result, and the §10 +validation came back uninformative with the hand-identified clean episodes **spent**. The +repo is frozen, not deleted, with three stated conditions for revisiting. + +Its §3 was resolved **2026-08-08**, the day this provider was written: + +- both launchd jobs (`crowdmon-publish`, `crowdmon-live-tests`) unloaded, plists deleted +- `cot-analyzer`'s `/damage` page removed in that repo's PR #22, with its artifact reader +- the one open work order closed unstarted +- **"This package now has no consumers at all; nothing in `npf` or `livebook` ever + imported it."** `~/code/crowdmon_store` is written by nothing and read by nothing + +### The requirement survives. Its justification has to move + +The both-tiers rule is **not** weakened by this, and §3 above should not be relaxed. But +it can no longer be argued from `crowdmon`, and it does not need to be — a live consumer +makes the same case harder. + +`npf/books/treasury_seasonal.py` (npf pushed 2026-08-06, no deprecation) sets +`RETURN_TIER = "propadj"` and records why, amended 2026-07-26 after its first run came +back void: + +> Norgate's `backadj` is ADDITIVE: roll gaps accumulate into the level, so the series is +> not a price and can cross zero. On the verdict window ZB's back-adjusted close runs +> −12.24 to 48.36 and is negative on 454 days, while the contract actually traded 72.66 to +> 112.19. A percent return needs a positive denominator, and **15 of ZB's 100 +> verdict-window trades had their SIGN INVERTED.** + +And, independently of `propadj`, the same file needs both stored frames anyway: + +> Roll DETECTION still reads backadj and unadj, because their difference IS the +> accumulated adjustment and is precisely what steps at a roll. + +Fifteen sign-inverted trades in a live book is a sharper argument than a `raise` inside a +weekly job in a package that no longer runs. **Anchor §4 on `npf`.** + +### What this does to §7.3 + +The work order says: *"Repoint `crowdmon`'s ten call sites. It is the smallest consumer +and the one whose tier requirements are strictest, so it fails loudest. **Do it first, not +last.**"* + +That instruction is now void. Repointing a frozen package with zero consumers is pure +waste, and `DEPRECATED.md` §2 asks for the opposite — it wants the live pins *neutralised* +because "a frozen repo should not have tests that depend on data collected after it was +frozen". Repointing would add a dependency on data collected after the freeze. + +**Do not repoint `crowdmon`.** The ten call sites leave §7.3 entirely, and with them the +step that was supposed to de-risk everything after it. What remains: + +| consumer | needs | status | +|---|---|---| +| `crowdmon` | both tiers + `propadj` | **dropped — deprecated, no consumers** | +| `npf` | both tiers + `propadj` (live book) | ADR-0007 **defers** this, with `livebook` | +| `cotmetrics` / `cot-analyzer` | `backadj` only | the only repoint actually left | + +So the "hard one first" ordering has evaporated, and what is left of §7.3–§7.4 is the +`backadj`-only repoint the work order calls "therefore easy". That is a genuine +simplification and also a genuine loss: the strictest consumer was the one that would have +proved the provider correct by failing loudly, and nothing else exercises `propadj` until +the deferred `npf` pass runs. **The Windows-box comparison in §7 is now the main evidence +that this port preserved the numbers**, which is an argument for running it sooner. + +Whether `npf`'s deferral still makes sense given it is now the *only* `propadj` consumer +is ADR-0007's call, not this handoff's. Flagging it because the deferral was decided when +it was one of two. + ## 6. Scope left out, deliberately -- **§7.3 repoint `crowdmon`** (6 modules, 10 call sites) and **§7.4 `cotmetrics` / - `cot-analyzer`**. Not started. The work order says do `crowdmon` first because its tier - requirements are strictest and it fails loudest, and that ordering still holds. +- **§7.3 repoint `crowdmon`** — **do not do this**, see §5a. **§7.4 `cotmetrics` / + `cot-analyzer`** is not started and is now the only repoint left standing. - One repoint task the work order's call-site table does not capture, found while - checking the `MME`/`MFS` question. `ContractMaster.load()` does not only *call* - `cotdata`; it parses the shape of `cotdata`'s manifest: + The rest of this bullet is retained because it documents a coupling the work order's + call-site table does not capture, and the same pattern may appear elsewhere. It was found + while checking the `MME`/`MFS` question, before §5a established that `crowdmon` should not + be repointed at all. `ContractMaster.load()` does not only *call* `cotdata`; it parses + the shape of `cotdata`'s manifest: ```python for name in load_manifest().get("prices", {}): From 92d5d28bfea2e882141b5eae21d6f3d3e581dd91 Mon Sep 17 00:00:00 2001 From: "Matt S." Date: Sun, 9 Aug 2026 01:20:40 +0000 Subject: [PATCH 4/4] =?UTF-8?q?Close=20out=20the=202026-08-04=20work=20ord?= =?UTF-8?q?er:=20=C2=A74=20re-anchored,=20=C2=A77.3=20void?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The register convention is to preserve a work order verbatim and append an outcome, because amending the body destroys the record of what was actually asked — which is the only thing that makes "the data contradicted the brief" a checkable claim. So §4 and §7.3 keep their prose and gain pointers, and the corrections live in an appended §8. That is also what the convention asks for directly: where a handoff was wrong, the outcome says so and the body carries the pointer, so a reader starting from the work order is not left with the premises it opened with. §8.1 — §4's constraint stands and the provider enforces it three ways. What fails is its argument: it reasons entirely from crowdmon, deprecated 2026-08-07 and consumer-less by 2026-08-08, so its closing warning describes a job that no longer runs. Re-anchored on npf's treasury seasonal book, which sets propadj after a void run in which 15 of ZB's 100 verdict-window trades had their sign inverted, and which reads both stored frames for roll detection regardless. §4's which-box table is untouched: it turns on Norgate supplying all the tiers, not on who consumes them. §8.2 — §7.3 is void. Repointing a frozen package with no consumers is waste, and DEPRECATED.md §2 asks for its live pins to be neutralised rather than given a new post-freeze dependency. Records what that costs: crowdmon was to go first precisely because it fails loudest, so dropping it drops the check that would have proved the provider before the easy consumers leaned on it. §8.3 and §8.4 carry the manifest-format coupling and the MME/MFS answer. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2 --- ...6-08-04-adr7-step2-price-producer-split.md | 107 ++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md b/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md index de31407..2959ed1 100644 --- a/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md +++ b/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md @@ -99,6 +99,12 @@ fixed by `propadj`, so this consumer has already been bitten by a tier choice. ## 4. The finding this handoff exists to record +> **SUPERSEDED IN ITS CONSUMER, NOT IN ITS CONCLUSION — see §8.1.** The constraint below +> is real and the producer enforces it. But `crowdmon` was deprecated on 2026-08-07, three +> days after this was written, and by 2026-08-08 had no consumers at all, so it can no +> longer carry the argument. §8.1 re-anchors it on `npf`, which makes the same case from a +> live book. **Read §8.1 before quoting this section.** + Stated nowhere before today, and it is the constraint that decides which machine can produce a store the newest consumer can use. @@ -192,7 +198,108 @@ into ADR-0007's open-questions section as resolved rather than leaving it readin 3. Repoint `crowdmon`'s ten call sites (§3). It is the smallest consumer and the one whose tier requirements are strictest, so it fails loudest if the provider is wrong. **Do it first, not last.** + > **VOID — see §8.2.** `crowdmon` is deprecated and has no consumers. Do not repoint it. + > This step, and the de-risking it was meant to provide, are gone. 4. Repoint `cotmetrics` / `cot-analyzer`, which are `backadj`-only and therefore easy. 5. Delete from `cotdata`, and update `crucible-stack` ADR-0007's "Status of the work". Step 3 stays out. `npf` and `livebook` are a separate pass with a live book behind them. + +--- + +## 8. Outcome, appended 2026-08-08 + +**Step 1 of §7 executed** (`marketdata` PR #7), and §7.2 came with it. §7.3–§7.5 are not +started. Full record: +[`2026-08-08-adr7-step2-provider-shipped.md`](2026-08-08-adr7-step2-provider-shipped.md). + +Body preserved verbatim above, per the register convention. This section carries the +corrections; §4 and §7.3 carry pointers to it. + +### 8.1 §4's constraint stands. Its consumer does not + +**The requirement is unchanged and is enforced three ways** in the shipped provider: both +tiers are fetched before either is written, a read finding one raises and names the +missing one, and a test covers both halves. Nothing here relaxes it. + +What changed is that §4 argued it entirely from `crowdmon`, and `crowdmon` is gone. +`crowdmon/DEPRECATED.md` decides deprecation on **2026-08-07** — one day after this +handoff's own companion document, `crowdmon/docs/design/amendments-2026-08-04.md` §D14 — +after four pre-registered tests returned no positive result and the §10 validation came +back uninformative with its clean episodes spent. Its §3 was resolved **2026-08-08**: both +launchd jobs unloaded, `cot-analyzer`'s `/damage` page removed (that repo's PR #22), the +one open work order closed unstarted, and the package left with **no consumers at all** — +"nothing in `npf` or `livebook` ever imported it". + +So §4's closing warning — that a `backadj`-only producer breaks `crowdmon` with a `raise` +inside `riskunits` on a weekly scheduled job — describes a job that no longer runs. + +**Re-anchor on `npf`, which makes the case from a live book.** +`npf/books/treasury_seasonal.py` sets `RETURN_TIER = "propadj"`, amended 2026-07-26 after +its first run came back void: + +> Norgate's `backadj` is ADDITIVE: roll gaps accumulate into the level, so the series is +> not a price and can cross zero. On the verdict window ZB's back-adjusted close runs +> −12.24 to 48.36 and is negative on 454 days, while the contract actually traded 72.66 to +> 112.19. A percent return needs a positive denominator, and **15 of ZB's 100 +> verdict-window trades had their SIGN INVERTED.** + +The same file needs both stored frames independently of `propadj`: *"Roll DETECTION still +reads backadj and unadj, because their difference IS the accumulated adjustment and is +precisely what steps at a roll."* + +Fifteen sign-inverted trades in a live book is a stronger argument than the one §4 made, +and it survives the deprecation. §4's table of which box can produce a consumable store is +unaffected — it turns on Norgate supplying all the tiers, not on who consumes them. + +### 8.2 §7.3 is void, and §7's ordering with it + +Repointing a frozen package with zero consumers is waste. `crowdmon/DEPRECATED.md` §2 +asks for the opposite: it wants that package's live pins **neutralised**, because "a frozen +repo should not have tests that depend on data collected after it was frozen". Repointing +would add a dependency on data collected after the freeze. + +The consumer list in §3 now reads: + +| consumer | needs | status | +|---|---|---| +| `crowdmon` (§3, 6 modules / 10 call sites) | both tiers + `propadj` | **dropped** | +| `npf` | both tiers + `propadj` (live book) | ADR-0007 **defers**, with `livebook` | +| `cotmetrics` / `cot-analyzer` | `backadj` only | the only repoint left | + +**What is lost with it.** §7.3's instruction was not arbitrary: `crowdmon` was to go first +*because* it fails loudest, so it would prove the provider before the easy consumers +depended on it. Removing it removes that check. Nothing now exercises `propadj` until the +deferred `npf` pass runs, so **§7's "compare against `cotdata`'s existing `ES_backadj`" on +the Windows box is the main remaining evidence that the port preserved the numbers.** Run +it early rather than late. + +**One question this handoff does not answer.** ADR-0007 defers `npf` on the grounds that it +is a live book, decided when it was one of two `propadj` consumers. It is now the only one. +Whether the deferral still holds is ADR-0007's call. + +### 8.3 A finding §3's call-site table does not capture + +`crowdmon`'s `ContractMaster.load()` couples to the *shape* of the manifest, not only to +`cotdata`'s functions: + +```python +for name in load_manifest().get("prices", {}): + sym, _, adj = str(name).rpartition("_") # "ES_backadj" -> ("ES", "backadj") +``` + +`marketdata`'s entries live under `"bars"` and read `futures/norgate/ES_backadj`, so that +parse yields `futures/norgate/ES` and matches no registry symbol — every symbol would go +non-joinable. It fails loudly rather than silently. Recorded even though §8.2 retires the +`crowdmon` repoint, because the coupling is a manifest *format* rather than a call, and §3's +method — counting call sites — would not have found it in any consumer. + +### 8.4 The §6 question about `MME`/`MFS`, answered + +They are **not ported** to `marketdata` (Norgate carries no continuous series for either; +`cotdata` prices them off the EEM and EFA ETF proxies). This costs `crowdmon` nothing even +had it been repointed: both already fail `coverage()` for the same underlying reason, +reporting `missing: specs,unadj_price,backadj_price`, pinned by +`tests/test_contract_master_live.py` as the only two non-joinable, absent from both vintage +panels per the 2026-08-04 spec inventory, and `Role: heldout` in the deployed `params.yaml`. +The counts agree: 49 of 51 joinable, and 49 futures symbols ported.