Skip to content

ADR-0007: record step 2 as implemented, and the one part that was not #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mspinola merged 1 commit into from
Aug 9, 2026
Merged

ADR-0007: record step 2 as implemented, and the one part that was not #22

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
210 changes: 157 additions & 53 deletions docs/adr/ADR-0007-cotdata-is-cot-only-bars-live-in-marketdata.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@
> a new `marketdata` sibling, and four consumers. Unlike ADR-0006 it **does** change a consumer
> read contract, which is most of its cost.

**Status:** Accepted (2026-07-27). The direction is settled: `cotdata` becomes COT-only and all
bar data lives in `marketdata`. Accepted with **step 1 shipped and steps 2 to 4 outstanding**,
which is a deliberate distinction from ADR-0006. That one was accepted after its architecture was
implemented and validated end to end. This one is accepted as a direction, on the strength of step
1 landing cleanly and of `marketdata` proving the receiving design in production. The disruptive
move has not happened, so acceptance here authorises the remaining steps rather than recording
them.
**Status:** Accepted (2026-07-27). **Implemented 2026-08-09, with one part of the decision
unimplemented: the databento provider did not move.** Steps 1 to 4 are otherwise done and every
consumer reads bars from `marketdata`. The original acceptance was as a *direction*, on the
strength of step 1 landing cleanly, with the disruptive move still ahead — a deliberate contrast
with ADR-0006, which was accepted after end-to-end validation. That move has now happened. See
[Status of the work](#status-of-the-work) for what shipped and for the one deviation, which is
outstanding work rather than a boundary this ADR drew.
**Date:** 2026-07-25 (accepted 2026-07-27)
**Deciders:** Matt (sole maintainer)

Expand Down Expand Up @@ -206,58 +206,162 @@ Extract in the ADR-0004 style, so the disruptive step is decoupled from the desi

1. **DONE (2026-07-26).** Make the seam explicit **inside** `cotdata`: separate store domains,
separate manifests, separate CLI entry points.
2. Move the price half into `marketdata` as the `futures` domain, beside the `equities` domain
already built, taking the contract-specs table and its `--metadata` flag with it. With step 1
done this is a file move plus a shim.
3. Migrate consumers one repo at a time behind the shim, `livebook` last. Before touching
`costs.py`, give it a positive assertion that the specs table loaded, so the migration cannot
pass silently on zero costs. **The npf half of that guard is built** (npf#159).
4. Remove the shim after a deprecation window. Optionally converge both packages on one store root.
2. **DONE (2026-08-09), except databento.** Move the price half into `marketdata` as the
`futures` domain, beside the `equities` domain already built, taking the contract-specs table
and its `--metadata` flag with it. With step 1 done this is a file move plus a shim.
3. **DONE (2026-08-09).** Migrate consumers one repo at a time behind the shim, `livebook` last.
Before touching `costs.py`, give it a positive assertion that the specs table loaded, so the
migration cannot pass silently on zero costs. **The npf half of that guard is built** (npf#159).
4. **VOID** — no shim was built. Remove the shim after a deprecation window. Optionally converge
both packages on one store root.

Step 1 delivers most of the clarity at a fraction of the risk and is worth doing on its own. That
prediction held: it landed without touching a single consumer read.

Two of step 2's predictions did not. It was **not** "a file move": `cot.py` importing only
`store` made the *seam* clean, but the Norgate provider had to be rewritten against the
tier-aware store rather than copied, since one frame per symbol cannot hold `backadj` and `unadj`
at once. And no shim was built or needed — see Status of the work.

## Status of the work

Updated 2026-07-27.

**Step 1 is complete.** `cotdata` PRs #55 (the seam), #56 (wrappers call the half-scoped entry
points), #59 (stop writing the legacy manifest, plus a one-shot migration) and #61 (`--check`
measures lag on write time). The live store now carries `manifests/cot.json` and
`manifests/prices.json` with the legacy `manifest.json` deleted, the domains are split
(`prices/` against `cot_legacy/`, `cot_disagg/`, `cot_tff/`, `metadata/`), and `cotdata-cot` and
`cotdata-prices` ship beside `cotdata-update`.

**Step 3's prerequisite is half built.** `npf.validation.costs.assert_specs_available` raises
rather than charging zero when the specs table cannot price what is being traded, and its loader is
source-agnostic so it names which package served the table and flags any other that also could:
mid-migration both can, and silently taking the first is how a stale table goes unnoticed.
`livebook` is NOT covered. It calls `contract_specs()` in six places at the default
`required=False`, so it would degrade to an empty frame and keep running. That is a live book and
the change is deliberately deferred.

**Step 2 has not started.** `prices.py` still lives in `cotdata`. What moves is roughly 1,700
lines: `providers/databento.py` (1015), `providers/norgate.py` (469), `prices.py` (167),
`providers/yfinance.py` (70), plus `metadata/contract_specs.parquet` and `--metadata`.
`marketdata` exists with the `equities` domain, the yfinance provider, the three-tier adjustment
derived on read, a pin test reproducing the vendor's own adjusted column, and a manifest-level
`universe_is_point_in_time` flag. The `futures` domain is declared in `adjust.DOMAIN_TIERS` so
error messages are correct from the first day, but has no provider yet.

`databento.py` arriving after this ADR was written makes `cotdata` *more* price-heavy than the
measurements above, which strengthens the case and enlarges step 2 at the same time.
Updated 2026-08-09.

**Steps 1 to 4 are complete, with one exception recorded below.** `cotdata` 0.4.0 ships with no
bar API, no Norgate and no yfinance; every consumer reads `marketdata.get_bars`.

| Step | State | Where |
|---|---|---|
| 1. Make the seam explicit inside `cotdata` | done 2026-07-26 | `cotdata` #55, #56, #59, #61 |
| 2. Move the price half into `marketdata` | done 2026-08-09, **minus databento** | `marketdata` #7, #8, #10; `cotdata` #103, #104, #105 |
| 3. Migrate consumers | done 2026-08-09 | `cotmetrics` #11, `cot-analyzer` #26, `npf` #196 and #197, `livebook` #10 |
| 4. Remove the shim | **void** — no shim was built, see below | — |

Contract specs moved with the producer as the 2026-07-26 amendment requires: `marketdata` owns
`metadata/contract_specs.parquet` and the `--metadata` flag, and `cotdata`'s
`store.{write,upsert,read}_metadata` are gone. `marketdata` is published as **`crucible-marketdata`**
(the import stays `marketdata`; the obvious distribution name is taken on PyPI by an abandoned
2020 project), which forced a dependency rename in `cotmetrics` #12 and `cot-analyzer` #28.

### The one deviation: databento did not move

**This ADR says it should have.** Two passages, both explicit:

> Norgate futures and databento are not two things. They are two providers of the same thing …
> Separating them would break the abstraction they were built to share.

and step 2's own inventory names `providers/databento.py` (1015 lines) as the **largest single
file** that moves. So databento remaining in `cotdata` is a deviation from the decision, not an
exception the decision carved out, and it should not be read as one.

**Why it stayed.** Not for an architectural reason. It was neither ported nor deleted:

- **Porting it is its own piece of work.** It is not a file move. It is a two-stage paid-API
producer with an append-only raw bronze store, its own ingest manifest, a two-directional
reconcile path, a windowed statistics fetch and a build-stage unit scale — around 1,100 lines
with no counterpart in `marketdata`, whose futures provider is Norgate-shaped.
- **Deleting it would have destroyed working, validated code with no replacement.** ADR-0006 is
Accepted on the strength of databento's end-to-end validation against Norgate, and it is the
fleet's only intraday-capable source. A deletion in service of a boundary would have thrown
that away to make a package description true.

So it was left in place as the lesser of two bad options. **It is outstanding work.**

**What it costs, concretely.** `cotdata`'s *public API* is COT-only; its *store* is not. Keeping
databento means keeping `store.write_prices` / `read_prices`, `config.prices_dir()` and the
`prices` half of the manifest, because databento writes through all of them. The line this ADR
drew — "`cotdata` keeps CFTC positioning and nothing else" — is now true of what a consumer can
import and false of what the store holds. Two packages can still write futures bars, which is
the coupling the contract-specs amendment argued against under "one vendor integration, one
home". And it sharpens the third open question below rather than leaving it where it was: a
databento-sourced dashboard would read bars from `$COTDATA_STORE/prices/` while everything else
in the fleet reads `$MARKETDATA_STORE/bars/futures/`.

**What it does not cost.** Nothing is broken today. databento is not the source for any deployed
consumer — the dash server is on synced Norgate per ADR-0006's Outcome — so the split store is a
latent inconsistency rather than a live one.

### No shim was built, and none was needed

The Consequences section planned "a re-export shim and a deprecation window rather than a clean
cut", and step 4 was to remove it. **The cut was clean.** Ordering made the shim unnecessary and
would have made it harmful:

- By the time `cotdata` #105 deleted `get_prices`, every consumer had already been repointed and
verified, so a shim would have protected nobody.
- A shim forwarding to `marketdata` would have been fine; a shim left reading `cotdata`'s own
store would not. That store is no longer filled by the nightly job, so the failure mode is a
number that looks right and is months old, which is strictly worse than an `AttributeError`.
`cotdata` therefore asserts in a test that `get_prices` and `roll_dates` stay gone.

Step 4's shim half is void. Its optional second half — converging the two packages on one store
root — is untouched and stays in Open questions.

### Two measurements from the ADR that came out differently

**The move was smaller than predicted, because databento stayed.** Step 2 estimated roughly 1,700
lines moving. About 700 did: `providers/norgate.py` (469, arriving as 546), `prices.py` (167) and
`providers/yfinance.py` (70). databento's ~1,100 did not, so `cotdata` remains roughly 18% price
code by line — the mismatch this ADR was written to remove is reduced, not eliminated.

**`propadj` got stronger than "derived on read".** The second open question is resolved and then
some: `marketdata` derives it from the two stored tiers and **refuses** when only one is present,
rather than returning an empty frame. The producer writes both tiers or neither. That closes the
failure the Treasury seasonal hit, where a wrong tier passed a spot check — additive
back-adjusted percent volatility is ~200x too high for soybeans and **0.47x** for gold, and 0.47x
never goes negative.

### A process finding worth carrying to the next extract

Step 2 ported the Norgate provider **without its tests**, and the gap was invisible for two weeks
because the test *files* existed on both sides. Diffing the test *names* across the 30 that
`cotdata` was about to delete gave **zero overlap** — which reads like a renaming and was, under
it, seven behaviours that would have had no test anywhere the moment the deletion landed
(volume reconstruction, the volume-rank contract pick, the incremental window, the full rebuild,
the NDU-down abort, the all-null spec-row skip). Found only because §7.5 went to delete them, and
fixed in `marketdata` #13.

The same shape appeared twice more in the same step. `finals_ready()` was ported in step 2 and
wired to no CLI flag, so `cotdata`'s `--require-final` was its only caller and deleting that would
have silently ungated the nightly capture (`marketdata` #12). And two databento parity harnesses
read the Norgate store **by path**, so no call-site grep found them.

The lesson generalises past this ADR: **for an extract in the ADR-0004 style, a call-site census
is not a coverage census.** Count what the tests assert and what reaches each entry point, not
what files exist.

## Open questions

- Whether the two packages converge on a single store root env var, and when. Both are now set
from the shell profile (`COTDATA_STORE`, `MARKETDATA_STORE`), and neither is set by any launcher,
so converging is still cheap.
- Whether the futures `propadj` tier stays derived-on-read as it is today. **Evidence arrived
2026-07-27.** The month-end Treasury seasonal read futures through `cotdata.get_prices` and its
first run was VOIDED because `backadj` percent returns sign-inverted 15 of ZB's 100 trades:
additive back-adjustment can cross zero. Moving to `propadj` fixed it, so `propadj` is not a
nicety, it decided a verdict. `marketdata` derives every tier on read, which is the behaviour
that turned out to matter.
- Whether the Linux server runs the `marketdata` futures producer itself after step 2, or keeps
receiving a synced store. databento is not currently declared in `cot-analyzer`'s own
requirements, so that wiring is unfinished either way.
Updated 2026-08-09. Two of the three below are resolved; the rest is what remains.

- **Port databento into `marketdata`, or accept the split permanently.** The live one, and the
only unimplemented part of the Decision (see Status of the work). Until it is settled,
`cotdata` holds a price store it has no consumer API for. Accepting the split permanently is a
legitimate answer, but it needs writing into the Decision as an amendment rather than being
left to read as unfinished work.
- **Whether the two packages converge on a single store root env var, and when.** Unchanged and
still cheap, but no longer only cosmetic: a launcher now has two roots to get right, and one
has already been found half-wired (below).
- ~~Whether the futures `propadj` tier stays derived-on-read.~~ **Resolved.** It stays derived,
and `marketdata` hardened it: deriving from one stored tier raises rather than returning empty,
and the producer writes both tiers or neither. The evidence that decided it is unchanged — the
month-end Treasury seasonal's first run was VOIDED because `backadj` percent returns
sign-inverted 15 of ZB's 100 trades.
- **Whether the Linux server runs the `marketdata` futures producer itself, or keeps receiving a
synced store.** Still open, and the databento deviation sharpens it: a databento-sourced server
now reads bars from a *different package's store* than every other consumer. The dash is on
synced Norgate per ADR-0006's Outcome, so this is latent rather than live.

Found while implementing, and not previously tracked here:

- **`livebook/bin/daily.sh` guards `COTDATA_STORE` and not `MARKETDATA_STORE`.** The live job's
bars and contract specs both come from `marketdata` now, so the guard checks the store that
matters less. It fails closed rather than silently — `marketdata.config.store_root()` raises by
name on an unset root, and `livebook.specs.broker_specs` asks at `required=True` — so the
consequence is a traceback mid-run instead of a clear refusal at the top. Worth making
symmetric.
- **The Status section's note that "`livebook` is NOT covered" for contract specs is out of
date.** `livebook.specs.broker_specs` now calls `contract_specs(required=True)` and asserts the
`Exchange` column resolves before the broker connects, which is the guard step 3 asked for.
`npf.validation.costs` resolves the table unambiguously from `marketdata` now that `cotdata`'s
copy is gone, and reports which package answered.
Loading