mspinola · mspinola · Aug 9, 2026 · Aug 9, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,52 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+### Removed — BREAKING
+
+- **The databento provider is gone, and with it every remaining trace of price data.**
+  ADR-0007 is now fully implemented: `cotdata` is CFTC positioning and nothing else.
+  databento moved to [`crucible-marketdata`](https://pypi.org/project/crucible-marketdata/)
+  alongside Norgate and Yahoo, where it writes `bars/futures/databento/` in that
+  package's store.
+
+  | gone from cotdata | use instead |
+  |---|---|
+  | `cotdata-update --ingest-databento` / `--build-databento` / `--reconcile-databento` | the same flags on `marketdata-update` |
+  | `cotdata-prices` | *(nothing — there is no price half)* |
+  | `cotdata[databento]` extra | `crucible-marketdata[databento]` |
+  | `$COTDATA_DATABENTO_RAW` | `$MARKETDATA_DATABENTO_RAW` |
+  | `store.write_prices` / `read_prices`, `config.prices_dir()` | `marketdata.store.write_bars` / `read_bars` |
+  | `registry.resolve_source`, `default_price_source`, `PRICE_SOURCES` | marketdata's registry |
+  | `Symbol.norgate` / `.yahoo` / `.databento` / `.price_source` | marketdata's registry |
+  | `$COTDATA_PRICE_SOURCE` | `$MARKETDATA_PRICE_SOURCE` |
+  | `scripts/validate_databento_vs_norgate.py`, `scripts/investigate_databento_roll_rule.py` | same paths in marketdata |
+
+  **This package now has no optional data dependency at all.** CFTC positioning is a
+  plain HTTP download of public files, so the `databento` extra was the last one and it
+  left with its provider.
+
+  **The registry lost its vendor columns.** `Symbol` is down to `internal`,
+  `asset_class`, `is_equity`, `report_type`, `cftc_code` and `hist_codes`. An earlier
+  note claimed the vendor mappings had to stay because a deployment might share one
+  registry file between the two packages via `$COTDATA_REGISTRY` — that was wrong, and
+  is corrected here: this loader hard-requires `cftc_code` and marketdata's equities do
+  not have one, so the two files can never be the same file.
+
+  **The producer-half machinery is gone**, because there is one producer. `cotdata-cot`
+  survives as an alias of `cotdata-update` — the scheduled jobs call it by name — but it
+  no longer scopes anything, and `_HALF_ACTIONS` / `_reject_other_half` are removed.
+  The manifest is still split (`manifests/cot.json`, `manifests/prices.json`) and the
+  `prices` and `metadata` domains are still DECLARED: a store built before the move
+  carries entries there, and an undeclared domain is skipped by `--migrate-manifests`,
+  which would strand them in the legacy aggregate forever. Read-only history, no writer.
+
+  **What did NOT come across:** databento's dormant per-symbol EOD path
+  (`fetch_daily_ohlc`, `run_batch_backfill`, `update_all_daily_prices`). It had no
+  caller anywhere in the fleet, it duplicated the two-stage producer, and the intraday
+  work it was nominally kept for would need an intraday schema rather than the
+  `ohlcv-1d` it actually fetched. It remains in this repo's git history.
+
+
 ### Removed — BREAKING
 
 - **Price bars, and the Norgate and Yahoo producers, are gone from this package**

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -79,10 +79,9 @@ There is none here any more. ADR-0007 moved the Norgate integration to
 [`crucible-marketdata`](https://github.com/mspinola/marketdata), so Norgate changes and
 their Windows-only constraints belong in that repo's CONTRIBUTING.
 
-Everything in this repo runs cross-platform with no vendor SDK: the CFTC parsers hit
-cftc.gov, and the databento provider's tests drive a synthetic raw store rather than the
-API. If a test needs a network or a paid key to pass, it does not belong in `tests/` —
-put it in `scripts/` and say so in its docstring, as the databento parity harnesses do.
+Everything in this repo runs cross-platform with no vendor SDK and no optional extra:
+the only source is cftc.gov over plain HTTP. If a test needs a network or a paid key to
+pass, it does not belong in `tests/`.
 
 ## Code Style
 

diff --git a/README.md b/README.md
diff --git a/docs/LINUX_SCHEDULING.md b/docs/LINUX_SCHEDULING.md
@@ -1,41 +1,30 @@
 # Scheduling cotdata on Linux (cron)
 
-For the cross-platform Databento producer path (no Norgate/Windows required) — see [Cross-platform prices without Norgate](../README.md#cross-platform-prices-without-norgate-databento) in the README for the one-time `--ingest-databento` / `--build-databento` setup before automating it here.
+> **Only COT is scheduled here now.** ADR-0007 moved every price producer to
+> [`marketdata`](https://pypi.org/project/crucible-marketdata/), databento included, so
+> the nightly price job on this box is `marketdata-update --ingest-databento` /
+> `--build-databento` against `$MARKETDATA_STORE`. See that package's README. The COT
+> half below is unchanged.
+>
+> **Upgrading?** Delete the old `run-prices.sh` — it calls `cotdata-prices`, which no
+> longer exists, so the job will fail every night until it is replaced or removed.
 
 ## Goal
 
-A databento server schedules **prices nightly** and **COT soon after its Friday ~3:30pm ET release**, with a daily catch-up for holiday delays. Two properties hold:
+**COT soon after its Friday ~3:30pm ET release**, with a daily catch-up for holiday delays. Two properties hold:
 
-- **Idempotent.** `--cot-all` HEAD-checks each CFTC year zip and skips it if unchanged. `--ingest-databento` resumes from the last fetched date, so a re-run pulls only new days. Running before new data lands is a harmless no-op.
-- **Fails loudly.** A run exits non-zero only on a hard fetch error (source unreachable), not when there is simply nothing new. Because ingest is resumable and COT is idempotent, a failed or missed run is picked up by the next one, so no explicit retry logic is needed.
+- **Idempotent.** `--cot-all` HEAD-checks each CFTC year zip and skips it if unchanged. Running before new data lands is a harmless no-op.
+- **Fails loudly.** A run exits non-zero only on a hard fetch error (source unreachable), not when there is simply nothing new. Because COT is idempotent, a failed or missed run is picked up by the next one, so no explicit retry logic is needed.
 
 ## Wrapper scripts
 
-Cron runs with a bare environment, so put the config and the venv path in a wrapper script (one per command, mirroring the Windows pair).
+Cron runs with a bare environment, so put the config and the venv path in a wrapper script.
 
-> **Ready-made templates:** copy [`docs/examples/linux/run-prices.sh`](examples/linux/run-prices.sh) and [`run-cot.sh`](examples/linux/run-cot.sh) out of the repo into your `<DIR>`, `chmod +x` them, and fill in the placeholders — keep them outside the repo so a `git pull` never clobbers your edited paths.
+> **Ready-made template:** copy [`run-cot.sh`](examples/linux/run-cot.sh) out of the repo into your `<DIR>`, `chmod +x` it, and fill in the placeholders — keep it outside the repo so a `git pull` never clobbers your edited paths.
 
-Inside the scripts, overwrite the plain-text markers: `REPLACE_WITH_STORE_PATH` = your store, `REPLACE_WITH_VENV_PATH` = your virtualenv, `REPLACE_WITH_DATABENTO_KEY` = your Databento key. (They're plain markers, not `<...>` placeholders, because an unedited `<...>` would be read as a shell redirection and the script would fail.) The `<DIR>` in the crontab lines below is normal fill-in notation.
+Inside it, overwrite the plain-text markers: `REPLACE_WITH_STORE_PATH` = your store, `REPLACE_WITH_VENV_PATH` = your virtualenv. (They're plain markers, not `<...>` placeholders, because an unedited `<...>` would be read as a shell redirection and the script would fail.) The `<DIR>` in the crontab lines below is normal fill-in notation.
 
-`run-prices.sh` — the two-stage databento build:
-
-```bash
-#!/usr/bin/env bash
-set -euo pipefail
-export COTDATA_STORE=REPLACE_WITH_STORE_PATH
-export DATABENTO_API_KEY=REPLACE_WITH_DATABENTO_KEY
-BIN=REPLACE_WITH_VENV_PATH/bin/cotdata-prices
-"$BIN" --ingest-databento     # Stage 1 (paid): raw .n.0/.n.1 to raw store
-"$BIN" --build-databento      # Stage 2 (free): back-adjusted prices
-```
-
-Databento is the only price producer left in this package. ADR-0007 moved the Norgate and
-Yahoo bar producers to [`marketdata`](https://pypi.org/project/crucible-marketdata/), so the
-markets databento does not cover — ICE softs, lumber, the MSCI ETF proxies — are fetched by
-`marketdata-update --bars` against `$MARKETDATA_STORE` on whichever box produces that store,
-not by this script.
-
-`run-cot.sh` — COT (note the different command):
+`run-cot.sh`:
 
 ```bash
 #!/usr/bin/env bash
@@ -44,21 +33,17 @@ export COTDATA_STORE=REPLACE_WITH_STORE_PATH
 REPLACE_WITH_VENV_PATH/bin/cotdata-cot --cot-all
 ```
 
-Make them executable:
+Make it executable:
 
 ```bash
-chmod +x run-prices.sh run-cot.sh
+chmod +x run-cot.sh
 ```
 
 ## Crontab entries
 
 Add the jobs with `crontab -e`. Cron uses the **server's local** timezone, so convert the ET times below if it is not on Eastern (or set the server to a known zone). `flock` stops a slow run from overlapping the next, and the redirect keeps a log:
 
 ```cron
-# Prices — nightly (Mon-Sat). GLBX settlements are disseminated the morning after the
-# session, so an early-morning run captures the prior session's finalized settlement.
-30 6 * * 1-6     flock -n /tmp/cotdata-prices.lock <DIR>/run-prices.sh >> <DIR>/prices.log 2>&1
-
 # COT — daily morning catch-up (holiday-delayed releases and a safety net).
 10 8 * * *       flock -n /tmp/cotdata-cot.lock <DIR>/run-cot.sh >> <DIR>/cot.log 2>&1
 
@@ -77,19 +62,19 @@ Set `MAILTO=you@example.com` at the top of the crontab to have cron email the ou
 
 ### Cron job runs manually but not on schedule
 
-Cron's environment is far barer than an interactive shell — no `PATH` beyond `/usr/bin:/bin`, no `.bashrc`/`.profile` sourced, no venv activation. This is exactly why the wrapper scripts above call the venv's binary by full path (`<VENV>/bin/cotdata-prices`, `<VENV>/bin/cotdata-cot`) rather than a bare command name, and `export` every variable instead of relying on a login shell to have set them. If a script works when you run it by hand but not under cron, the first thing to check is whether it depends on something your interactive shell set up implicitly.
+Cron's environment is far barer than an interactive shell — no `PATH` beyond `/usr/bin:/bin`, no `.bashrc`/`.profile` sourced, no venv activation. This is exactly why the wrapper script above calls the venv's binary by full path (`<VENV>/bin/cotdata-cot`) rather than a bare command name, and `export` every variable instead of relying on a login shell to have set them. If a script works when you run it by hand but not under cron, the first thing to check is whether it depends on something your interactive shell set up implicitly.
 
 ### Job silently does nothing
 
-Check `<DIR>/prices.log` or `<DIR>/cot.log` first — the wrappers redirect both stdout and stderr there. If the log is empty or missing entirely, cron likely never ran the job: check `grep CRON /var/log/syslog` (Debian/Ubuntu) or `journalctl -u cron` (systemd) for the scheduled time to confirm cron invoked it at all.
+Check `<DIR>/cot.log` first — the wrapper redirects both stdout and stderr there. If the log is empty or missing entirely, cron likely never ran the job: check `grep CRON /var/log/syslog` (Debian/Ubuntu) or `journalctl -u cron` (systemd) for the scheduled time to confirm cron invoked it at all.
 
 ### Overlapping runs / stale lock
 
-`flock -n` fails fast (doesn't block) if another instance already holds the lock file, so a slow `run-prices.sh` won't stack with the next scheduled run — the second invocation just no-ops and exits. The lock releases automatically when the holding process exits, including on a crash, so a stale lock that blocks forever generally indicates a *hung*, still-running process, not manual cleanup — check with `ps aux | grep cotdata` before deleting anything under `/tmp`.
+`flock -n` fails fast (doesn't block) if another instance already holds the lock file, so a slow `run-cot.sh` won't stack with the next scheduled run — the second invocation just no-ops and exits. The lock releases automatically when the holding process exits, including on a crash, so a stale lock that blocks forever generally indicates a *hung*, still-running process, not manual cleanup — check with `ps aux | grep cotdata` before deleting anything under `/tmp`.
 
 ### Permission denied running the wrapper
 
-Confirm `chmod +x` was applied to both `.sh` files, and that the shebang (`#!/usr/bin/env bash`) resolves — run `which bash` to confirm it's on `PATH` for the cron user (usually is, but matters more on minimal containers).
+Confirm `chmod +x` was applied to the `.sh` file, and that the shebang (`#!/usr/bin/env bash`) resolves — run `which bash` to confirm it's on `PATH` for the cron user (usually is, but matters more on minimal containers).
 
 ### Timezone confusion on the Friday window
 

diff --git a/docs/SYNCING.md b/docs/SYNCING.md
@@ -7,14 +7,18 @@ This page is about **what** to move and what to leave behind. The transport is t
 part and comes last.
 
 > [!NOTE]
-> **Two stores since ADR-0007.** Daily bars moved to
+> **Two stores since ADR-0007.** Every bar — Norgate, databento and Yahoo alike — moved to
 > [`crucible-marketdata`](https://pypi.org/project/crucible-marketdata/) and its own
 > `$MARKETDATA_STORE`. The Windows box still produces both and still pushes both to the same
 > replicas, so the topology, exclusions, auth gotchas and preflight advice below apply
 > unchanged — there are now **two source directories to mirror instead of one**, and the
-> `prices/`, `metadata/` rows in the exclusion table describe the bar store rather than this
-> one. Where a command names `cotdata-prices --prices`, read
+> `bars/`, `metadata/` and `_raw/` rows describe the bar store rather than this one. Where a
+> command names `cotdata-prices --prices`, read
 > `marketdata-update --bars --domain futures --require-final`.
+>
+> A cotdata store built before the move still has `prices/`, `metadata/` and `_raw/` sitting
+> in it. Nothing writes them any more; leaving them is harmless and deleting them is safe
+> once the bar store is confirmed synced.
 ## This deployment: one Norgate producer, two replicas
 
 A single Windows server is the only producer (Norgate prices, CFTC COT). It feeds two
@@ -72,12 +76,15 @@ maintenance than a per-symbol roll-rule table.
   give it an explicit writable `-o UserKnownHostsFile=`, and use cygdrive (`/cygdrive/c/…`)
   paths throughout, including the key. The example script wires all three.
 
-**Provider cutover (one-time).** The server previously held a databento-built store, so its
-`prices/` and `manifests/prices.json` carry databento data under the very keys the Norgate
-push writes. `sync_preflight.py` will (correctly) refuse: it sees the same
-`prices/<SYM>_<adj>.parquet` produced by a different source on each side, the 94-collision
-case from "Check before you mirror". That refusal is the tool working, not a
-misconfiguration. Resolve it once by clearing the server's `prices/` and `manifests/`
+**Provider cutover (one-time, and now historical).** The server previously held a
+databento-built store, so its `prices/` and `manifests/prices.json` carry databento data
+under the very keys the Norgate push writes. `sync_preflight.py` will (correctly) refuse: it
+sees the same `prices/<SYM>_<adj>.parquet` produced by a different source on each side, the
+94-collision case from "Check before you mirror". That refusal is the tool working, not a
+misconfiguration. **ADR-0007 removed the possibility**: the vendor is a directory in
+marketdata's layout (`bars/futures/norgate/` beside `bars/futures/databento/`), so two
+vendors cannot contend for one path at all. Resolve the legacy case once by clearing the
+server's `prices/` and `manifests/`
 before the first Norgate push, so the store is rebuilt as a clean Norgate replica; every
 push after that is a same-source mirror with nothing to collide.
 
@@ -114,20 +121,19 @@ about two producers writing the *same* files.
 
 This is the part that matters, and on a real store it is most of the bytes.
 
-Per store, since there are now two. `bars/` and `metadata/` live in `$MARKETDATA_STORE`;
-everything else here is `$COTDATA_STORE`. `prices/` appears in the cotdata store only when
-the databento producer is in use.
+Per store, since there are now two. `bars/`, `metadata/` and `_raw/` live in
+`$MARKETDATA_STORE`; everything else here is `$COTDATA_STORE`.
 
 | Directory | Sync? | Why |
 |---|---|---|
 | `bars/` (marketdata) | **yes** | the data |
 | `metadata/` (marketdata) | **yes** | contract specs |
 | `cot_legacy/`, `cot_disagg/`, `cot_tff/` | **yes** | the data |
-| `prices/` | **yes**, if you run databento | the ADR-0006 alternative producer's output |
+| `prices/` | **NO** (legacy) | pre-ADR-0007 leftover; bars live in the marketdata store now |
 | `manifests/` | **yes** | per-half bookkeeping |
 | `status.json` | yes | the producer's own view, useful on the replica |
 | `_cache/` | **NO** | cotdata's own download cache of CFTC source zips, producer-internal, free to rebuild |
-| `_raw/` | **NO** | databento append-only raw store, producer-internal |
+| `_raw/` | **NO** | databento's append-only PAID raw store, producer-internal (marketdata's now) |
 | anything a consumer added by hand | **NO** | no producer creates it, so a mirror deletes it (see below) |
 | `manifest.json` | **NO** | legacy aggregate, nothing writes it (see below) |
 

diff --git a/docs/WINDOWS_SCHEDULING.md b/docs/WINDOWS_SCHEDULING.md
@@ -3,7 +3,7 @@
 New to Python and cotdata on Windows? Start with the [Windows Setup Guide](WINDOWS_SETUP.md) — install Python, create the venv, and confirm `cotdata-update --cot-legacy` works by hand before automating it.
 
 > [!IMPORTANT]
-> **The price task now runs a different package.** ADR-0007 moved Norgate bar production to
+> **The price task runs a different package.** ADR-0007 moved ALL bar production to
 > [`crucible-marketdata`](https://pypi.org/project/crucible-marketdata/), so the nightly job is
 > `marketdata-update --bars --domain futures --require-final` against `$MARKETDATA_STORE`, not
 > `cotdata-prices --prices`. It is still scheduled here, on the same box, at the same time,
@@ -12,8 +12,8 @@ New to Python and cotdata on Windows? Start with the [Windows Setup Guide](WINDO
 > store variable moved.
 >
 > **Upgrading?** Edit `run-prices.cmd` to the new command and give it `MARKETDATA_STORE`. A
-> wrapper still calling `cotdata-prices --prices` now dies on an unrecognised flag, which
-> Task Scheduler shows as a failed run — loud, not silent.
+> wrapper still calling `cotdata-prices` now fails to resolve the command at all — that entry
+> point no longer exists — which Task Scheduler shows as a failed run. Loud, not silent.
 
 ## Goal
 
@@ -24,7 +24,7 @@ New to Python and cotdata on Windows? Start with the [Windows Setup Guide](WINDO
 
 ## Wrapper scripts
 
-Create **two** wrapper scripts — they run *different* commands from *different* packages: `marketdata-update` for the bars, `cotdata-cot` for the COT half. cotdata's own entry points stay half-scoped (`cotdata-cot` / `cotdata-prices`) and each refuses the other half's flags, so a host is scoped to one job and a price box cannot quietly become a second COT producer.
+Create **two** wrapper scripts — they run *different* commands from *different* packages: `marketdata-update` for the bars, `cotdata-cot` for the COT. (`cotdata-cot` is an alias of `cotdata-update`; it used to be half of a scoped pair, and the other half went with the price producers.)
 
 > **Ready-made templates:** copy [`docs/examples/windows/run-prices.cmd`](examples/windows/run-prices.cmd) and [`run-cot.cmd`](examples/windows/run-cot.cmd) out of the repo into your `<DIR>` (e.g. `C:\Users\you\cotdata\scheduler\`) rather than retyping them — then just fill in the placeholders. Keep them outside the repo so a `git pull` never clobbers your edited paths.