Skip to content

Remove databento: cotdata is CFTC positioning, in full - #108

Merged
mspinola merged 1 commit into
mainfrom
claude/remove-databento
Aug 9, 2026
Merged

Remove databento: cotdata is CFTC positioning, in full#108
mspinola merged 1 commit into
mainfrom
claude/remove-databento

Conversation

@mspinola

@mspinola mspinola commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Companion to marketdata#15, which is where databento now lives. Merge that one first.

ADR-0007 is now implemented rather than mostly implemented: there is no price surface here of any kind. 3,424 lines out, 450 in.

Gone

providers/databento.py · store.write_prices / read_prices · --ingest-databento / --build-databento / --reconcile-databento / --batch / --windowed-n1-stats · the cotdata-prices entry point · the [databento] extra · both parity harnesses · four test files. Full migration table in the CHANGELOG.

This package now has no optional data dependency at all — CFTC positioning is a plain HTTP download of public files, and the databento extra was the last one.

The registry lost its vendor columns

Symbol is down to internal, asset_class, is_equity, report_type, cftc_code, hist_codes. PRICE_SOURCES, resolve_source, default_price_source and $COTDATA_PRICE_SOURCE go with the norgate/yahoo/databento mappings.

A comment I left in the previous PR was wrong, and the code now says so. It justified keeping those columns on the grounds that a deployment might share one registry file between the two packages via $COTDATA_REGISTRY. It cannot: this loader hard-requires cftc_code and marketdata's equities have none, so the two files can never be the same file. Nothing outside this repo read the fields — a fleet sweep found three hits, all prose.

The producer-half machinery is gone

There is one producer. cotdata-cot survives as a bare alias of cotdata-update because the scheduled jobs call it by name; cotdata-prices does not, because there is no price half for it to run. _HALF_ACTIONS and _reject_other_half are removed.

What stays, deliberately

prices and metadata remain declared domains with no writer and no reader, and config.prices_dir()/metadata_dir() survive to serve them. A real store carries entries under both, and an undeclared domain is skipped by --migrate-manifests — undeclaring them would strand those entries in the legacy aggregate forever and make --reconcile resolve their directory by guess instead of by declaration.

The seam tests were rewritten, not repointed

Their premise changed. They were about two live halves; they are now about one live half beside retired ones. So prices appears throughout as legacy data written by hand — there is no writer — and the property under test becomes: a COT write leaves that file byte-for-byte alone.

Two were replaced outright rather than adapted: one asserted cotdata-prices refuses a COT action (no such entry point), and one asserted every action flag is assigned to a half (no such halves). Both were replaced with guards for what actually matters now — that the retired flags are refused rather than silently ignored, and that no price-writing function can come back.

Verification

  • ruff check src tests scripts clean; 199 pass.
  • CLI smoke: the nothing-to-do message names marketdata-update and every retired flag exits non-zero.
  • npf's full suite against this branch plus the ported marketdata: 38 failures, identical to the pre-port baseline, none new — all pre-existing sibling-checkout gaps in this sandbox.

Generated by Claude Code

The companion to marketdata's port. ADR-0007 is now implemented rather than
mostly implemented -- there is no price surface here of any kind.

Gone: providers/databento.py, store.write_prices/read_prices, the
--ingest-databento/--build-databento/--reconcile-databento/--batch/
--windowed-n1-stats flags, the cotdata-prices entry point, the [databento]
extra, both parity harnesses and four test files.

THE REGISTRY LOST ITS VENDOR COLUMNS. Symbol is down to internal, asset_class,
is_equity, report_type, cftc_code and hist_codes; PRICE_SOURCES, resolve_source,
default_price_source and COTDATA_PRICE_SOURCE are gone with the norgate/yahoo/
databento mappings. A comment I left in the previous PR justified keeping them on
the grounds that a deployment might SHARE one registry file between the two
packages via COTDATA_REGISTRY. That was wrong, and the code now says so: this
loader hard-requires cftc_code and marketdata's equities have none, so the two
files can never be the same file. Nothing outside this repo read those fields --
the three fleet hits were prose.

THE PRODUCER-HALF MACHINERY IS GONE, because there is one producer. cotdata-cot
survives as a bare alias of cotdata-update, since the scheduled jobs call it by
name; cotdata-prices does not, because there is no price half for it to run.

WHAT STAYS, deliberately: `prices` and `metadata` are still DECLARED domains with
no writer and no reader. A real store carries entries under both, and an
undeclared domain is SKIPPED by --migrate-manifests -- undeclaring them would
strand those entries in the legacy aggregate forever and make --reconcile resolve
their directory by guess instead of by declaration. config.prices_dir() and
metadata_dir() survive for exactly that path and say so.

The manifest-seam tests needed rewriting rather than repointing, because their
premise changed. They were about two LIVE halves; they are now about one live half
beside retired ones, so `prices` appears throughout as legacy data written by hand
-- there is no writer -- and the property under test is that a COT write leaves
that file byte-for-byte alone. Two tests were replaced outright: one asserted
cotdata-prices refuses a COT action (no such entry point), and one asserted every
action flag is assigned to a half (no such halves).

Verified: ruff clean, 199 pass. npf's full suite against this branch plus the
ported marketdata gives the same 38 failures as the pre-port baseline, none new --
all pre-existing sibling-checkout gaps in this sandbox.
@mspinola
mspinola marked this pull request as ready for review August 9, 2026 14:58
@mspinola
mspinola merged commit f5b001d into main Aug 9, 2026
5 checks passed
@mspinola
mspinola deleted the claude/remove-databento branch August 9, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants