Remove databento: cotdata is CFTC positioning, in full - #108
Merged
Conversation
The companion to marketdata's port. ADR-0007 is now implemented rather than mostly implemented -- there is no price surface here of any kind. Gone: providers/databento.py, store.write_prices/read_prices, the --ingest-databento/--build-databento/--reconcile-databento/--batch/ --windowed-n1-stats flags, the cotdata-prices entry point, the [databento] extra, both parity harnesses and four test files. THE REGISTRY LOST ITS VENDOR COLUMNS. Symbol is down to internal, asset_class, is_equity, report_type, cftc_code and hist_codes; PRICE_SOURCES, resolve_source, default_price_source and COTDATA_PRICE_SOURCE are gone with the norgate/yahoo/ databento mappings. A comment I left in the previous PR justified keeping them on the grounds that a deployment might SHARE one registry file between the two packages via COTDATA_REGISTRY. That was wrong, and the code now says so: this loader hard-requires cftc_code and marketdata's equities have none, so the two files can never be the same file. Nothing outside this repo read those fields -- the three fleet hits were prose. THE PRODUCER-HALF MACHINERY IS GONE, because there is one producer. cotdata-cot survives as a bare alias of cotdata-update, since the scheduled jobs call it by name; cotdata-prices does not, because there is no price half for it to run. WHAT STAYS, deliberately: `prices` and `metadata` are still DECLARED domains with no writer and no reader. A real store carries entries under both, and an undeclared domain is SKIPPED by --migrate-manifests -- undeclaring them would strand those entries in the legacy aggregate forever and make --reconcile resolve their directory by guess instead of by declaration. config.prices_dir() and metadata_dir() survive for exactly that path and say so. The manifest-seam tests needed rewriting rather than repointing, because their premise changed. They were about two LIVE halves; they are now about one live half beside retired ones, so `prices` appears throughout as legacy data written by hand -- there is no writer -- and the property under test is that a COT write leaves that file byte-for-byte alone. Two tests were replaced outright: one asserted cotdata-prices refuses a COT action (no such entry point), and one asserted every action flag is assigned to a half (no such halves). Verified: ruff clean, 199 pass. npf's full suite against this branch plus the ported marketdata gives the same 38 failures as the pre-port baseline, none new -- all pre-existing sibling-checkout gaps in this sandbox.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to marketdata#15, which is where databento now lives. Merge that one first.
ADR-0007 is now implemented rather than mostly implemented: there is no price surface here of any kind. 3,424 lines out, 450 in.
Gone
providers/databento.py·store.write_prices/read_prices·--ingest-databento/--build-databento/--reconcile-databento/--batch/--windowed-n1-stats· thecotdata-pricesentry point · the[databento]extra · both parity harnesses · four test files. Full migration table in the CHANGELOG.This package now has no optional data dependency at all — CFTC positioning is a plain HTTP download of public files, and the
databentoextra was the last one.The registry lost its vendor columns
Symbolis down tointernal,asset_class,is_equity,report_type,cftc_code,hist_codes.PRICE_SOURCES,resolve_source,default_price_sourceand$COTDATA_PRICE_SOURCEgo with thenorgate/yahoo/databentomappings.A comment I left in the previous PR was wrong, and the code now says so. It justified keeping those columns on the grounds that a deployment might share one registry file between the two packages via
$COTDATA_REGISTRY. It cannot: this loader hard-requirescftc_codeand marketdata's equities have none, so the two files can never be the same file. Nothing outside this repo read the fields — a fleet sweep found three hits, all prose.The producer-half machinery is gone
There is one producer.
cotdata-cotsurvives as a bare alias ofcotdata-updatebecause the scheduled jobs call it by name;cotdata-pricesdoes not, because there is no price half for it to run._HALF_ACTIONSand_reject_other_halfare removed.What stays, deliberately
pricesandmetadataremain declared domains with no writer and no reader, andconfig.prices_dir()/metadata_dir()survive to serve them. A real store carries entries under both, and an undeclared domain is skipped by--migrate-manifests— undeclaring them would strand those entries in the legacy aggregate forever and make--reconcileresolve their directory by guess instead of by declaration.The seam tests were rewritten, not repointed
Their premise changed. They were about two live halves; they are now about one live half beside retired ones. So
pricesappears throughout as legacy data written by hand — there is no writer — and the property under test becomes: a COT write leaves that file byte-for-byte alone.Two were replaced outright rather than adapted: one asserted
cotdata-pricesrefuses a COT action (no such entry point), and one asserted every action flag is assigned to a half (no such halves). Both were replaced with guards for what actually matters now — that the retired flags are refused rather than silently ignored, and that no price-writing function can come back.Verification
ruff check src tests scriptsclean; 199 pass.marketdata-updateand every retired flag exits non-zero.Generated by Claude Code