Repository navigation
Prepare optional market, classification and loading inputs #111
Description
User workflow and ownership
Implement the accepted Engineer preparation/migration unit below. Preserve existing native defaults, dataframe contracts and independent use with supplied data. Data owns source adjustments/revision selection; Diagnostic owns the only Alphalens facade/cleaner and analytical workflows; Backtest owns simulation/accounting. Engineer produces native features, eligibility and outcomes, not competing engines.
Dependency: the dated feature eligibility and basis mapping in #109 (merged by #115), where reused. Valid external tables allow
normalization and consumer-independent tests to start earlier. Keep related preparation with that feature workflow if
it uses the same bounded implementation; split only independently risky behavior, not each helper.
Use existing contracts/Polars joins and optional conversion recipes, not new provider or analysis
engines. Only introduce a reusable public helper when the actual recipes demonstrate duplication.
Convert legacy price/volume field-indexed wide frames and date/asset loading frames into keyed
long inputs. Accept already native tables unchanged after validation. Static sector dicts are
explicitly static, not evidence of historical truth. Dated sectors/loadings use effective and
knowledge times, named factor columns and preserved coverage. Price, share/volume basis, currency
and multipliers must agree. Do not infer equity units for other asset classes or impute missing
loadings as zero. Missing optional inputs affect only the dependent preparation/analysis route.
Keep raw market observations separate from derived liquidity features. Pyfolio's capacity
denominator uses lagged rolling dollar volume, mean(price*volume).shift(1), not current price
times lagged volume. Name the derived product, window, lag and availability if Engineer produces
it. Do not double-lag already prepared inputs. Descriptive post-period attribution and causal
pre-trade preparation have different information sets and must be declared as such.
Proposed focused file: tests/test_analytical_input_preparation.py. Verify with
uv run pytest tests/test_analytical_input_preparation.py -q. Use actual public preparation with
split-compatible raw/adjusted fields, a delayed loading, changing sector, missing asset/factor,
genuine zero volume and no-volume record. Compare retained key/coverage/unit oracles and the
independent lagged dollar-volume path; apply future price/sector/loading shocks and assert that
earlier causal prepared output is unchanged. Inputs and saved artifacts remain unchanged on error.
Exposure/capacity arithmetic is only a boundary oracle here, not Engineer production functionality.
Premises
Input: optional analytical inputs
| Question | Answer | Kind |
|---|---|---|
| record | One market observation, supplied factor loading or static/dated classification with declared units and basis. | definitional |
| absent-record | Missing volume/loading/sector is uncovered optional data, not zero liquidity, zero exposure or default sector. | definitional |
| ordering-spacing | Explicit effective/as-of keys and lagged completed windows define preparation; no median-spacing inference. | definitional |
| entity-identity | Stable asset IDs and factor names join records; monetary holdings and fractional weights are distinct. | definitional |
| known-at | Causal preparation uses only available completed records; retrospective purposes are declared separately. | definitional |
| institution | Supplied Data metadata and caller declarations establish basis/units; receiving analysis owns its financial definitions. | definitional |
Claims
| Id | Claim | How it could be false |
|---|---|---|
| optional-input-integrity | Preparation preserves identity, units, missing coverage and the selected information set. | A join duplicates entities, fills a missing loading or uses a future classification. |
| liquidity-definition | Optional derived liquidity inputs use the declared product/window/lag exactly once. | Current price times lagged volume or double lagging changes the denominator. |
Closes: optional-analytical-inputs.record, optional-analytical-inputs.absent-record, optional-analytical-inputs.ordering-spacing, optional-analytical-inputs.entity-identity, optional-analytical-inputs.known-at, optional-analytical-inputs.institution, optional-input-integrity, liquidity-definition
Intake and completion
- Reproduce public producer defects before repairs and retain exact failing/passing revisions.
- Complete the methods, negative/error cases, runnable tests and migration route above.
- Verify actual installed outputs and the applicable final-revision quality/platform gates.
- Record producer/consumer dependencies separately from standalone qualification.
Plan reference: accepted Engineer preparation and Reloaded migration plan, unit 5 in milestone unreleased - Engineer preparation and Reloaded migration.
Completion evidence: #116 at reviewed head 254905b723c5d96a385f150bc0bcd4ce3d25d1e0 passed 5,670 native and 5,670 no-JIT tests, critical coverage, strict docs, types, lint and hooks. All 49 hosted contexts completed with 48 successes and one configured exception-validator skip, including all 12 installed/native/shared matrix cells and final Build Package. Actual built-wheel/source-archive and identified producer/consumer journeys passed. These qualify Engineer preparation; native Diagnostic capacity/supplied-loading analytics and whole-portfolio replacement are not claimed.
AI assistance: Codex assisted with implementation, regressions, documentation, installed qualification and these updates. The source, independent expectations and actual workflow results were reviewed in the owning maintainer session.
Activity
fix(packaging): include runnable migration recipes and documentation …
feat(migration): prepare optional inputs and qualify installed workfl…