diff --git a/docs/api/pipeline-ports.md b/docs/api/pipeline-ports.md new file mode 100644 index 00000000..cd4057a9 --- /dev/null +++ b/docs/api/pipeline-ports.md @@ -0,0 +1,88 @@ +# Pipeline operation ports + +This reference maps the factor, filter and classifier surface inspected in +Zipline Reloaded 3.2.0 source `cf649ac06fddce76f6caa2403cd8494ac84f72ad`. +Routes are native Polars operations or explicit recipes, not legacy graph +objects. `factor_expression` and `return_factor_features` live under +`ml4t.engineer.compat.zipline`; rolling methods live under `features.rolling`, +relationships under `features.paired`, and selections under `features.selection`. +Use the [migration how-to](../user-guide/pipeline-migration.md) to prepare inputs. + +## Named factors and aliases + +| Legacy name | Native route | Definition and choices | +|---|---|---| +| `Returns` | `factor_expression` or precise `features.utils.arithmetic.returns` | N prices become N-1 changes; signed reference denominator; fraction, not percentage points. | +| `DailyReturns` | Same recipe with two prices | One change; retain both endpoint quote masks. | +| `PercentChange` | `absolute_percent_change` or recipe | Absolute reference denominator; fractional result. | +| `RateOfChangePercentage` | `features.utils.arithmetic.percentage_change` or recipe | Signed reference denominator, percentage points (100 times the fractional return). | +| `SimpleMovingAverage` | `finite_mean` or recipe | Full row window, omit null/NaN values from mean denominator. | +| `WeightedAverageValue`, `VWAP` | `finite_weighted_mean` or recipes | Independent NaN sums of value*weight and weights; a missing value does not erase its weight. | +| `AverageDollarVolume` | `average_dollar_volume` or recipe | Sum of price*volume divided by fixed N; distinguish quote units and volume basis. | +| `LinearWeightedMovingAverage` | `linear_weighted_mean` or recipe | Weights 1..N and fixed weight denominator; omit missing numerator terms. | +| `ExponentialWeightedMovingAverage`, `EWMA` | `finite_ewma` or recipes | Finite normalized weights, missing values propagate. Not native recursive EMA. | +| `ExponentialWeightedMovingStdDev`, `EWMSTD` | `finite_ewm_std` or recipes | Same finite weights plus unbiased weighted variance correction. | +| EWMA/EWMSTD `.from_span`, `.from_halflife`, `.from_center_of_mass` | Recipe suffix or `exponential_decay` | Exactly one weight selector; finite-window semantics remain unchanged. | +| `AnnualizedVolatility` | `annualized_volatility` or recipe | Population stddev, default annualization factor 252 independent of calendar; default price recipe derives daily returns with extra warmup. Supplied returns avoid that dependency. | +| `MaxDrawdown` | `finite_max_drawdown` or recipe | Legacy selected largest absolute fall and trough denominator; not ordinary peak-denominator portfolio drawdown. | +| `RSI` | `rsi_cutler` or recipe | 15-price default, 14 finite changes, omit changes adjacent to missing prices; flat windows undefined. Native `rsi` is Wilder. | +| `BollingerBands` | `finite_bollinger_bands` or recipe | Population stddev, default multiplier 2, upper/middle/lower named outputs. | +| `Aroon` | `finite_aroon` or recipe | First extrema ties; indices divided by N-1. Recipe preserves legacy low/high input order. | +| `FastStochasticOscillator` | `fast_stochastic` or recipe | Unsmoothed K, no D output; full price window. | +| `IchimokuKinkoHyo` | `finite_ichimoku` or recipe | Unshifted named outputs; legacy zero Tenkan/Kijun options select the entire window, not an inferred default. | +| `MovingAverageConvergenceDivergenceSignal`, `MACDSignal` | `finite_macd_signal` or recipes | Finite normalized exponential windows; not recursively seeded TA-Lib MACD. | +| `TrueRange` | `finite_true_range` or recipe | Two-price dependency, NaN-max of high-low and gaps; not ATR. | +| `RollingPearson`, `RollingSpearman` | Recipes or paired rolling functions | Full paired windows, explicit missing-count allowance; Spearman averages ties inside each window. | +| Factor `.pearsonr`, `.spearmanr`, `.linear_regression`; `RollingLinearRegression` | Paired functions or recipes | Explicit target alignment; regression Struct contains alpha, beta, r_value, p_value, stderr and requires the optional `stats` extra. | +| `RollingPearsonOfReturns`, `RollingSpearmanOfReturns`, `RollingLinearRegressionOfReturns` | `return_factor_features` | Align common grid before deriving both price returns; `returns_length` counts prices, correlation/regression length counts return rows. | +| `SimpleBeta` | `return_factor_features` | Two-price daily returns, default maximum missing fraction .25 converted to a row count. Target is declared stable ID, not opaque Asset. | +| `BusinessDaysSincePreviousEvent`, `BusinessDaysUntilNextEvent` | `event_business_days` or recipes | Supplied known event Dates, end-exclusive Monday-Friday arithmetic including exchange holidays. Not exchange-session windows. | +| `PeerCount` | `cross_section(method="peer_count", groupby=...)` | Eligible peers with known classifier; explicit missing sentinel to null first. | +| Numeric, categorical or Boolean `Latest` | `pl.col(...)` or `Latest` recipe | Preserve dtype and missingness; no implicit historical fill. | +| `Clip` and Factor `.clip` | `pl.col(...).clip(...)` or recipe | Explicit bounds, nulls preserved; apply output mask after history. | +| `Factor`, `CustomFactor`, `RecarrayField` | Native expressions/feature registry, Struct field selection | Supporting graph roles are ports, not automatically executable legacy objects. | + +## Factor methods and expression operators + +| Operation | Native route | Material choices | +|---|---|---| +| Arithmetic `+`, `-`, `*`, `/`, `**`, `%`, reflected forms, unary minus | Corresponding `pl.Expr` operations | Native null and integer overflow rules; do not cast large tick prices before differencing. | +| Comparisons `<`, `<=`, `>`, `>=`, `.eq`, `!=`; Boolean `&`, `|`, `~` | Native expressions over explicitly normalized missing inputs | Do not use Python `and`/`or`. Polars orders NaNs differently from NumExpr comparisons; normalize NaN to null and declare the Boolean null policy, or preserve a specifically required legacy NaN comparison explicitly. Final eligibility still excludes undefined features. | +| `sin`, `cos`, `tan`, `arcsin`, `arccos`, `arctan`, `sinh`, `cosh`, `tanh`, `arcsinh`, `arccosh`, `arctanh` | Same-named Polars expression functions | Domain-invalid real values remain undefined. | +| `log`, `log10`, `log1p`, `exp`, `expm1`, `sqrt`, `abs` | `.log()`, `.log(10)`, `.log1p()`, `.exp()`, `numpy.expm1(expr)`, `.sqrt()`, `.abs()` | Use expm1 directly for tiny inputs, not subtraction from a rounded exponential. | +| `.isnan`, `.notnan`, `.isfinite` | `is_missing`, inverse `is_missing`, `.is_finite().fill_null(False)` | Native nulls join NaNs in missingness; infinities are not missing but are nonfinite. | +| `.rank` | `cross_section(method="rank")` | ordinal/min/max/dense/average, ascending direction; ordinal ties use stable ascending IDs, not input row order. | +| `.top`, `.bottom` | Same-named cross-section methods | Exact N, stable ordinal ties; mask determines input population. | +| `.demean`, `.zscore` | Same-named cross-section methods | Contemporary date/group population, population ddof=0 default; zero variance undefined. Not a train-fitted scaler. | +| `.winsorize` | Same-named cross-section method | Floor/ceil legacy order-statistic bounds, not linear quantile clipping. | +| `.quantiles`, `.quartiles`, `.quintiles`, `.deciles` | `cross_section(method="quantiles", bins=...)` | 0-based labels, stable linear interpolation, duplicate-edge error, null exclusions. Nonmissing infinite values fail explicitly. Aliases use 4/5/10 bins. Floating values retain their local scale; integer/Decimal differences precede float conversion. | +| `.percentile_between` | Same-named cross-section method | Inclusive linear bounds; convert legacy percent values 0..100 to fractions 0..1. | +| `.mean`, `.stddev`, `.max`, `.min`, `.median`, `.sum`, `.notnull_count` daily summaries | Mask input then aggregate with native expressions over decision-time keys | Omit null/NaN, population stddev; sum of all missing values is zero, count is zero. Return one row per decision, not one per asset. | +| Asset slicing and named multi-output fields | Exact keyed filter/join; `.struct.field(name)` | Stable keys and explicit output fields replace opaque Asset lookup/record wrappers. | + +## Filters and classifiers + +| Legacy surface | Native route | Material choices | +|---|---|---| +| `All`, `Any`, `AtLeastN` | `rolling_filter` with all/any/at_least | Full original windows, missing predicates false; apply `.over(entity_col)`. | +| `AllPresent` | `rolling_filter(~is_missing(value), method="all")` | Known row presence is separate from actual value usability. | +| `MaximumFilter` | `cross_section(method="top", n=1)` | One maximum per eligible group, stable-ID first tie. The pinned legacy implementation reinterprets float storage as signed integers and selects the wrong maximum for all-negative groups. Native numeric ordering deliberately corrects that defect. | +| `NullFilter`, `NotNullFilter` | `is_missing`, inverse | Normalize legacy sentinels explicitly; no automatic -1 interpretation. | +| `PercentileFilter` | Cross-section percentile_between | Inclusive bounds and eligible population. | +| `SingleAsset`, `StaticAssets`, `StaticSids` | Stable-ID equality or `.is_in(...)` | Explicitly resolve legacy Asset.sid through its owner; static membership is not a historical universe. Legacy SingleAsset additionally raises unless its asset is alive throughout the requested range. Check that lifetime condition explicitly if retaining its strict behavior; a native equality filter alone does not raise. | +| `ArrayPredicate`, `NumExprFilter` | Native elementwise Boolean expressions | Preserve declared null behavior; do not execute arbitrary legacy expression strings. | +| `Filter.if_else` | `pl.when(predicate).then(left).otherwise(right)` | Declare unknown-predicate handling explicitly. | +| `CustomFilter`, base `Filter` | Boolean expressions or reusable registered features | No graph interpreter; supply required history and knowledge clocks. | +| Classifier `eq`, `ne`, `element_of` | Equality, inequality, `.is_in(...)` | Map missing sentinel to null; fill null Boolean outputs as intended. | +| `startswith`, `endswith`, `has_substring` | `.str.starts_with`, `.str.ends_with`, `.str.contains(..., literal=True)` | Literal substring, not accidental regex interpretation. | +| `matches` | Explicit `re.compile(pattern).match(value)` mapping | Legacy Python re.match starts at the beginning and supports Python regex features. Polars unanchored Rust regex is not a general equivalent. | +| `relabel` | `.replace_strict(mapping, default=None)` or explicit mapping function | Preserve nulls and declare behavior for unknown labels. | +| `Quantiles` | Cross-section quantiles | Same bins/interpolation/missing choices as Factor method. | +| `Everything` | A declared constant valid group | Missing classifications require a separate policy, not automatic inclusion. | +| `Classifier.peer_count` | Cross-section peer_count | Contemporary eligible group count, not full static universe size. | +| `CustomClassifier`, base `Classifier` | Supplied categories or typed native expression | Retain dtype, nulls and known-at information; encode numeric sentinels only at an explicit receiver boundary. | + +Daily summaries, elementary operators and custom ports use existing Polars +functionality rather than duplicate Engineer kernels. The reference identifies +routes and material differences; listing a route alone is not a claim that every +possible legacy graph or regular expression has been qualified. diff --git a/docs/user-guide/features.md b/docs/user-guide/features.md index 1b05bfc5..53e742c2 100644 --- a/docs/user-guide/features.md +++ b/docs/user-guide/features.md @@ -138,6 +138,376 @@ expressions, not portfolio statistics. Use explicit grouping for multiple assets validate the declared bar grid before deriving returns as described below. Row-only data must already have the intended observation order. +### Explicit finite-window methods + +The next minor version adds the following registered methods for workflows that need +finite observation windows rather than the native recursive or TA-Lib definitions. +Existing `rsi`, `ema`, `macd`, `aroon`, `bollinger_bands` and risk features keep their +defaults. These additions do not provide a Pipeline graph interpreter. + +| Feature | Definition and missingness | +|---|---| +| `rsi_cutler` | Simple gains/losses over `period` changes (`period + 1` prices). Omit changes adjacent to missing quotes; a flat window is undefined. | +| `absolute_percent_change` | Fractional change over `period` changes with an absolute reference denominator. This accepts signed values and does not multiply by 100. | +| `finite_mean` | Require `period` rows, then divide by the count of nonmissing prices. | +| `finite_weighted_mean` | Sum valid price-times-volume products, divided by the independently summed volume. Missing prices do not remove denominator weights. | +| `average_dollar_volume` | Sum valid price-times-volume products, divided by the fixed `period` row count. Input price, share and currency basis must already agree. | +| `linear_weighted_mean` | Weights `1..period`, oldest to newest. Omit missing products but retain every denominator weight. | +| `finite_ewma` | Normalize `decay_rate**age` over exactly `period` prices. Missing prices propagate; this is not recursive EMA. | +| `finite_ewm_std` | The same finite weights, with weighted bias correction. Insufficient effective degrees of freedom yield NaN. | +| `annualized_volatility` | Population standard deviation (`ddof=0`) of supplied `returns`, omitting missing values after the full window. Multiply by `sqrt(annualization_factor)`, default 252. Calendar choice does not set this scale. | +| `finite_max_drawdown` | Select the first largest absolute price drop, then divide by its trough price. This explicit legacy definition is not a peak-denominator portfolio drawdown. | +| `finite_bollinger_bands` | NaN-omitting mean and population standard deviation; fields `lower`, `middle`, `upper`, with multiplier `k`. | +| `finite_aroon` | First extreme wins ties; fields `down`, `up`. Scale the position in `period` prices by `100 / (period - 1)`. | +| `fast_stochastic` | Unsmoothed K from the latest close and NaN-omitting high/low extrema. Zero range remains undefined. | +| `finite_ichimoku` | Unshifted `tenkan_sen`, `kijun_sen`, `senkou_span_a`, `senkou_span_b`. `chikou_position` indexes from the oldest price in the window, unlike a chart displacement. | +| `finite_macd_signal` | Finite normalized fast/slow exponential means, followed by a finite signal mean. Length-one means use their sole observation. | +| `finite_true_range` | Two-price true range with NaN-omitting high/low/previous-close components. There is no implicit larger-window reduction. | + +Except for Cutler RSI, absolute percent change and the composed MACD signal, `period` counts input rows and the +first structurally usable row is `period - 1`. Missing observations still occupy their +original row positions. Numeric results are Float64 and retain every input row; +Ichimoku's selected `chikou_span` price retains its original dtype. Multi-output +features use the usual `_` column names. Ichimoku +propagates missing high/low values within each field's subwindow; all fields require +the declared full window. Its zero tenkan/kijun lengths select that full window, +matching the legacy `[-0:]` slice, while `chikou_position` must be in `[0, period)`. + + +```python +import math +import polars as pl +from ml4t.engineer import compute_features + +prices = pl.DataFrame({"close": [10.0, 13.0, 12.0, 14.0, 13.0]}).with_columns( + pl.col("close").alias("open"), + pl.col("close").alias("high"), + pl.col("close").alias("low"), + pl.lit(100.0).alias("volume"), +) +result = compute_features(prices.lazy(), [ + {"name": "rsi_cutler", "params": {"period": 3}, "output": "cutler"}, + {"name": "rsi", "params": {"period": 3}, "output": "wilder"}, +], calendar=None).collect() +assert math.isclose(result["cutler"][4], 50.0, rel_tol=1e-14, abs_tol=0.0) +assert result["wilder"][4] != result["cutler"][4] +assert result.height == prices.height +``` + +Use explicit grouping or `compute_features` for multiple assets. A `calendar=None` +call counts the supplied observation rows without checking schedules. With a named +calendar, declare the complete grid before computing any feature. The finite methods +do not infer frequency, select source revisions, fill gaps or make future information +eligible. They currently support observation/expected-bar windows; do not reinterpret +their kernels as session or elapsed windows. Variance methods subtract an original +window-local typed price anchor before floating conversion, so adjacent large integer ticks do +not collapse to zero variance. Output remains Float64. + +### Factor-name expression recipes + +`compat.zipline.factor_expression` translates supported built-in names into the +native methods above. It returns a Polars expression or a dictionary of named output +expressions, not a Pipeline Term. It never imports Zipline or evaluates an opaque +legacy graph. Use native column operators for arithmetic/comparisons and the existing +feature decorator for custom calculations. + +Recipe `window_length` counts prices/inputs as in Pipeline. Return and Cutler recipes +translate N prices into N-1 changes. The default annualized-volatility recipe derives +daily returns from `close` and keeps that extra dependency row of warmup. Explicit +`inputs=["returns"]` instead consumes supplied returns. EWMA/EWMSTD accept one explicit +`decay_rate`, `span`, `halflife` or `center_of_mass`, including the three legacy +constructor suffixes. Statistical recipes can consume already aligned column pairs; +their legacy correlation/regression lengths and beta missing percentage are explicit. + + +```python +import math +import polars as pl +from ml4t.engineer.compat.zipline import factor_expression + +frame = pl.DataFrame({"close": [1.0, 3.0, 9.0], "eligible": [True, False, True]}) +expression = factor_expression( + "EWMA.from_span", inputs=["close"], window_length=3, span=3, mask="eligible", +) +result = frame.lazy().with_columns(expression.alias("finite_mean")).collect() +assert math.isclose(result["finite_mean"][2], 43 / 7, rel_tol=1e-14) +assert result.height == frame.height +``` + +The output mask in this example does not erase the middle observation from history. +Apply `.over(entity_col)` for independent asset histories. These expression recipes +do not validate a market schedule. For bar data, validate the declared grid once with +`compute_features(data, [], feed_spec=...)` before deriving returns/features, or +explicitly choose `calendar=None` for observation rows. Recipe output masks are not +dated universe/knowledge selection or final output screens. Named fields stay explicit; +for example, select `factor_expression("BollingerBands", window_length=20)["middle"]`. + +### Eligible cross-sectional transforms and boolean windows + +`features.selection.cross_section` computes `rank`, `top`, `bottom`, `demean`, +`zscore`, `winsorize`, `quantiles`, `percentile_between` or `peer_count` separately +at each supplied timestamp and optional classification group. Its `mask` selects +the input population before the transform. A later dataframe filter is a final +screen and does not change those inputs. Unknown groups, nulls and NaNs are +excluded. Map legacy classifier sentinels such as -1 to null explicitly. + +Ranks support ordinal/min/max/dense/average ties. Ordinal ties and top/bottom +selection use ascending stable entity IDs, independently of input row order. +Supply unique timestamp/entity keys; display symbols are not stable identities. +Z-scores use population standard deviation by default (`ddof=0`), not a fitted +training scaler. Constant groups have undefined z-scores. Typed local differences +retain adjacent large integer ticks, and scaled moments avoid variance overflow +or underflow for representable outputs. + + +```python +from datetime import date +import polars as pl +from ml4t.engineer.features.selection import cross_section + +signals = pl.DataFrame({ + "timestamp": [date(2025, 7, 7)] * 4, "asset": ["D", "B", "A", "C"], + "value": [3., 1., 1., 999.], "eligible": [True, True, True, False], +}) +ranked = signals.with_columns( + cross_section("value", method="rank", mask="eligible").alias("rank"), + cross_section("value", method="bottom", n=1, mask="eligible").alias("selected"), +) +assert ranked["rank"].to_list() == [3., 2., 1., None] +assert ranked.filter("selected")["asset"].to_list() == ["A"] +``` + +Winsorization uses the legacy floor/ceil order-statistic cutoffs, not linear +percentile clipping. Quantile bins and inclusive percentile filters use linear +edges. Duplicate quantile edges raise instead of silently changing the number of +bins. Native quantile labels are zero-based integers with null missing values. +These feature selectors do not invoke Diagnostic's analytical factor cleaner. + +`features.selection.rolling_filter` supplies full-row `any`, `all` and `at_least` +windows. Apply `.over(entity_col)` to isolate histories. Missing predicates count +false; leading incomplete windows are null. `is_missing` recognizes native null +and floating NaN, without treating infinity or a classifier sentinel as missing. +Ordinary Polars arithmetic, comparisons, boolean operators and `is_in` cover the +corresponding elementwise factor/filter/classifier operations. None of these +expressions selects Data revisions or establishes availability by itself. + +### Decision-time feature preparation + +`features.preparation.prepare_feature_panel` consumes already prepared native +histories and supplied universe facts. Data or the caller selects source revisions +and adjustment perspective first. Engineer validates keys, preserves pre-entry +warmup and computes feature eligibility at each requested decision. It does not +choose the latest revision, apply corporate actions or fetch provider history. + +Supply a canonical `FeedSpec` instance. For a canonical serialized handoff, restore +it with `FeedSpec.from_mapping(...)` once. Unresolved config objects are rejected; +explicit column arguments cannot conflict with the declaration. A named calendar +requires the existing complete bar-grid declaration before any return/feature is +derived. Explicit `calendar=None` keeps observation-row windows. + +`decisions` has the observation cutoff, a separate aware `decision_time` and an +explicit decision-effective universe key. The default `universe_time_col` selects +`effective_session`, a supplied session identifier. It need not equal the prior +price session used for a morning decision. For instant-effective universes, set +`universe_time_col` to the explicitly supplied instant column, such as +`decision_time`. No effective session is inferred from a UTC date or from prices. +Universe rows have stable entity keys, `effective_from`, exclusive +`effective_to` (null means open), aware `available_at`, nullable boolean `member` +and optional classification columns. Effective intervals use that declared universe +clock. Date identifiers stay dates; instant comparisons preserve physical timezone +equivalence and ns boundaries. Missing/unknown facts never imply +membership. Overlapping known active facts fail with an upstream-resolution error. + + +```python +from datetime import UTC, date, datetime +import polars as pl +from ml4t.specs import FeedSpec +from ml4t.engineer.features.preparation import prepare_feature_panel +from ml4t.engineer.features.rolling import finite_mean + +days = [date(2025, 7, d) for d in (7, 8, 9)] +history = pl.DataFrame({ + "timestamp": days, "asset": ["A"] * 3, "close": [1., 2., 3.], + "available_at": [datetime(2025, 7, d.day, 16, tzinfo=UTC) for d in days], +}) +decisions = pl.DataFrame({ + "timestamp": [days[-1]], "decision_time": [datetime(2025, 7, 9, 17, tzinfo=UTC)], + "effective_session": [days[-1]], +}) +universe = pl.DataFrame({ + "asset": ["A"], "effective_from": [days[-1]], + "effective_to": pl.Series([None], dtype=pl.Date), "member": [True], + "available_at": [datetime(2025, 7, 8, 12, tzinfo=UTC)], +}) +result = prepare_feature_panel( + history.lazy(), features={"average": finite_mean("close", period=3)}, + lookbacks={"average": 3}, input_columns=["close"], + decisions=decisions, universe=universe, price_basis="raw", + feed_spec=FeedSpec(time_contract_version=2, calendar=None, entity_col="asset"), +).collect() +assert result["average"].to_list() == [2.] +assert result["eligible"].to_list() == [True] +``` + +This asset enters the universe on July 9 but retains July 7-8 warmup. Expressions +are bare per-asset expressions; the preparer applies entity grouping. Declare each +finite row lookback and all input columns. Only complete, known, usable required +windows become eligible. Unavailable or input-masked rows remain as null inputs, +not shorter windows. A missing endpoint does not carry an earlier feature forward. +For kernels with deliberate omission rules, select +`missing_input_policy="known_missing"` to allow confirmed null/NaN or input-masked +values within the full original window. Every row still needs a publication time +known by the decision. Unknown/future rows and infinite inputs remain ineligible. +This option neither adds pre-listing rows nor fills absent scheduled bars; consume +an explicitly supplied complete panel when that is what the original loader emits. +Undefined/nonfinite outputs, including undefined fields in regression Structs, +cannot become eligible. The output retains the observation/decision clocks, +conservative contributing availability, membership, eligibility, the declared +universe key and price basis. Membership is evaluated at that explicit key using +only facts known at the decision, not at the historical price cutoff. + +For per-decision restated histories, Data prepares each perspective and the caller +adds an aware snapshot column. Pass its name as `history_decision_time_col`; the +preparer requires an exact matching snapshot for each decision. Without that +column, duplicate history keys fail. It never selects or forward-fills a source +revision/perspective implicitly. Preserve the Data snapshot's provenance alongside +the resulting table. + +`input_mask` affects history before computation. Membership does not erase +pre-entry history. `screen` runs after all features are computed. Alternatively, +apply `cross_section(..., timestamp_col="decision_time", mask="eligible")` to the +output, then filter final selections. If one instant has multiple declared universe +contexts, include their universe key in the group keys too. Cross-sectional masks determine the contemporary population and are +not a substitute for information-time validation. + +Owned `FeatureSchema` records optional `available_at_col`, `decision_time_col` and +`eligible_col`, plus `universe_time_col`; `entity_col` preserves one column name +or a list of compound identity columns. `FeatureDefinition` records the canonical `feed_spec`, +`window_unit`, `price_basis`, finite `lookbacks` and optional `missing_input_policy`, with existing source-artifact +lineage and join keys. `FeatureSpec.to_dict()` preserves omitted calendar versus +explicit null; unused fields leave the old wire keys unchanged. Store actual +typed frames separately, for example with Parquet. These records do not select +source revisions or add a mandatory dataframe wrapper. + +### Paired relationships and supplied event dates + +`features.paired.rolling_pearson`, `rolling_spearman`, `rolling_beta` and +`rolling_regression` take explicit dependent/independent columns or expressions. +Align their actual time/entity keys before computing and apply `.over(entity_col)` +for asset-isolated windows. `period` counts original rows. `allowed_missing` is a +count, not a percentage; a missing pair is removed from both sides without +compressing the timeline. At least two valid pairs remain necessary. Spearman +recomputes average tie ranks inside each window. Pearson/beta/regression propagate +undefined nonmissing infinities. Constant independent regression windows produce +NaN fields instead of aborting the whole asset panel. + +Regression returns a Struct with `alpha`, `beta`, `r_value`, `p_value`, `stderr`. +Its inferential fields require the optional `stats` extra (`uv add +'ml4t-engineer[stats]'`). The other paired methods and base imports do not need SciPy. +This is predictive rolling regression on supplied observations, not trend-on-time +regression or portfolio attribution. + +For price panels, `features.paired.target_return_features` performs the exact +time/identity join and derives returns before applying the paired statistic. +Declare the target as a mapping of physical identity columns to stable values, +including all components for a compound identity. `price_basis` records the +supplied Data/caller declaration; this function does not adjust prices. + +The default `index_mode="common_grid"` temporarily retains absent quotes on the +supplied union clock through both return derivation and rolling calculation. +`per_asset` instead counts each asset's own observations. Neither mode fills +quotes, compresses missing pairs out of rolling windows, or emits synthetic rows. +A named calendar validates the original declared grid before calculating any +return. Explicit `calendar=None` selects observation rows. These finite statistics +support observation/expected-bar units, not session or elapsed windows. + + +```python +from datetime import date +import math +import polars as pl +from ml4t.specs import FeedSpec +from ml4t.engineer.features.paired import target_return_features + +days = [date(2025, 7, day) for day in (7, 8, 9, 10)] +prices = pl.DataFrame({ + "timestamp": days * 2, "asset": ["A"] * 4 + ["B"] * 4, + "close": [100., 120., 80., 112., 100., 110., 99., 118.8], +}) +result = target_return_features( + prices.lazy(), target={"asset": "B"}, period=3, method="beta", + price_basis="raw", feed_spec=FeedSpec(time_contract_version=2, calendar=None), +).collect() +assert math.isclose(result.filter(pl.col("asset") == "A")["target_beta"][-1], 52 / 21) +assert result.select(prices.columns).equals(prices) +assert result["target_available_at"].null_count() == prices.height +``` + +Added return columns expose the actual paired inputs. `_available_at` is the latest +publication instant across both assets' price dependencies in the full window. +Without complete supplied timezone-aware `available_at_col` values it stays +unknown. Observation timestamps are not fabricated publication times. The caller +must still enforce decision-time eligibility before using these features. + +`compat.zipline.return_factor_features` maps `RollingPearsonOfReturns`, +`RollingSpearmanOfReturns`, `RollingLinearRegressionOfReturns` and `SimpleBeta` +onto this native preparation. It translates `returns_length=N` to an N-1-price +lag, accepts the corresponding correlation/regression length, and converts +SimpleBeta's allowed-missing percentage to a count. Its optional output mask +does not remove the target's prices or warmup history. Pass stable target IDs, +not opaque Zipline Asset objects. This recipe returns a native frame, not a Term. + + +```python +import math +from importlib.util import find_spec +import polars as pl +from ml4t.engineer.features.paired import rolling_regression + +paired = pl.DataFrame({"market": [1.0, 2.0, 3.0, 4.0], "feature": [1.0, 2.0, 2.0, 4.0]}) +if find_spec("scipy") is None: + try: + rolling_regression("feature", "market", period=4) + except ImportError as error: + assert "ml4t-engineer[stats]" in str(error) + else: + raise AssertionError("Inferential regression must require its optional dependency") +else: + result = paired.lazy().select( + rolling_regression("feature", "market", period=4).alias("regression") + ).unnest("regression").collect() + assert math.isclose(result["beta"][3], 0.9, rel_tol=1e-14) + assert math.isclose(result["stderr"][3], math.sqrt(0.07), rel_tol=1e-14) + assert math.isclose(result["p_value"][3], 1 - 9 / math.sqrt(95), rel_tol=1e-14) +``` + +`features.events.event_business_days` counts Monday through Friday, including +exchange holidays, with the end date excluded. Supply separate event/reference +Date labels and timezone-aware publication/decision instants. Unknown publication +or publication after the decision yields null. This method neither selects an +event provider nor infers exchange sessions. `direction="until"` permits a known +future event; `direction="since"` uses the same supplied date with reversed endpoints. + + +```python +from datetime import date, datetime, timezone +import polars as pl +from ml4t.engineer.features.events import event_business_days + +events = pl.DataFrame({ + "event_date": [date(2025, 7, 4)], + "session": [date(2025, 7, 7)], + "known_at": [datetime(2025, 7, 1, tzinfo=timezone.utc)], + "decision": [datetime(2025, 7, 7, 14, tzinfo=timezone.utc)], +}) +result = events.select(event_business_days( + "event_date", reference_date_col="session", known_at_col="known_at", + decision_time_col="decision", direction="since", +).alias("weekdays")) +assert result["weekdays"][0] == 1.0 # July 4 is a weekday, despite the NYSE closure. +``` + ### Calendar and bar grid The calendar-aware default targets the next minor version (0.2), separate from the 0.1.7 patch. diff --git a/docs/user-guide/pipeline-migration.md b/docs/user-guide/pipeline-migration.md new file mode 100644 index 00000000..4b8c0e72 --- /dev/null +++ b/docs/user-guide/pipeline-migration.md @@ -0,0 +1,159 @@ +# Migrate Pipeline feature selection + +Port finite factors and dated screens to native Polars expressions. Keep source +history preparation in ML4T Data and order/account processing in ML4T Backtest. +Engineer does not run a Zipline graph or import Zipline in production. + +## Replace the calculation and retain its input contract + +The original momentum example constructs `RSI().top(3)` and `RSI().bottom(3)` in a +Pipeline. Its pricing loader supplies the previous session's close before the +current session opens. Preserve those different clocks when replacing the graph. + +Supply a long price panel with stable IDs, an observation timestamp and actual +publication times. Supply decisions with an observation cutoff, an aware +`decision_time` and an `effective_session` for membership. Supply dated universe +facts with `effective_from`, exclusive `effective_to`, `available_at`, `member` +and any classifications. Unknown membership is not membership. Data selects +source revisions, identities and adjustment perspectives before this boundary. + + +```python +from datetime import UTC, date, datetime +import polars as pl +from ml4t.specs import FeedSpec +from ml4t.engineer.compat.zipline import factor_expression +from ml4t.engineer.features.preparation import prepare_feature_panel +from ml4t.engineer.features.selection import cross_section + +days = [date(2025, 7, d) for d in (7, 8, 9)] +prices = pl.DataFrame({ + "timestamp": days * 3, + "asset": [17] * 3 + [23] * 3 + [31] * 3, + "close": [10., 11., 10., 10., 12., 13., 10., 9., 8.], + "available_at": [datetime(2025, 7, d.day, 20, tzinfo=UTC) for d in days] * 3, +}) +decisions = pl.DataFrame({ + "timestamp": [days[-1]], + "decision_time": [datetime(2025, 7, 10, 13, tzinfo=UTC)], + "effective_session": [date(2025, 7, 10)], +}) +universe = pl.DataFrame({ + "asset": [17, 23, 31], "effective_from": [date(2025, 7, 10)] * 3, + "effective_to": pl.Series([None] * 3, dtype=pl.Date), + "available_at": [datetime(2025, 7, 1, tzinfo=UTC)] * 3, + "member": [True] * 3, +}) +# This small example deliberately counts observation rows, not scheduled bars. +feed = FeedSpec(time_contract_version=2, calendar=None, entity_col="asset") +rsi = factor_expression("RSI", inputs=["close"], window_length=3) +assert isinstance(rsi, pl.Expr) +signals = prepare_feature_panel( + prices.lazy(), features={"rsi": rsi}, lookbacks={"rsi": 3}, + input_columns=["close"], decisions=decisions, universe=universe, + price_basis="raw", feed_spec=feed, missing_input_policy="known_missing", +).with_columns( + cross_section("rsi", method="top", n=1, timestamp_col="decision_time", + mask="eligible").alias("longs"), + cross_section("rsi", method="bottom", n=1, timestamp_col="decision_time", + mask="eligible").alias("shorts"), +).filter(pl.col("member").fill_null(False)).collect() +assert signals["rsi"].to_list() == [50., 100., 0.] +assert signals.filter("longs")["asset"].to_list() == [23] +assert signals.filter("shorts")["asset"].to_list() == [31] +assert signals["timestamp"].to_list() == [date(2025, 7, 9)] * 3 +assert signals["effective_session"].to_list() == [date(2025, 7, 10)] * 3 +``` + +The default legacy RSI uses 15 prices, so use `window_length=15` and +`lookbacks={"rsi": 15}` for that workflow. Native `rsi` remains Wilder RSI; +the recipe selects the separately named `rsi_cutler` definition. Select `n=3` +to reproduce the original selection count. + +For scheduled daily inputs, install `ml4t-engineer[calendars]` and declare +`calendar="NYSE"`, `timestamp_semantics="session_label"`, +`data_frequency="1d"`, and `bar_type="time"` in the feed instead of selecting +row mode. Omitted calendar follows feed metadata, then NYSE. Explicit `None` +disables calendars and does not mean a continuous schedule. Intraday bars also +need timezone, anchor, label side and partial-bar policy. Do not infer them from +the observations. Missing scheduled rows fail before a return or feature is +interpreted as one-bar data. + +The `known_missing` option permits a kernel's declared null/NaN omission rules +over a full, supplied row window. It does not invent pre-listing rows or treat +unknown publication as confirmed absence. A loader that emits a complete panel +with explicit pre-listing null cells must hand over those actual cells. Keep the +strict default when every input must be usable. Apply an input mask before the +calculation and a final screen after it. Filtering history to current membership +loses pre-entry warmup. Partition cross-sections by decision time, not the prior +price session; add the universe key if one instant has multiple contexts. + +## Port custom terms + +Run `uv run python examples/pipeline_migration.py` in the source checkout. The +example includes a finite custom mean, a positive-value custom filter and an +explicit sector-code classifier alongside the momentum screen. Replace a +`CustomFactor.compute` loop with a bare per-entity expression and declare its +finite lookback and all input columns. The preparer supplies entity grouping. +For reusable calculations, use the existing feature decorator and registry. +Replace `CustomFilter` with a Boolean expression, and `CustomClassifier` with a +supplied classification or an explicit native mapping. Retain missing categories +as null; do not convert them into a real category. Select multi-output Struct +fields explicitly. Opaque legacy objects, loaders and graph execution are not +accepted as expressions. + +## Adapt presentation only where needed + +Keep the native table for subsequent feature or label preparation. For a caller +that still expects a pandas MultiIndex, select `effective_session` and stable +asset IDs as its keys, then convert at this explicit eager boundary. With +PyArrow installed, `signals.select("effective_session", "asset", "longs", +"shorts").to_pandas(date_as_object=True).set_index(["effective_session", +"asset"])` preserves civil Date labels rather than claiming they are instants. +Rename the index levels to `date` and `asset` if that is the caller's contract. +For instant keys, preserve their timezone and nanoseconds. Never go through +Python `datetime` values to reconstruct a nanosecond index. + +If a retained caller needs Zipline Asset objects, map stable IDs through that +caller's AssetFinder explicitly. Do not stringify compound IDs or introduce a +Zipline dependency to native calculations. Native compound entity columns and +Struct IDs can remain unchanged; an old pandas caller must declare its own +hashable compound-key representation. This is a caller presentation edit, not +an automatic emulation of `Pipeline.run_pipeline`. + +## Preserve prepared-history metadata + +Consume `AdjustedHistory.prices` and the canonical feed in its +`metadata["feed_spec"]`. Use the declared price column and price basis, not a +hard-coded raw close. Data's daily frequency declaration is not by itself a +complete Engineer time-bar grid; add the fixture/feed owner's explicit bar-grid +declarations. For per-decision adjustment perspectives, retain the snapshot +decision key and pass `history_decision_time_col`. Do not reuse retrospectively +adjusted history as a causal snapshot. Store Data provenance alongside the +FeatureSpec and typed frame. + +The installed migration checks qualified Engineer development source +`cce590132846dbc7d558971b2d5a7f703d5cd9a2` against Zipline Reloaded 3.2.0 +(`cf649ac06fddce76f6caa2403cd8494ac84f72ad`), pandas 2.3.3 and NumPy 2.5.4. +Both original momentum PipelineEngine journeys used the real shifted pricing +loader and matched 135 keyed rows, RSI values and long/short selections each, +with complete quotes and a known-missing close, listing/delisting and supplied +lifetimes. This is development qualification, not an available release or proof +of order, fill, commission, leverage or portfolio-performance equivalence. + +Development source `3c5665cd2d08bad2dc5dc25b4a97573a84a7de88` additionally +passed both missingness profiles with actual CustomFactor, CustomFilter and +CustomClassifier terms, and 33 installed operation comparisons covering all 19 +elementary math functions, seven daily summaries, five return-definition/window +cases and both weekday-event classes. `examples/pipeline_expression_ports.py` +contains the executable math and daily-summary ports. Data's unpublished +`0.2.2.dev41+gd48d849f7` artifact at +`d48d849f77d3ff75d05383cbaaed9b42f6ca95b7` fed actual known-at histories into +Engineer source `cce590132846dbc7d558971b2d5a7f703d5cd9a2`: an appended future +correction left the earlier mean 199 unchanged, and the later adjusted history +retained stable asset 17 across OLD/NEW symbols with a final two-price mean 99.5. +These artifacts are qualification inputs, not published install recommendations. + +See the [operation reference](../api/pipeline-ports.md) for individual methods, +defaults and material differences, and [native features](features.md) for the +preparation and expression contracts. diff --git a/examples/pipeline_expression_ports.py b/examples/pipeline_expression_ports.py new file mode 100644 index 00000000..da4da9f1 --- /dev/null +++ b/examples/pipeline_expression_ports.py @@ -0,0 +1,67 @@ +"""Native ports of elementwise math and daily summaries, not a graph interpreter.""" + +import json + +import numpy as np +import polars as pl + + +def elementary_features(frame: pl.DataFrame | pl.LazyFrame) -> pl.DataFrame | pl.LazyFrame: + value = pl.col("value") + functions = { + "sin": value.sin(), + "cos": value.cos(), + "tan": value.tan(), + "arcsin": value.arcsin(), + "arccos": value.arccos(), + "arctan": value.arctan(), + "sinh": value.sinh(), + "cosh": value.cosh(), + "tanh": value.tanh(), + "arcsinh": value.arcsinh(), + "arccosh": value.arccosh(), + "arctanh": value.arctanh(), + "log": value.log(), + "log10": value.log(10), + "log1p": value.log1p(), + "exp": value.exp(), + "expm1": np.expm1(value), + "sqrt": value.sqrt(), + "abs": value.abs(), + } + return frame.with_columns(expr.alias(name) for name, expr in functions.items()) + + +def daily_summary(frame: pl.DataFrame | pl.LazyFrame) -> pl.DataFrame | pl.LazyFrame: + value = pl.when(pl.col("eligible").fill_null(False)).then(pl.col("value").fill_nan(None)) + return ( + frame.group_by("decision_time") + .agg( + value.mean().alias("mean"), + value.std(ddof=0).alias("stddev"), + value.max().alias("max"), + value.min().alias("min"), + value.median().alias("median"), + value.sum().alias("sum"), + value.count().alias("notnull_count"), + ) + .sort("decision_time") + ) + + +if __name__ == "__main__": + values = pl.DataFrame( + { + "decision_time": [1] * 4 + [2] * 2, + "value": [1.0, 3.0, float("nan"), 999.0, None, 5.0], + "eligible": [True, True, True, False, True, False], + } + ) + summary = daily_summary(values) + assert summary["mean"].to_list() == [2.0, None] + assert summary["stddev"].to_list() == [1.0, None] + assert summary["sum"].to_list() == [4.0, 0.0] + assert summary["notnull_count"].to_list() == [2, 0] + elementary = elementary_features(pl.DataFrame({"value": [1e-20, None]})) + assert elementary["expm1"].to_list() == [1e-20, None] + print(json.dumps(summary.to_dicts(), ensure_ascii=True, default=str)) diff --git a/examples/pipeline_migration.py b/examples/pipeline_migration.py new file mode 100644 index 00000000..ab69074c --- /dev/null +++ b/examples/pipeline_migration.py @@ -0,0 +1,99 @@ +"""Port finite custom factors, classifications and screens without a legacy graph runtime.""" + +import json +from datetime import UTC, date, datetime + +import polars as pl +from ml4t.specs import FeedSpec + +from ml4t.engineer.compat.zipline import factor_expression +from ml4t.engineer.features.preparation import prepare_feature_panel +from ml4t.engineer.features.selection import cross_section + + +def momentum_screen( + prices: pl.DataFrame | pl.LazyFrame, + decisions: pl.DataFrame, + universe: pl.DataFrame, + *, + feed_spec: FeedSpec, + price_basis: str, + window_length: int = 15, + n: int = 3, +) -> pl.DataFrame | pl.LazyFrame: + """Return supplied-member RSI selections plus a finite custom mean/classifier.""" + rsi = factor_expression("RSI", inputs=[feed_spec.price_col], window_length=window_length) + assert isinstance(rsi, pl.Expr) + panel = prepare_feature_panel( + prices, + features={ + "rsi": rsi, + "custom_mean": pl.col(feed_spec.price_col).rolling_mean(3, min_samples=1), + }, + lookbacks={"rsi": window_length, "custom_mean": 3}, + input_columns=[feed_spec.price_col], + decisions=decisions, + universe=universe, + price_basis=price_basis, + feed_spec=feed_spec, + missing_input_policy="known_missing", + ) + return panel.with_columns( + cross_section( + "rsi", method="top", n=n, timestamp_col="decision_time", mask="eligible" + ).alias("longs"), + cross_section( + "rsi", method="bottom", n=n, timestamp_col="decision_time", mask="eligible" + ).alias("shorts"), + pl.col("sector").replace_strict({"tech": 1, "bank": 2}, default=None).alias("sector_code"), + (pl.col("custom_mean") > 0).fill_null(False).alias("custom_filter"), + ).filter(pl.col("member").fill_null(False)) + + +def sample() -> pl.DataFrame: + days = [date(2025, 7, day) for day in (7, 8, 9)] + prices = pl.DataFrame( + { + "timestamp": days * 3, + "asset": [17] * 3 + [23] * 3 + [31] * 3, + "close": [10.0, 11.0, 10.0, 10.0, 12.0, 13.0, 10.0, 9.0, 8.0], + "available_at": [datetime(2025, 7, day.day, 20, tzinfo=UTC) for day in days] * 3, + } + ) + decisions = pl.DataFrame( + { + "timestamp": [days[-1]], + "decision_time": [datetime(2025, 7, 10, 13, tzinfo=UTC)], + "effective_session": [date(2025, 7, 10)], + } + ) + universe = pl.DataFrame( + { + "asset": [17, 23, 31], + "effective_from": [date(2025, 7, 10)] * 3, + "effective_to": pl.Series([None] * 3, dtype=pl.Date), + "available_at": [datetime(2025, 7, 1, tzinfo=UTC)] * 3, + "member": [True] * 3, + "sector": ["tech", "bank", None], + } + ) + result = momentum_screen( + prices.lazy(), + decisions, + universe, + feed_spec=FeedSpec(time_contract_version=2, calendar=None, entity_col="asset"), + price_basis="raw", + window_length=3, + n=1, + ) + assert isinstance(result, pl.LazyFrame) + return result.collect() + + +if __name__ == "__main__": + result = sample() + assert result["rsi"].to_list() == [50.0, 100.0, 0.0] + assert result.filter("longs")["asset"].to_list() == [23] + assert result.filter("shorts")["asset"].to_list() == [31] + assert result["sector_code"].to_list() == [1, 2, None] + print(json.dumps(result.to_dicts(), ensure_ascii=True, default=str)) diff --git a/mkdocs.yml b/mkdocs.yml index 827dec05..fbc29947 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -107,6 +107,7 @@ nav: - Quickstart: getting-started/quickstart.md - User Guide: - Features: user-guide/features.md + - Pipeline Migration: user-guide/pipeline-migration.md - Feature Discovery: user-guide/discovery.md - Labeling: user-guide/labeling.md - Alternative Bars: user-guide/bars.md @@ -116,6 +117,7 @@ nav: - ML Readiness: user-guide/ml-readiness.md - API Reference: - api/index.md + - Pipeline Operation Ports: api/pipeline-ports.md - Book Guide: - book-guide/index.md diff --git a/pyproject.toml b/pyproject.toml index b593d2e4..f5baa985 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -112,6 +112,7 @@ calendars = [ # Statistical diagnostics stats = [ + "scipy>=1.14.0", "statsmodels>=0.14.0", ] diff --git a/scripts/check_coverage.py b/scripts/check_coverage.py index 5ff83dc9..aed3ee89 100644 --- a/scripts/check_coverage.py +++ b/scripts/check_coverage.py @@ -38,6 +38,8 @@ "src/ml4t/engineer/core/validation.py", "src/ml4t/engineer/dataset.py", "src/ml4t/engineer/discovery/catalog.py", + "src/ml4t/engineer/features/paired.py", + "src/ml4t/engineer/features/preparation.py", "src/ml4t/engineer/labeling/atr_barriers.py", "src/ml4t/engineer/labeling/calendar.py", "src/ml4t/engineer/labeling/forward_returns.py", diff --git a/src/ml4t/engineer/artifacts/features.py b/src/ml4t/engineer/artifacts/features.py index fbe1b99f..0b0fba2c 100644 --- a/src/ml4t/engineer/artifacts/features.py +++ b/src/ml4t/engineer/artifacts/features.py @@ -5,7 +5,14 @@ from dataclasses import dataclass, field from typing import Any -from ml4t.specs.base import ArtifactKind, ArtifactProvenance, ArtifactSpec, ArtifactStorage +from ml4t.specs import FeedSpec, WindowUnit +from ml4t.specs.base import ( + ArtifactKind, + ArtifactProvenance, + ArtifactSpec, + ArtifactStorage, + optional_str, +) @dataclass(frozen=True, slots=True) @@ -13,8 +20,21 @@ class FeatureSchema: """Column layout for a feature artifact.""" timestamp_col: str = "timestamp" - entity_col: str = "asset" + entity_col: str | tuple[str, ...] = "asset" feature_columns: tuple[str, ...] = () + available_at_col: str | None = None + decision_time_col: str | None = None + eligible_col: str | None = None + universe_time_col: str | None = None + + def __post_init__(self) -> None: + columns = [self.entity_col] if isinstance(self.entity_col, str) else list(self.entity_col) + if not columns or any(not isinstance(c, str) or not c for c in columns): + raise ValueError("entity_col must declare nonempty stable entity columns") + if len(set(columns)) != len(columns): + raise ValueError("entity_col columns must be distinct") + if not isinstance(self.entity_col, str): + object.__setattr__(self, "entity_col", tuple(columns)) @classmethod def from_mapping(cls, mapping: dict[str, Any] | None) -> FeatureSchema: @@ -23,8 +43,12 @@ def from_mapping(cls, mapping: dict[str, Any] | None) -> FeatureSchema: feature_columns = mapping.get("feature_columns", ()) return cls( timestamp_col=str(mapping.get("timestamp_col", "timestamp")), - entity_col=str(mapping.get("entity_col", "asset")), + entity_col=mapping.get("entity_col", "asset"), feature_columns=tuple(str(item) for item in feature_columns), + available_at_col=optional_str(mapping.get("available_at_col")), + decision_time_col=optional_str(mapping.get("decision_time_col")), + eligible_col=optional_str(mapping.get("eligible_col")), + universe_time_col=optional_str(mapping.get("universe_time_col")), ) @@ -35,6 +59,32 @@ class FeatureDefinition: family: str = "financial" join_keys: tuple[str, ...] = () source_artifacts: tuple[str, ...] = () + feed_spec: FeedSpec | None = None + window_unit: WindowUnit | None = None + price_basis: str | None = None + lookbacks: dict[str, int] | None = None + missing_input_policy: str | None = None + + def __post_init__(self) -> None: + if self.feed_spec is not None: + object.__setattr__(self, "feed_spec", FeedSpec.from_any(self.feed_spec)) + if self.window_unit is not None: + object.__setattr__(self, "window_unit", WindowUnit(self.window_unit)) + if self.price_basis is not None and not self.price_basis.strip(): + raise ValueError("price_basis must be a nonempty declaration") + if self.missing_input_policy not in {None, "complete", "known_missing"}: + raise ValueError("missing_input_policy must be complete or known_missing") + if self.lookbacks is not None: + if any( + not isinstance(name, str) + or not name + or isinstance(value, bool) + or not isinstance(value, int) + or value < 1 + for name, value in self.lookbacks.items() + ): + raise ValueError("lookbacks must map feature names to positive integer row counts") + object.__setattr__(self, "lookbacks", dict(self.lookbacks)) @classmethod def from_mapping(cls, mapping: dict[str, Any] | None) -> FeatureDefinition: @@ -46,6 +96,13 @@ def from_mapping(cls, mapping: dict[str, Any] | None) -> FeatureDefinition: family=str(mapping.get("family", "financial")), join_keys=tuple(str(item) for item in join_keys), source_artifacts=tuple(str(item) for item in source_artifacts), + feed_spec=FeedSpec.from_any(mapping["feed_spec"]) + if mapping.get("feed_spec") is not None + else None, + window_unit=mapping.get("window_unit"), + price_basis=optional_str(mapping.get("price_basis")), + lookbacks=mapping.get("lookbacks"), + missing_input_policy=optional_str(mapping.get("missing_input_policy")), ) @@ -57,6 +114,24 @@ class FeatureSpec(ArtifactSpec): schema: FeatureSchema = field(default_factory=FeatureSchema) definition: FeatureDefinition = field(default_factory=FeatureDefinition) + def to_dict(self) -> dict[str, Any]: + payload = ArtifactSpec.to_dict(self) + for name in ("available_at_col", "decision_time_col", "eligible_col", "universe_time_col"): + if payload["schema"][name] is None: + del payload["schema"][name] + for name in ( + "feed_spec", + "window_unit", + "price_basis", + "lookbacks", + "missing_input_policy", + ): + if payload["definition"][name] is None: + del payload["definition"][name] + if self.definition.feed_spec is not None: + payload["definition"]["feed_spec"] = self.definition.feed_spec.to_dict() + return payload + @classmethod def from_mapping(cls, mapping: dict[str, Any]) -> FeatureSpec: return cls( diff --git a/src/ml4t/engineer/compat/__init__.py b/src/ml4t/engineer/compat/__init__.py new file mode 100644 index 00000000..326f3ee5 --- /dev/null +++ b/src/ml4t/engineer/compat/__init__.py @@ -0,0 +1 @@ +"""Explicit migration recipes; no legacy graph runtime or analytical facade.""" diff --git a/src/ml4t/engineer/compat/zipline.py b/src/ml4t/engineer/compat/zipline.py new file mode 100644 index 00000000..b0b98003 --- /dev/null +++ b/src/ml4t/engineer/compat/zipline.py @@ -0,0 +1,305 @@ +"""Zipline factor-name recipes over native expressions, without importing Zipline. + +These return Polars expressions or named output expressions, not Pipeline Terms. +Validate the input feed/grid once, keep historical inputs, then apply output +screens. Dated source eligibility belongs to native feature preparation. +""" + +import inspect +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass +from typing import Any, Literal + +import polars as pl + +from ml4t.engineer.features import events, paired, rolling +from ml4t.engineer.features.utils import arithmetic + +Expression = pl.Expr | dict[str, pl.Expr] + +__all__ = ["factor_expression", "return_factor_features"] + + +def return_factor_features( + data: pl.DataFrame | pl.LazyFrame, + name: str, + *, + target: Mapping[str, Any], + returns_length: int = 2, + correlation_length: int | None = None, + regression_length: int | None = None, + allowed_missing_percentage: float | None = None, + mask: str | pl.Expr | None = None, + **native_options: Any, +) -> pl.DataFrame | pl.LazyFrame: + """Map return-factor constructors to native keyed price preparation. + + target contains stable physical IDs, not an opaque Zipline Asset. Declare + price_basis, feed_spec/calendar and column mappings through native_options. + The mask changes the statistic output only, retaining target input history. + This is a dataframe recipe, not a Pipeline interpreter or legacy constructor. + """ + methods = { + "RollingPearsonOfReturns": "pearson", + "RollingSpearmanOfReturns": "spearman", + "RollingLinearRegressionOfReturns": "regression", + "SimpleBeta": "beta", + } + if name not in methods: + raise ValueError(f"Unsupported return-factor recipe {name!r}") + method = methods[name] + rolling._period(returns_length, minimum=2) + if mask is not None and not isinstance(mask, str | pl.Expr): + raise TypeError("mask must be an input column name or expression") + correlation = method in {"pearson", "spearman"} + width = correlation_length if correlation else regression_length + wrong_length = regression_length if correlation else correlation_length + if width is None or wrong_length is not None: + raise ValueError("Declare only the correlation_length or regression_length for this recipe") + rolling._period(width, minimum=3 if method == "beta" else 2) + allowed_missing = 0 + if method == "beta": + if returns_length != 2: + raise ValueError("SimpleBeta uses daily (two-price) returns") + fraction = 0.25 if allowed_missing_percentage is None else allowed_missing_percentage + if ( + isinstance(fraction, bool) + or not isinstance(fraction, int | float) + or not 0 <= fraction <= 1 + ): + raise ValueError("allowed_missing_percentage must be in [0,1]") + allowed_missing = int(width * fraction) + elif allowed_missing_percentage is not None: + raise ValueError("Only SimpleBeta has allowed_missing_percentage") + output = paired.target_return_features( + data, + target=target, + period=width, + returns_periods=returns_length - 1, + method=method, + allowed_missing=allowed_missing, + **native_options, + ) + if mask is None: + return output + predicate = pl.col(mask) if isinstance(mask, str) else mask + column = f"{native_options.get('output', 'target')}_{method}" + return output.with_columns(pl.when(predicate).then(pl.col(column)).alias(column)) + + +@dataclass(frozen=True) +class _Recipe: + function: Callable[..., Expression] + columns: tuple[str, ...] + window: int | None + required_inputs: bool = False + order: tuple[int, ...] | None = None + window_arg: str | None = "period" + adjustment: int = 0 + + +def _clip(value: str, min_bound: float, max_bound: float) -> pl.Expr: + return pl.col(value).clip(min_bound, max_bound) + + +def _latest(value: str) -> pl.Expr: + return pl.col(value) + + +_RECIPES = { + "Returns": _Recipe(arithmetic.returns, ("close",), None, window_arg="periods", adjustment=-1), + "DailyReturns": _Recipe(arithmetic.returns, ("close",), 2, window_arg="periods", adjustment=-1), + "PercentChange": _Recipe( + rolling.absolute_percent_change, ("close",), None, True, adjustment=-1 + ), + "RateOfChangePercentage": _Recipe( + arithmetic.percentage_change, ("close",), None, True, window_arg="periods", adjustment=-1 + ), + "SimpleMovingAverage": _Recipe(rolling.finite_mean, ("close",), None, True), + "WeightedAverageValue": _Recipe(rolling.finite_weighted_mean, ("close", "volume"), None, True), + "VWAP": _Recipe(rolling.finite_weighted_mean, ("close", "volume"), None), + "AverageDollarVolume": _Recipe(rolling.average_dollar_volume, ("close", "volume"), None), + "LinearWeightedMovingAverage": _Recipe(rolling.linear_weighted_mean, ("close",), None, True), + "MaxDrawdown": _Recipe(rolling.finite_max_drawdown, ("close",), None, True), + "AnnualizedVolatility": _Recipe(rolling.annualized_volatility, ("returns",), 252), + "RSI": _Recipe(rolling.rsi_cutler, ("close",), 15, adjustment=-1), + "BollingerBands": _Recipe(rolling.finite_bollinger_bands, ("close",), None), + "Aroon": _Recipe(rolling.finite_aroon, ("low", "high"), None, order=(1, 0)), + "FastStochasticOscillator": _Recipe( + rolling.fast_stochastic, ("close", "low", "high"), 14, order=(2, 1, 0) + ), + "IchimokuKinkoHyo": _Recipe(rolling.finite_ichimoku, ("high", "low", "close"), 52), + "TrueRange": _Recipe(rolling.finite_true_range, ("high", "low", "close"), 2, window_arg=None), + "Clip": _Recipe(_clip, ("close",), 1, True, window_arg=None), + "Latest": _Recipe(_latest, ("close",), 1, True, window_arg=None), + "RollingPearson": _Recipe(paired.rolling_pearson, ("value", "target"), None, True), + "RollingSpearman": _Recipe(paired.rolling_spearman, ("value", "target"), None, True), + "RollingLinearRegression": _Recipe(paired.rolling_regression, ("value", "target"), None, True), + "SimpleBeta": _Recipe(paired.rolling_beta, ("returns", "target_returns"), None, True), + "ExponentialWeightedMovingAverage": _Recipe(rolling.finite_ewma, ("close",), None, True), + "ExponentialWeightedMovingStdDev": _Recipe(rolling.finite_ewm_std, ("close",), None, True), + "MovingAverageConvergenceDivergenceSignal": _Recipe( + rolling.finite_macd_signal, ("close",), None, window_arg=None + ), +} +_ALIASES = { + "EWMA": "ExponentialWeightedMovingAverage", + "EWMSTD": "ExponentialWeightedMovingStdDev", + "MACDSignal": "MovingAverageConvergenceDivergenceSignal", +} +_FACTORIES = { + "from_span": "span", + "from_halflife": "halflife", + "from_center_of_mass": "center_of_mass", +} +_EVENTS: dict[str, Literal["since", "until"]] = { + "BusinessDaysSincePreviousEvent": "since", + "BusinessDaysUntilNextEvent": "until", +} + + +def _inputs(inputs: Sequence[str] | None, columns: tuple[str, ...], required: bool) -> list[str]: + if inputs is None: + if required: + raise ValueError("This factor requires explicit input column names") + return list(columns) + if ( + not isinstance(inputs, Sequence) + or isinstance(inputs, str) + or len(inputs) != len(columns) + or any(not isinstance(c, str) or not c for c in inputs) + ): + raise ValueError(f"inputs must contain {len(columns)} nonempty column names") + return list(inputs) + + +def factor_expression( + name: str, + *, + inputs: Sequence[str] | None = None, + window_length: int | None = None, + mask: str | pl.Expr | None = None, + **params: Any, +) -> Expression: + """Map a supported factor name to one native calculation and explicit field names. + + window_length counts prices/inputs as in Pipeline; return/RSI recipes translate + N prices to N-1 changes. A mask controls output only and never removes warmup + history. False/null predicate values yield native null outputs; mask=None + leaves outputs unmasked. Factory suffixes support + the three EWMA/EWMSTD constructors. Opaque Terms/Assets are not executed. + """ + if not isinstance(name, str): + raise TypeError("name must identify a supported factor recipe") + pieces = name.split(".") + if len(pieces) > 2: + raise ValueError("Unsupported factor constructor") + canonical = _ALIASES.get(pieces[0], pieces[0]) + factory = pieces[1] if len(pieces) == 2 else None + if factory is not None and ( + factory not in _FACTORIES + or canonical not in {"ExponentialWeightedMovingAverage", "ExponentialWeightedMovingStdDev"} + ): + raise ValueError("Only the declared finite exponential constructors are supported") + if mask is not None and not isinstance(mask, str | pl.Expr): + raise TypeError("mask must be an input column name or expression") + if canonical in _EVENTS: + if isinstance(window_length, bool) or window_length not in (None, 0): + raise ValueError("Event weekday counts do not use rolling windows") + column = _inputs(inputs, ("event_date",), True)[0] + output: Expression = events.event_business_days( + column, direction=_EVENTS[canonical], **params + ) + else: + if canonical not in _RECIPES: + raise ValueError( + f"Unsupported numeric recipe {name!r}; port custom calculations to native expressions/registered features" + ) + route = _RECIPES[canonical] + columns = _inputs(inputs, route.columns, route.required_inputs) + if route.order is not None: + columns = [columns[index] for index in route.order] + options = dict(params) + if canonical.startswith("ExponentialWeightedMoving"): + if factory is not None and _FACTORIES[factory] not in options: + raise ValueError(f"{factory} requires {_FACTORIES[factory]}") + selectors = { + key: options.pop(key) + for key in ("decay_rate", "span", "halflife", "center_of_mass") + if key in options + } + options["decay_rate"] = rolling.exponential_decay(**selectors) + if canonical == "IchimokuKinkoHyo": + for old, new in ( + ("tenkan_sen_length", "tenkan_length"), + ("kijun_sen_length", "kijun_length"), + ("chikou_span_length", "chikou_position"), + ): + if old in options: + if new in options: + raise ValueError(f"Conflicting {old}/{new}") + options[new] = options.pop(old) + lengths = ( + ("correlation_length",) + if canonical in {"RollingPearson", "RollingSpearman"} + else ( + ("regression_length",) + if canonical in {"RollingLinearRegression", "SimpleBeta"} + else () + ) + ) + for legacy in lengths: + if legacy in options: + if window_length is not None: + raise ValueError("Declare one window_length or legacy statistical length") + window_length = options.pop(legacy) + width = window_length if window_length is not None else route.window + if route.window_arg is not None: + if width is None: + raise ValueError("Declare window_length") + rolling._period(width) + if route.window_arg in options: + raise ValueError("Use window_length rather than a second native window argument") + options[route.window_arg] = width + route.adjustment + elif canonical == "MovingAverageConvergenceDivergenceSignal": + if window_length is not None: + raise ValueError("MACDSignal derives its window from the three periods") + else: + assert width is not None + rolling._period(width) + if canonical == "TrueRange" and width != 2: + raise ValueError("TrueRange requires its two-price definition") + if canonical == "SimpleBeta": + assert width is not None + if width < 3: + raise ValueError("SimpleBeta regression_length must be >=3") + fraction = options.pop("allowed_missing_percentage", 0.25) + if ( + isinstance(fraction, bool) + or not isinstance(fraction, int | float) + or not 0 <= fraction <= 1 + ): + raise ValueError("allowed_missing_percentage must be in [0,1]") + if "allowed_missing" in options: + raise ValueError("Declare allowed_missing_percentage only once") + options["allowed_missing"] = int(width * fraction) + signature = inspect.signature(route.function) + column_params = list(signature.parameters)[: len(columns)] + unknown = set(options) - (set(signature.parameters) - set(column_params)) + if unknown: + raise ValueError(f"Unknown factor options: {sorted(unknown)}") + if canonical == "AnnualizedVolatility" and inputs is None: + output = route.function(arithmetic.returns("close"), **options) + assert isinstance(output, pl.Expr) and width is not None + output = rolling._ready(output, width + 1) + else: + output = route.function(*columns, **options) + if canonical in {"Clip", "Latest"}: + assert isinstance(output, pl.Expr) and width is not None + output = rolling._ready(output, width) + if mask is None: + return output + predicate = pl.col(mask) if isinstance(mask, str) else mask + if isinstance(output, dict): + return {key: pl.when(predicate).then(value) for key, value in output.items()} + return pl.when(predicate).then(output) diff --git a/src/ml4t/engineer/core/bar_grid.py b/src/ml4t/engineer/core/bar_grid.py index 31c4d031..78d789b9 100644 --- a/src/ml4t/engineer/core/bar_grid.py +++ b/src/ml4t/engineer/core/bar_grid.py @@ -51,14 +51,17 @@ def validate_declared_grid( if isinstance(frequency, str): import pandas as pd + duration_warning = getattr(pd.errors, "PandasDeprecationWarning", FutureWarning) try: with warnings.catch_warnings(): + # Deprecated pandas unit spellings are invalid declarations; + # unrelated dependency deprecations do not invalidate a duration. warnings.simplefilter("error", FutureWarning) - warnings.simplefilter("error", DeprecationWarning) + warnings.simplefilter("error", duration_warning) parsed = pd.Timedelta("1D" if frequency == "1d" else frequency) - except (ValueError, TypeError, FutureWarning, DeprecationWarning) as exc: + except (ValueError, TypeError, FutureWarning, duration_warning) as exc: raise ValueError("declared grid data_frequency must be a fixed duration") from exc - if not isinstance(parsed, pd.Timedelta) or parsed <= pd.Timedelta(0): + if not isinstance(parsed, pd.Timedelta) or parsed <= timedelta(0): raise ValueError("declared grid data_frequency requires a positive finite duration") frequency = parsed if not isinstance(frequency, timedelta): diff --git a/src/ml4t/engineer/core/lookbacks.py b/src/ml4t/engineer/core/lookbacks.py index 4818e414..c8c1ff1f 100644 --- a/src/ml4t/engineer/core/lookbacks.py +++ b/src/ml4t/engineer/core/lookbacks.py @@ -70,6 +70,23 @@ def _ulcer_index(params: Mapping[str, Any]) -> int: FEATURE_LOOKBACKS: dict[str, LookbackCalculator] = { + "absolute_percent_change": _parameter("period"), + # Explicit finite-window methods, independent of the native TA-Lib defaults. + "rsi_cutler": _parameter("period"), + "finite_mean": _parameter("period", -1), + "finite_weighted_mean": _parameter("period", -1), + "average_dollar_volume": _parameter("period", -1), + "linear_weighted_mean": _parameter("period", -1), + "finite_ewma": _parameter("period", -1), + "finite_ewm_std": _parameter("period", -1), + "annualized_volatility": _parameter("period", -1), + "finite_max_drawdown": _parameter("period", -1), + "finite_bollinger_bands": _parameter("period", -1), + "finite_aroon": _parameter("period", -1), + "fast_stochastic": _parameter("period", -1), + "finite_ichimoku": _parameter("period", -1), + "finite_macd_signal": _sum("slow_period", "signal_period", adjustment=-2), + "finite_true_range": _constant(1), # Volume "ad": _constant(0), "adosc": _parameter("slowperiod", -1), diff --git a/src/ml4t/engineer/features/AGENTS.md b/src/ml4t/engineer/features/AGENTS.md index 6435dea9..d440217c 100644 --- a/src/ml4t/engineer/features/AGENTS.md +++ b/src/ml4t/engineer/features/AGENTS.md @@ -7,17 +7,17 @@ This directory contains the registry-backed indicator implementations used by | Category | Count | Location | Examples | |----------|-------|----------|----------| -| momentum | 31 | `momentum/` | `rsi`, `macd`, `stoch`, `adx`, `mfi` | -| trend | 10 | `trend/` | `sma`, `ema`, `wma`, `kama`, `donchian_channels` | -| volatility | 15 | `volatility/` | `atr`, `bollinger_bands`, `yang_zhang_volatility`, `garch_forecast` | +| momentum | 36 | `momentum/`, `rolling.py` | `rsi`, `rsi_cutler`, `absolute_percent_change`, `finite_macd_signal` | +| trend | 15 | `trend/`, `rolling.py` | `sma`, `ema`, `finite_mean`, `finite_ewma` | +| volatility | 18 | `volatility/`, `rolling.py` | `atr`, `bollinger_bands`, `annualized_volatility` | | microstructure | 15 | `microstructure/` | `kyle_lambda`, `amihud_illiquidity`, `realized_spread` | | ml | 14 | `ml/` plus `fdiff.py` | lag features, entropy, Fourier, percentile-rank features | -| statistics | 14 | `statistics/` | `stddev`, `var`, `linearreg`, `structural_break` | -| volume | 3 | `volume/` | `obv`, `ad`, `adosc` | +| statistics | 15 | `statistics/`, `rolling.py` | `stddev`, `var`, `finite_ewm_std`, `structural_break` | +| volume | 4 | `volume/`, `rolling.py` | `obv`, `ad`, `adosc`, `average_dollar_volume` | | price_transform | 5 | `price_transform/` | `avgprice`, `typprice`, `medprice`, `wclprice`, `midprice` | | math | 3 | `math/` | rolling `max_`, `min_`, `sum_` | | regime | 4 | `regime.py` | `hurst_exponent`, `choppiness_index`, `fractal_efficiency` | -| risk | 6 | `risk.py` | drawdown, downside deviation, tail ratio | +| risk | 7 | `risk.py`, `rolling.py` | drawdown, downside deviation, tail ratio | ## Standalone Helpers @@ -26,6 +26,10 @@ These are not registry entries but remain important: | File | Purpose | |------|---------| | `cross_asset.py` | multi-asset and panel features such as rolling correlation and beta | +| `paired.py` | full-window paired correlation/beta/regression with explicit missing-count limits | +| `events.py` | supplied known event weekday counts, separate from exchange-session counts | +| `selection.py` | eligible contemporary group transforms, stable selectors and boolean windows | +| `preparation.py` | decision-time eligibility over supplied prepared histories and universe facts | | `fdiff.py` | fractional differencing utilities such as `ffdiff()` and `find_optimal_d()` | | `composite.py` | composite indicators and z-score combinations | @@ -70,6 +74,6 @@ durations. Do not resample, infer missing sessions or reinterpret recursive indi ## Trust Signals -- 120 total features across 11 categories +- 136 total features across 11 categories - 60 TA-Lib validated indicators at `1e-6` tolerance - registry metadata supports discovery by category, normalization, and TA-Lib parity diff --git a/src/ml4t/engineer/features/__init__.py b/src/ml4t/engineer/features/__init__.py index 2f77ed86..bf28eebd 100644 --- a/src/ml4t/engineer/features/__init__.py +++ b/src/ml4t/engineer/features/__init__.py @@ -29,14 +29,19 @@ from ml4t.engineer.features import ( composite, cross_asset, + events, fdiff, math, # noqa: F401 microstructure, # noqa: F401 ml, # noqa: F401 momentum, # noqa: F401 + paired, + preparation, price_transform, # noqa: F401 regime, risk, + rolling, + selection, statistics, # noqa: F401 trend, # noqa: F401 volatility, # noqa: F401 @@ -47,7 +52,12 @@ # Advanced feature modules (single files) "composite", "cross_asset", + "events", + "paired", + "preparation", "fdiff", "regime", "risk", + "rolling", + "selection", ] diff --git a/src/ml4t/engineer/features/events.py b/src/ml4t/engineer/features/events.py new file mode 100644 index 00000000..a12c3c12 --- /dev/null +++ b/src/ml4t/engineer/features/events.py @@ -0,0 +1,55 @@ +"""Predictive weekday counts from supplied event dates and publication instants.""" + +from typing import Literal + +import polars as pl + + +def _date(column: str) -> pl.Expr: + def check(value: pl.Expr, dtype: pl.DataType) -> pl.Expr: + if dtype != pl.Date: + raise ValueError( + f"{column!r} must be a civil/session Date, not an inferred instant date" + ) + return value + + return pl.col(column).pipe_with_dtype(check) + + +def _instant_ticks(column: str) -> pl.Expr: + def check(value: pl.Expr, dtype: pl.DataType) -> pl.Expr: + if not isinstance(dtype, pl.Datetime) or dtype.time_zone is None: + raise ValueError( + f"{column!r} must declare a timezone-aware publication/decision instant" + ) + scale = {"ms": 1_000_000, "us": 1_000, "ns": 1}[dtype.time_unit] + # Compare physical instants without truncating ns or overflowing a + # Datetime(ns) conversion for otherwise valid wider-range us/ms dates. + return value.cast(pl.Int64).cast(pl.Int128) * scale + + return pl.col(column).pipe_with_dtype(check) + + +def event_business_days( + event_date_col: str, + *, + reference_date_col: str, + known_at_col: str, + decision_time_col: str, + direction: Literal["since", "until"] = "since", +) -> pl.Expr: + """Weekdays since/until a supplied event known by the decision instant. + + Event/reference columns are civil or session Date labels. Publication and + decision columns are separate timezone-aware instants. Monday through Friday + count, including exchange holidays; the end date is excluded. No exchange + calendar, date provider, latest-event lookup or observation-spacing inference + is used. Missing or not-yet-known event dates yield null. Signed negative + counts remain possible if the caller supplies a date on the opposite side. + """ + if direction not in {"since", "until"}: + raise ValueError("direction must be 'since' or 'until'") + event, reference = _date(event_date_col), _date(reference_date_col) + start, end = (event, reference) if direction == "since" else (reference, event) + eligible = _instant_ticks(known_at_col) <= _instant_ticks(decision_time_col) + return pl.when(eligible).then(pl.business_day_count(start, end).cast(pl.Float64)) diff --git a/src/ml4t/engineer/features/paired.py b/src/ml4t/engineer/features/paired.py new file mode 100644 index 00000000..7ee98c37 --- /dev/null +++ b/src/ml4t/engineer/features/paired.py @@ -0,0 +1,383 @@ +"""Finite paired-observation correlations and regression features. + +Inputs must already be aligned by their actual keys. Apply expressions within +asset groups. Missing pairs occupy their original window positions and are +removed from both sides together, never by compressing the input timeline. +""" + +from collections.abc import Mapping, Sequence +from typing import Any + +import numpy as np +import numpy.typing as npt +import polars as pl +from ml4t.specs import UNSET_CALENDAR, FeedSpec, WindowUnit + +from ml4t.engineer._numba import jit +from ml4t.engineer.features.cross_asset import beta_to_market, rolling_correlation +from ml4t.engineer.features.rolling import _period, _ready +from ml4t.engineer.utils.dependencies import require_dependency + + +def target_return_features( + data: pl.DataFrame | pl.LazyFrame, + *, + target: Mapping[str, Any], + period: int, + price_basis: str, + returns_periods: int = 1, + method: str = "pearson", + feed_spec: FeedSpec | None = None, + timestamp_col: str | None = None, + entity_col: str | list[str] | None = None, + price_col: str | None = None, + calendar: Any = UNSET_CALENDAR, + window_unit: str | WindowUnit | None = None, + index_mode: str = "common_grid", + allowed_missing: int = 0, + missing_price_policy: str = "raise", + available_at_col: str | None = None, + output: str = "target", +) -> pl.DataFrame | pl.LazyFrame: + """Prepare keyed price returns and a rolling statistic against one stable target. + + The default common_grid uses the supplied union clock, retaining absent quotes + through both return derivation and the rolling window. per_asset instead counts + each asset's observations before joining target returns on exact timestamps. + Neither mode fills prices. A named calendar validates the original declared grid + before calculation. calendar=None selects observation rows, not a 24/7 schedule. + + Original columns, keys, order and row count survive. Added columns are + _asset_return, _target_return, _, _available_at and _price_basis. + Availability is unknown unless all price dependencies in the full window have + supplied timezone-aware publication instants. It is not a decision-time screen. + price_basis records a caller declaration; this function does not adjust prices. + """ + from ml4t.engineer.core.bar_grid import validate_declared_grid + from ml4t.engineer.features.events import _instant_ticks + from ml4t.engineer.features.utils.arithmetic import returns + + _validate(period, allowed_missing) + _period(returns_periods) + if period > 2**63 - 1 or returns_periods > 2**63 - 1: + raise ValueError("Window counts must fit signed 64-bit integers") + methods = { + "pearson": rolling_pearson, + "spearman": rolling_spearman, + "beta": rolling_beta, + "regression": rolling_regression, + } + if method not in methods: + raise ValueError(f"method must be one of {tuple(methods)}") + if not isinstance(price_basis, str) or not price_basis.strip(): + raise ValueError("Declare a nonempty price_basis") + if not isinstance(output, str) or not output: + raise ValueError("output must be a nonempty prefix") + if index_mode not in {"common_grid", "per_asset"}: + raise ValueError("index_mode must be common_grid or per_asset") + if missing_price_policy not in {"raise", "null"}: + raise ValueError("missing_price_policy must be raise or null") + if feed_spec is not None and not isinstance(feed_spec, FeedSpec): + raise TypeError("feed_spec must be a canonical FeedSpec instance, not an unresolved config") + contract = FeedSpec(time_contract_version=2) if feed_spec is None else feed_spec + + def bind(explicit: Any, declared: Any, default: Any, role: str) -> Any: + if explicit is not None and feed_spec is not None and declared is not None: + left = [explicit] if isinstance(explicit, str) else list(explicit) + right = [declared] if isinstance(declared, str) else list(declared) + if left != right: + raise ValueError(f"Conflicting {role} and feed_spec declaration") + return explicit if explicit is not None else declared if declared is not None else default + + timestamp = bind(timestamp_col, contract.timestamp_col, "timestamp", "timestamp_col") + identity = bind(entity_col, contract.entity_col, "asset", "entity_col") + entities = [identity] if isinstance(identity, str) else list(identity) + price = bind(price_col, contract.price_col, "close", "price_col") + if not entities or len(set(entities)) != len(entities): + raise ValueError("Declare distinct entity columns") + if not isinstance(target, Mapping) or set(target) != set(entities): + raise ValueError("target must supply exactly the declared stable entity columns") + if any(value is None for value in target.values()): + raise ValueError("target identity cannot be null") + if len({timestamp, *entities, price}) != len(entities) + 2: + raise ValueError("Timestamp, entity and price roles must be distinct") + resolved_calendar = contract.resolve_calendar(calendar=calendar) + unit = ( + WindowUnit(window_unit) + if window_unit is not None + else (WindowUnit.OBSERVATION if resolved_calendar is None else WindowUnit.EXPECTED_BAR) + ) + expected_unit = WindowUnit.OBSERVATION if resolved_calendar is None else WindowUnit.EXPECTED_BAR + if unit != expected_unit: + raise ValueError(f"This finite-row statistic requires window_unit={expected_unit.value!r}") + schema = data.collect_schema() if isinstance(data, pl.LazyFrame) else data.schema + required = [timestamp, *entities, price, *([available_at_col] if available_at_col else [])] + if set(required) - set(schema): + raise ValueError(f"Missing input columns: {sorted(set(required) - set(schema))}") + if schema[timestamp].base_type() not in {pl.Date, pl.Datetime}: + raise ValueError("timestamp_col must contain Date labels or Datetime instants") + if not schema[price].is_numeric(): + raise ValueError("price_col must be numeric") + names = [ + f"{output}_{name}" + for name in ("asset_return", "target_return", method, "available_at", "price_basis") + ] + if set(names) & set(schema) or any(c.startswith("__ml4t_target_") for c in schema): + raise ValueError("Output or reserved preparation columns already exist") + result_schema = dict(schema) + result_schema.update(dict.fromkeys(names[:2], pl.Float64())) + result_schema[names[2]] = _REGRESSION_DTYPE if method == "regression" else pl.Float64() + result_schema[names[3]] = pl.Datetime("ns", "UTC") + result_schema[names[4]] = pl.String() + keys = [*entities, timestamp] + prefix = "__ml4t_target_" + quote, known, pair_known, row_id = (prefix + s for s in ("quote", "known", "pair", "row")) + + def calculate(frame: pl.DataFrame) -> pl.DataFrame: + if frame.is_empty(): + return frame.with_columns( + [pl.lit(None, dtype=result_schema[c]).alias(c) for c in names] + ) + if frame.select(pl.any_horizontal(pl.col(c).is_null() for c in keys).any()).item(): + raise ValueError("Timestamp and entity keys cannot be null") + if frame.select(pl.struct(keys).is_duplicated().any()).item(): + raise ValueError("Duplicate timestamp/entity keys") + predicate = pl.all_horizontal( + pl.col(c) == pl.lit(target[c], dtype=schema[c]) for c in entities + ) + if frame.filter(predicate).is_empty(): + raise ValueError("target identity is absent from supplied input") + valid_price = (pl.col(price) > 0) & pl.col(price).cast(pl.Float64).is_finite() + if missing_price_policy == "raise" and frame.filter(~valid_price.fill_null(False)).height: + raise ValueError("Prices must be finite, positive and nonmissing") + if resolved_calendar is not None: + validate_declared_grid( + frame, + feed_spec=contract, + calendar=resolved_calendar, + timestamp_col=timestamp, + group_cols=entities, + ) + known_expr = ( + _instant_ticks(available_at_col) if available_at_col else pl.lit(None, pl.Int128) + ) + prepared = frame.select( + *keys, pl.when(valid_price).then(pl.col(price)).alias(quote), known_expr.alias(known) + ) + if ( + available_at_col + and prepared.filter((pl.col(known) < -(2**63)) | (pl.col(known) > 2**63 - 1)).height + ): + raise ValueError("Publication instants exceed Datetime(ns) range") + if index_mode == "common_grid": + grid = ( + frame.select(entities).unique().join(frame.select(timestamp).unique(), how="cross") + ) + prepared = grid.join(prepared, on=keys, how="left", validate="1:1") + prepared = prepared.sort(keys) + lag = min(returns_periods, prepared.height) + return_known = pl.when( + pl.col(known).is_not_null() & pl.col(known).shift(lag).is_not_null() + ).then(pl.max_horizontal(pl.col(known), pl.col(known).shift(lag))) + prepared = prepared.with_columns( + returns(quote, lag).cast(pl.Float64).over(entities).alias(names[0]), + return_known.over(entities).alias(known), + ) + benchmark = prepared.filter(predicate).select( + timestamp, pl.col(names[0]).alias(names[1]), pl.col(known).alias(pair_known) + ) + prepared = prepared.join(benchmark, on=timestamp, how="left", validate="m:1") + prepared = prepared.with_columns( + pl.when(pl.col(known).is_not_null() & pl.col(pair_known).is_not_null()) + .then(pl.max_horizontal(known, pair_known)) + .cast(pl.Int64) + .alias(pair_known) + ) + statistic = methods[method](names[0], names[1], period, allowed_missing=allowed_missing) + warmup = returns_periods + period + availability = ( + pl.when(pl.col(pair_known).is_not_null().cast(pl.Int64).rolling_sum(period) == period) + .then(pl.col(pair_known).rolling_max(period)) + .cast(pl.Datetime("ns", "UTC")) + ) + prepared = prepared.with_columns( + _ready(statistic, warmup).over(entities).alias(names[2]), + _ready(availability, warmup).over(entities).alias(names[3]), + pl.lit(price_basis).alias(names[4]), + ) + return ( + frame.with_row_index(row_id) + .join(prepared.select(*keys, *names), on=keys, how="left", validate="1:1") + .sort(row_id) + .drop(row_id) + ) + + if isinstance(data, pl.LazyFrame): + return data.map_batches( + calculate, + schema=result_schema, + predicate_pushdown=False, + projection_pushdown=False, + slice_pushdown=False, + streamable=False, + ) + return calculate(data) + + +def _validate(period: int, allowed_missing: int) -> None: + _period(period, minimum=2) + if ( + isinstance(allowed_missing, bool) + or not isinstance(allowed_missing, int) + or not 0 <= allowed_missing <= period + ): + raise ValueError("allowed_missing must be an integer in [0, period]") + + +def _expressions( + dependent: str | pl.Expr, independent: str | pl.Expr +) -> tuple[pl.Expr, pl.Expr, pl.Expr]: + y = pl.col(dependent) if isinstance(dependent, str) else dependent + x = pl.col(independent) if isinstance(independent, str) else independent + y, x = y.cast(pl.Float64), x.cast(pl.Float64) + missing = y.is_null() | x.is_null() | y.is_nan() | x.is_nan() + return pl.when(~missing).then(y), pl.when(~missing).then(x), y.is_infinite() | x.is_infinite() + + +def rolling_pearson( + dependent: str | pl.Expr, + independent: str | pl.Expr, + period: int = 20, + *, + allowed_missing: int = 0, +) -> pl.Expr: + """Pearson correlation over a full row window with a paired missing-count limit.""" + _validate(period, allowed_missing) + y, x, infinite = _expressions(dependent, independent) + result = rolling_correlation(y, x, window=period, min_periods=max(2, period - allowed_missing)) + return _ready( + pl.when(infinite.fill_null(False).rolling_max(period)).then(float("nan")).otherwise(result), + period, + ) + + +def rolling_beta( + dependent: str | pl.Expr, + independent: str | pl.Expr, + period: int = 60, + *, + allowed_missing: int = 0, +) -> pl.Expr: + """Slope Cov(independent,dependent)/Var(independent) using identical paired rows.""" + _validate(period, allowed_missing) + y, x, infinite = _expressions(dependent, independent) + result = beta_to_market(y, x, window=period, min_periods=max(2, period - allowed_missing)) + return _ready( + pl.when(infinite.fill_null(False).rolling_max(period)).then(float("nan")).otherwise(result), + period, + ) + + +@jit(nopython=True, cache=True) # type: ignore[misc] +def _ranks(values: npt.NDArray[np.float64]) -> npt.NDArray[np.float64]: + order = np.argsort(values) + result = np.empty(len(values)) + start = 0 + while start < len(order): + stop = start + 1 + while stop < len(order) and values[order[stop]] == values[order[start]]: + stop += 1 + rank = (start + 1 + stop) / 2 + for j in range(start, stop): + result[order[j]] = rank + start = stop + return result + + +@jit(nopython=True, cache=True) # type: ignore[misc] +def _spearman_windows( + y: npt.NDArray[np.float64], x: npt.NDArray[np.float64], period: int, allowed_missing: int +) -> npt.NDArray[np.float64]: + result = np.full(len(y), np.nan) + for end in range(period - 1, len(y)): + yy, xx = y[end - period + 1 : end + 1], x[end - period + 1 : end + 1] + valid = ~np.isnan(yy) & ~np.isnan(xx) + count = valid.sum() + if count < 2 or period - count > allowed_missing: + continue + yr, xr = _ranks(yy[valid]), _ranks(xx[valid]) + yr, xr = yr - yr.mean(), xr - xr.mean() + denominator = np.sqrt((yr * yr).sum() * (xr * xr).sum()) + if denominator > 0: + result[end] = (yr * xr).sum() / denominator + return result + + +def rolling_spearman( + dependent: str | pl.Expr, + independent: str | pl.Expr, + period: int = 20, + *, + allowed_missing: int = 0, +) -> pl.Expr: + """Spearman correlation with average ties recomputed inside each paired window.""" + _validate(period, allowed_missing) + y, x, _ = _expressions(dependent, independent) + return pl.map_batches( + [y, x], + lambda values: pl.Series( + _spearman_windows(values[0].to_numpy(), values[1].to_numpy(), period, allowed_missing) + ), + return_dtype=pl.Float64, + ) + + +_REGRESSION_FIELDS = ("alpha", "beta", "r_value", "p_value", "stderr") +_REGRESSION_DTYPE = pl.Struct(dict.fromkeys(_REGRESSION_FIELDS, pl.Float64)) + + +def rolling_regression( + dependent: str | pl.Expr, + independent: str | pl.Expr, + period: int = 20, + *, + allowed_missing: int = 0, +) -> pl.Expr: + """OLS of dependent on independent, returning a five-field Struct expression. + + Requires the stats extra for SciPy's inferential outputs. Constant independent + windows are undefined and yield NaN fields instead of failing the whole panel. + Nonfinite nonmissing inputs propagate undefined fields. No trend-on-time or + portfolio attribution is substituted for the supplied independent observations. + """ + _validate(period, allowed_missing) + require_dependency("scipy", "rolling regression inferential fields") + from scipy.stats import linregress + + y, x, _ = _expressions(dependent, independent) + + def compute(values: Sequence[pl.Series]) -> pl.Series: + yy, xx = values[0].to_numpy(), values[1].to_numpy() + output = np.full((len(yy), 5), np.nan) + for end in range(period - 1, len(yy)): + ys, xs = yy[end - period + 1 : end + 1], xx[end - period + 1 : end + 1] + valid = ~np.isnan(ys) & ~np.isnan(xs) + count = np.count_nonzero(valid) + if count < 2 or period - count > allowed_missing: + continue + ys, xs = ys[valid], xs[valid] + if not np.isfinite(ys).all() or not np.isfinite(xs).all() or np.ptp(xs) == 0: + continue + result = linregress(xs, ys) + output[end] = ( + result.intercept, + result.slope, + result.rvalue, + result.pvalue, + result.stderr, + ) + return pl.DataFrame( + {name: output[:, i] for i, name in enumerate(_REGRESSION_FIELDS)} + ).to_struct() + + return pl.map_batches([y, x], compute, return_dtype=_REGRESSION_DTYPE) diff --git a/src/ml4t/engineer/features/preparation.py b/src/ml4t/engineer/features/preparation.py new file mode 100644 index 00000000..abdb846b --- /dev/null +++ b/src/ml4t/engineer/features/preparation.py @@ -0,0 +1,461 @@ +"""Decision-time feature eligibility over caller/Data-prepared native histories.""" + +from collections.abc import Mapping, Sequence +from typing import Any, Literal + +import polars as pl +from ml4t.specs import UNSET_CALENDAR, FeedSpec, TimestampSemantics, WindowUnit + +from ml4t.engineer.core.bar_grid import validate_declared_grid +from ml4t.engineer.features.events import _instant_ticks +from ml4t.engineer.features.rolling import _period + + +def _usable_value(value: pl.Expr, dtype: pl.DataType) -> pl.Expr: + valid = value.is_not_null() + if dtype.is_float(): + valid &= value.is_finite() + elif isinstance(dtype, pl.Struct): + valid &= pl.all_horizontal( + _usable_value( + value.struct.field(field.name), + field.dtype if isinstance(field.dtype, pl.DataType) else field.dtype(), + ) + for field in dtype.fields + ) + return valid + + +def prepare_feature_panel( + data: pl.DataFrame | pl.LazyFrame, + *, + features: Mapping[str, pl.Expr], + lookbacks: Mapping[str, int], + input_columns: Sequence[str], + decisions: pl.DataFrame | pl.LazyFrame, + universe: pl.DataFrame | pl.LazyFrame, + price_basis: str, + feed_spec: FeedSpec | None = None, + timestamp_col: str | None = None, + entity_col: str | list[str] | None = None, + calendar: Any = UNSET_CALENDAR, + window_unit: str | WindowUnit | None = None, + available_at_col: str = "available_at", + decision_time_col: str = "decision_time", + universe_time_col: str = "effective_session", + history_decision_time_col: str | None = None, + missing_input_policy: Literal["complete", "known_missing"] = "complete", + input_mask: str | pl.Expr | None = None, + screen: str | pl.Expr | None = None, +) -> pl.DataFrame | pl.LazyFrame: + """Calculate finite per-asset features with explicit causal eligibility. + + decisions supplies an observation cutoff, an aware decision instant and the + universe's decision-effective key (effective_session by default). These need + not identify the same session. universe_time_col can select an explicit + instant column for instant-effective facts; no session is inferred from dates. + universe supplies stable IDs, effective_from/inclusive, effective_to/exclusive + (null means open), available_at and nullable boolean member, plus optional + classifications. Known active overlaps are ambiguous and raise. Missing facts + never imply membership. Pre-entry price history is retained for warmup. + + Data/callers select revisions and adjustment perspective before this boundary. + For different prepared histories at different decisions, label those snapshots + with history_decision_time_col; no snapshot/revision is implicitly selected or + carried forward. The declared price_basis is retained, not calculated. + + Supply bare per-asset expressions, their finite row lookbacks, and every input + column they require. Full required windows must contain known, usable inputs + by default. known_missing permits confirmed null/NaN or input-masked values + to follow the supplied kernel's missing-value definition. All original rows + must still have known publication times; infinite inputs remain invalid. + Unavailable/masked rows stay in history with null inputs, never compressed or + filled. A final screen runs only after all values and eligibility are computed. + Cross-sectional transforms operate on the resulting eligible population. + """ + if not features or set(features) != set(lookbacks): + raise ValueError("Declare the same nonempty feature names and lookbacks") + if any( + not isinstance(name, str) or not name or not isinstance(expr, pl.Expr) + for name, expr in features.items() + ): + raise ValueError("features must map nonempty names to native expressions") + for count in lookbacks.values(): + _period(count) + if count > 2**63 - 1: + raise ValueError("Feature lookbacks must fit signed 64-bit row counts") + if ( + isinstance(input_columns, str) + or not input_columns + or len(set(input_columns)) != len(input_columns) + ): + raise ValueError("Declare distinct input_columns for the supplied expressions") + if not isinstance(price_basis, str) or not price_basis.strip(): + raise ValueError("Declare a nonempty price_basis") + if missing_input_policy not in {"complete", "known_missing"}: + raise ValueError("missing_input_policy must be complete or known_missing") + if any( + value is not None and not isinstance(value, str | pl.Expr) for value in (input_mask, screen) + ): + raise TypeError("input_mask and screen must be column names or expressions") + if feed_spec is not None and not isinstance(feed_spec, FeedSpec): + raise TypeError("feed_spec must be a canonical FeedSpec instance, not an unresolved config") + contract = FeedSpec(time_contract_version=2) if feed_spec is None else feed_spec + for name, explicit, declared in ( + ("timestamp_col", timestamp_col, contract.timestamp_col), + ("entity_col", entity_col, contract.entity_col), + ): + if ( + explicit is not None + and declared is not None + and feed_spec is not None + and ([explicit] if isinstance(explicit, str) else list(explicit)) + != ([declared] if isinstance(declared, str) else list(declared)) + ): + raise ValueError(f"Conflicting {name} and feed_spec declaration") + timestamp = timestamp_col or contract.timestamp_col + identity = entity_col if entity_col is not None else contract.entity_col + entities = ( + [identity] + if isinstance(identity, str) + else list(identity) + if identity is not None + else ["asset"] + ) + if not entities or len(set(entities)) != len(entities): + raise ValueError("Declare distinct stable entity columns") + keys = [timestamp, *entities] + if len(set(keys)) != len(keys) or set(input_columns) & set(keys): + raise ValueError("Input columns and timestamp/entity roles must be distinct") + resolved = contract.resolve_calendar(calendar=calendar) + unit = ( + WindowUnit(window_unit) + if window_unit is not None + else (WindowUnit.OBSERVATION if resolved is None else WindowUnit.EXPECTED_BAR) + ) + if unit != (WindowUnit.OBSERVATION if resolved is None else WindowUnit.EXPECTED_BAR): + raise ValueError( + "Finite feature preparation requires observation or declared expected_bar rows" + ) + decision_frame = ( + decisions.collect() if isinstance(decisions, pl.LazyFrame) else decisions.clone() + ) + universe_frame = universe.collect() if isinstance(universe, pl.LazyFrame) else universe.clone() + schema = data.collect_schema() if isinstance(data, pl.LazyFrame) else data.schema + for label, columns, required in ( + ( + "history", + schema, + [ + *keys, + *input_columns, + available_at_col, + *([history_decision_time_col] if history_decision_time_col else []), + ], + ), + ("decisions", decision_frame.schema, [timestamp, decision_time_col, universe_time_col]), + ( + "universe", + universe_frame.schema, + [*entities, "effective_from", "effective_to", available_at_col, "member"], + ), + ): + if set(required) - set(columns): + raise ValueError(f"Missing {label} columns: {sorted(set(required) - set(columns))}") + if schema[timestamp].base_type() not in {pl.Date, pl.Datetime}: + raise ValueError("Observation keys require Date labels or declared Datetime instants") + if resolved is not None: + validate_declared_grid( + pl.DataFrame(schema=schema), + feed_spec=contract, + calendar=resolved, + timestamp_col=timestamp, + group_cols=entities, + ) + if schema[timestamp] != decision_frame.schema[timestamp]: + raise ValueError("Decision observation cutoffs must preserve the input timestamp dtype") + universe_dtype = decision_frame.schema[universe_time_col] + if universe_frame.schema["effective_to"] == pl.Null: + universe_frame = universe_frame.with_columns(pl.col("effective_to").cast(universe_dtype)) + if universe_dtype.base_type() not in {pl.Date, pl.Datetime} or any( + universe_frame.schema[c].base_type() != universe_dtype.base_type() + for c in ("effective_from", "effective_to") + ): + raise ValueError( + "Universe effective intervals must match the declared decision-effective clock" + ) + if universe_frame.schema["member"] != pl.Boolean: + raise ValueError("Universe member must be nullable boolean, not an inferred classifier") + roles = [*keys, decision_time_col, available_at_col, "member", "eligible", "price_basis"] + if len(set(roles)) != len(roles) or set(features) & set(roles): + raise ValueError("Feature and output roles collide") + if universe_time_col in { + *entities, + available_at_col, + "member", + "eligible", + "price_basis", + *features, + }: + raise ValueError("The universe clock collides with a different output role") + if universe_time_col not in roles: + roles.append(universe_time_col) + extra = [ + c + for c in universe_frame.columns + if c not in [*entities, "effective_from", "effective_to", available_at_col, "member"] + ] + if set(extra) & (set(roles) | set(features)): + raise ValueError("Universe classification names collide with feature/output roles") + prefix = "__ml4t_feature_" + if any( + c.startswith(prefix) + for c in [*schema, *universe_frame.columns, *decision_frame.columns, *features] + ): + raise ValueError("Input columns collide with reserved feature preparation names") + known, cutoff, source_usable, source_ready, feature_known = ( + prefix + c for c in ("known", "cutoff", "usable", "ready", "time") + ) + + def clock( + frame: pl.DataFrame, + column: str, + output: str, + *, + required: bool = False, + range_guard: bool = True, + ) -> pl.DataFrame: + expr = ( + pl.lit(None, pl.Int128) if frame.schema[column] == pl.Null else _instant_ticks(column) + ) + result = frame.with_columns(expr.alias(output)) + if required and result[output].null_count(): + raise ValueError(f"{column} must contain known decision/snapshot instants") + if ( + range_guard + and result.filter((pl.col(output) < -(2**63)) | (pl.col(output) > 2**63 - 1)).height + ): + raise ValueError(f"{column} exceeds Datetime(ns) output range") + return result + + decision_frame = clock(decision_frame, decision_time_col, cutoff, required=True) + timestamp_dtype = schema[timestamp] + if ( + isinstance(timestamp_dtype, pl.Datetime) + and contract.timestamp_semantics != TimestampSemantics.SESSION_LABEL + ): + observation = pl.col(timestamp) + if timestamp_dtype.time_zone is None: + if contract.timezone is None: + raise ValueError("Naive observation instants require a declared timezone") + observation = observation.dt.replace_time_zone(contract.timezone) + decision_frame = decision_frame.with_columns(observation.alias(prefix + "observation")) + decision_frame = clock( + decision_frame, prefix + "observation", prefix + "observation_ticks", required=True + ) + if decision_frame.filter(pl.col(prefix + "observation_ticks") > pl.col(cutoff)).height: + raise ValueError("Observation cutoff cannot follow its decision instant") + universe_frame = clock(universe_frame, available_at_col, known) + universe_from, universe_to, universe_target = ( + "effective_from", + "effective_to", + universe_time_col, + ) + if isinstance(universe_dtype, pl.Datetime): + + def effective_clock(frame: pl.DataFrame, column: str, target: str) -> pl.DataFrame: + dtype = frame.schema[column] + assert isinstance(dtype, pl.Datetime) + value = pl.col(column) + if dtype.time_zone is None: + if contract.timezone is None: + raise ValueError("Naive universe instants require a declared timezone") + value = value.dt.replace_time_zone(contract.timezone) + temporary = target + "_aware" + return clock( + frame.with_columns(value.alias(temporary)), temporary, target, range_guard=False + ).drop(temporary) + + universe_from, universe_to, universe_target = ( + prefix + name for name in ("from", "to", "target") + ) + universe_frame = effective_clock(universe_frame, "effective_from", universe_from) + universe_frame = effective_clock(universe_frame, "effective_to", universe_to) + decision_frame = effective_clock(decision_frame, universe_time_col, universe_target) + if decision_frame[universe_target].null_count(): + raise ValueError("Decision-effective universe keys cannot be unknown") + if ( + decision_frame[timestamp].null_count() + or decision_frame.select( + pl.struct(list(dict.fromkeys([timestamp, cutoff, universe_target]))) + .is_duplicated() + .any() + ).item() + ): + raise ValueError("Decision observation/instant keys must be nonnull and unique") + if universe_frame.select( + pl.any_horizontal(pl.col(c).is_null() for c in [*entities, "effective_from"]).any() + ).item(): + raise ValueError("Universe identity/effective start cannot be unknown") + if universe_frame.filter(pl.col(universe_to) <= pl.col(universe_from)).height: + raise ValueError("Universe effective intervals must be nonempty and end-exclusive") + template = ( + pl.DataFrame(schema=schema) + .select(*keys, *input_columns) + .lazy() + .with_columns(expr.over(entities).alias(name) for name, expr in features.items()) + .collect_schema() + ) + result_schema = { + timestamp: schema[timestamp], + **{c: schema[c] for c in entities}, + decision_time_col: pl.Datetime("ns", "UTC"), + **{name: template[name] for name in features}, + available_at_col: pl.Datetime("ns", "UTC"), + "member": pl.Boolean, + "eligible": pl.Boolean, + "price_basis": pl.String, + **{c: universe_frame.schema[c] for c in extra}, + } + if universe_time_col not in result_schema: + result_schema[universe_time_col] = decision_frame.schema[universe_time_col] + + def calculate(frame: pl.DataFrame) -> pl.DataFrame: + source = clock(frame, available_at_col, known) + unique_keys = [*keys] + if history_decision_time_col: + source = clock(source, history_decision_time_col, prefix + "snapshot", required=True) + unique_keys.append(prefix + "snapshot") + if source.select(pl.any_horizontal(pl.col(c).is_null() for c in keys).any()).item(): + raise ValueError("History timestamp/entity keys cannot be unknown") + if source.select(pl.struct(unique_keys).is_duplicated().any()).item(): + raise ValueError( + "Duplicate history keys; select source revisions/perspectives upstream" + ) + output = [] + for i in range(decision_frame.height): + decision = decision_frame.slice(i, 1) + bound = pl.lit(decision[timestamp]).first() + instant = decision[cutoff].item() + effective = ( + pl.lit(decision[universe_target].item(), dtype=pl.Int128) + if isinstance(universe_dtype, pl.Datetime) + else pl.lit(decision[universe_time_col]).first() + ) + history = source + if history_decision_time_col: + history = history.filter(pl.col(prefix + "snapshot") == instant) + if history.is_empty(): + raise ValueError("Missing prepared history snapshot for a requested decision") + history = history.filter(pl.col(timestamp) <= bound).sort([*entities, timestamp]) + facts = universe_frame.filter( + (pl.col(universe_from) <= effective) + & (pl.col(universe_to).is_null() | (pl.col(universe_to) > effective)) + & (pl.col(known) <= instant) + ) + if facts.select(pl.struct(entities).is_duplicated().any()).item(): + raise ValueError("Ambiguous overlapping known universe facts; resolve upstream") + if resolved is not None and not history.is_empty(): + validate_declared_grid( + history, + feed_spec=contract, + calendar=resolved, + timestamp_col=timestamp, + group_cols=entities, + ) + usable = pl.col(known).is_not_null() & (pl.col(known) <= instant) + confirmed = usable + for column in input_columns: + value = pl.col(column) + valid = _usable_value(value, schema[column]) + missing = value.is_null() + if schema[column].is_float(): + missing |= value.is_nan() + confirmed &= valid | missing + usable &= valid + if input_mask is not None: + usable &= ( + pl.col(input_mask) if isinstance(input_mask, str) else input_mask + ).fill_null(False) + history = history.with_columns( + usable.alias(source_usable), + (usable if missing_input_policy == "complete" else confirmed).alias(source_ready), + ) + assets = pl.concat( + [history.filter(pl.col(known) <= instant).select(entities), facts.select(entities)] + ).unique() + history = history.with_columns( + pl.when(pl.col(source_usable)).then(pl.col(c)).alias(c) for c in input_columns + ) + available = [] + for name in features: + count = lookbacks[name] + full = pl.col(source_ready).cast(pl.Int64).rolling_sum(count) == count + available.append( + pl.when(full) + .then(pl.col(known).cast(pl.Int64).rolling_max(count)) + .over(entities) + .alias(prefix + name) + ) + history = ( + history.with_columns(available) + .select(*keys, *input_columns, *[prefix + name for name in features]) + .with_columns(expr.over(entities).alias(name) for name, expr in features.items()) + ) + history = history.with_columns( + pl.when(pl.col(prefix + name).is_not_null()).then(pl.col(name)).alias(name) + for name in features + ) + knowledge = pl.when( + pl.all_horizontal(pl.col(prefix + name).is_not_null() for name in features) + ).then(pl.max_horizontal(pl.col(prefix + name) for name in features)) + latest = history.filter(pl.col(timestamp) == bound).select( + *entities, *features, knowledge.alias(feature_known) + ) + result = assets.join(latest, on=entities, how="left", validate="1:1").join( + facts.select( + *entities, "member", *extra, pl.col(known).cast(pl.Int64).alias(known) + ), + on=entities, + how="left", + validate="1:1", + ) + result = result.with_columns( + bound.alias(timestamp), + pl.lit(instant).cast(pl.Datetime("ns", "UTC")).alias(decision_time_col), + pl.when(pl.col(feature_known).is_not_null()) + .then(pl.max_horizontal(feature_known, known)) + .cast(pl.Datetime("ns", "UTC")) + .alias(available_at_col), + ( + pl.col("member").fill_null(False) + & pl.col(feature_known).is_not_null() + & pl.all_horizontal( + _usable_value(pl.col(name), template[name]) for name in features + ) + ).alias("eligible"), + pl.lit(price_basis).alias("price_basis"), + ) + if universe_time_col not in {*keys, decision_time_col}: + result = result.with_columns( + pl.lit(decision[universe_time_col]).first().alias(universe_time_col) + ) + output.append(result.select(*result_schema)) + result = pl.concat(output) if output else pl.DataFrame(schema=result_schema) + result = result.sort( + list(dict.fromkeys([decision_time_col, timestamp, universe_time_col, *entities])) + ) + if screen is not None: + result = result.filter(pl.col(screen) if isinstance(screen, str) else screen) + return result + + if isinstance(data, pl.LazyFrame): + return data.map_batches( + calculate, + schema=result_schema, + predicate_pushdown=False, + projection_pushdown=False, + slice_pushdown=False, + streamable=False, + ) + return calculate(data) diff --git a/src/ml4t/engineer/features/rolling.py b/src/ml4t/engineer/features/rolling.py new file mode 100644 index 00000000..de2b92fe --- /dev/null +++ b/src/ml4t/engineer/features/rolling.py @@ -0,0 +1,621 @@ +"""Finite observation-window methods with explicit missing-value definitions. + +These methods are independent of the recursively seeded TA-Lib indicators. +Window sizes count input observations, including missing observations. Expressions +must be applied within the caller's asset groups, as compute_features does. +""" + +import math +from numbers import Real +from typing import Any + +import numpy as np +import numpy.typing as npt +import polars as pl + +from ml4t.engineer._numba import jit +from ml4t.engineer.core.decorators import feature + + +def _period(period: int, *, minimum: int = 1) -> int: + if isinstance(period, bool) or not isinstance(period, int) or period < minimum: + raise ValueError(f"period must be an integer >= {minimum}, got {period!r}") + return period + + +def _ready(expression: pl.Expr, period: int) -> pl.Expr: + return pl.when(pl.int_range(pl.len()) >= period - 1).then(expression) + + +def _numeric(column: str) -> pl.Expr: + return pl.col(column).cast(pl.Float64).fill_nan(None) + + +def _difference(price: pl.Expr, reference: pl.Expr, dtype: pl.DataType) -> pl.Expr: + if dtype.is_integer(): + price, reference = price.cast(pl.Int128), reference.cast(pl.Int128) + if dtype == pl.Int128 or isinstance(dtype, pl.Decimal): + same_sign = ((price >= 0) & (reference >= 0)) | ((price <= 0) & (reference <= 0)) + safe_price = pl.when(same_sign).then(price).otherwise(0) + safe_reference = pl.when(same_sign).then(reference).otherwise(0) + exact = (safe_price - safe_reference).cast(pl.Float64) + return ( + pl.when(same_sign) + .then(exact) + .otherwise(price.cast(pl.Float64) - reference.cast(pl.Float64)) + ) + if dtype.is_integer(): + return (price - reference).cast(pl.Float64) + return price.cast(pl.Float64) - reference.cast(pl.Float64) + + +def _paired_difference(left: pl.Expr, right: pl.Expr) -> pl.Expr: + def left_type(value: pl.Expr, dtype: pl.DataType) -> pl.Expr: + def right_type(reference: pl.Expr, reference_dtype: pl.DataType) -> pl.Expr: + if dtype.is_integer() and reference_dtype.is_integer(): + return _difference(value, reference, pl.Int128()) + if isinstance(dtype, pl.Decimal) and isinstance(reference_dtype, pl.Decimal): + return _difference(value, reference, dtype) + return value.cast(pl.Float64) - reference.cast(pl.Float64) + + return right.pipe_with_dtype(right_type) + + return left.pipe_with_dtype(left_type) + + +@feature( + name="absolute_percent_change", + category="momentum", + description="Fractional change with absolute reference denominator", + normalized=True, + value_range=(float("-inf"), float("inf")), + formula="(value - previous) / abs(previous)", +) +def absolute_percent_change(close: str, period: int = 1) -> pl.Expr: + """Fractional change of signed values; not percentage points or signed-denominator returns.""" + _period(period) + value = pl.col(close) + previous = value.shift(period) + return _paired_difference(value, previous) / previous.cast(pl.Float64).abs() + + +def exponential_decay( + *, + decay_rate: float | None = None, + span: float | None = None, + halflife: float | None = None, + center_of_mass: float | None = None, +) -> float: + """Resolve exactly one finite-weight selector, with no implicit recursive-EMA substitution.""" + choices = { + "decay_rate": decay_rate, + "span": span, + "halflife": halflife, + "center_of_mass": center_of_mass, + } + supplied = [(name, value) for name, value in choices.items() if value is not None] + if len(supplied) != 1: + raise ValueError("Declare exactly one of decay_rate, span, halflife, center_of_mass") + name, value = supplied[0] + if isinstance(value, bool) or not isinstance(value, Real): + raise ValueError(f"{name} must be a finite real scalar") + value = float(value) + if not math.isfinite(value): + raise ValueError(f"{name} must be a finite real scalar") + if name == "span": + if value <= 1: + raise ValueError("span must be > 1") + value = 1 - 2 / (1 + value) + elif name == "halflife": + if value <= 0: + raise ValueError("halflife must be positive") + value = math.exp(math.log(0.5) / value) + elif name == "center_of_mass": + if value <= 0: + raise ValueError("center_of_mass must be positive") + value = 1 - 1 / (1 + value) + if not 0 < value <= 1: + raise ValueError("Resolved decay_rate must be in (0, 1]") + return value + + +@feature( + name="rsi_cutler", + category="momentum", + description="Finite simple-average RSI over period changes, omitting missing changes", + normalized=True, + value_range=(0.0, 100.0), + formula="100 * sum(positive changes) / sum(absolute changes)", +) +def rsi_cutler(close: str, period: int = 14) -> pl.Expr: + """Cutler RSI over period changes (period+1 prices); flat windows yield NaN. + + Missing quotes invalidate their adjacent changes. The remaining changes in + the full observation window are reduced without filling the missing prices. + Integer changes are subtracted in Int128 before floating conversion. + """ + _period(period) + + def change(price: pl.Expr, dtype: pl.DataType) -> pl.Expr: + return _difference(price, price.shift(1), dtype) + + delta = pl.col(close).pipe_with_dtype(change).fill_nan(None) + gain = delta.clip(lower_bound=0).rolling_mean(period, min_samples=1) + loss = (-delta.clip(upper_bound=0)).rolling_mean(period, min_samples=1) + return _ready(100 * gain / (gain + loss), period + 1) + + +@feature( + name="finite_mean", + category="trend", + description="Full-window simple mean with NaN/null omission and valid-count denominator", + formula="sum(valid prices) / count(valid prices)", +) +def finite_mean(close: str, period: int = 20) -> pl.Expr: + """Require period rows, then average the nonmissing values in that window.""" + _period(period) + return _ready(_numeric(close).rolling_mean(period, min_samples=1), period) + + +@feature( + name="finite_weighted_mean", + category="trend", + description="Finite weighted mean with independently reduced weight denominator", + input_type="OHLCV", + formula="nansum(close * volume) / nansum(volume)", +) +def finite_weighted_mean(close: str, volume: str, period: int = 20) -> pl.Expr: + """Weights paired with missing prices remain in the independent denominator. + + This explicit definition reproduces WeightedAverageValue/VWAP in Pipeline. + Callers requiring a paired-valid denominator must mask weights themselves. + """ + _period(period) + numerator = (_numeric(close) * _numeric(volume)).rolling_sum(period, min_samples=0) + denominator = _numeric(volume).rolling_sum(period, min_samples=0) + return _ready(numerator / denominator, period) + + +@feature( + name="average_dollar_volume", + category="volume", + description="Finite price-times-volume sum divided by the full observation count", + input_type="OHLCV", + formula="nansum(close * volume) / period", +) +def average_dollar_volume(close: str, volume: str, period: int = 20) -> pl.Expr: + """Mean dollar volume with fixed-count denominator; no currency/basis conversion.""" + _period(period) + total = (_numeric(close) * _numeric(volume)).rolling_sum(period, min_samples=0) + return _ready(total / period, period) + + +@jit(nopython=True, cache=True) # type: ignore[misc] +def _weighted_windows( + values: npt.NDArray[np.float64], + weights: npt.NDArray[np.float64], + omit: bool, +) -> npt.NDArray[np.float64]: + width = len(weights) + result = np.full(len(values), np.nan) + weight_sum = weights.sum() + for end in range(width - 1, len(values)): + total = 0.0 + missing = False + for j in range(width): + value = values[end - width + 1 + j] + if np.isnan(value): + missing = True + else: + total += (weights[j] / weight_sum) * value + if missing and not omit: + continue + result[end] = total + return result + + +def _variance_windows( + values: Any, + missing: npt.NDArray[np.bool_], + weights: npt.NDArray[np.float64], + omit: bool, + bias_corrected: bool, +) -> npt.NDArray[np.float64]: + width = len(weights) + result = np.full(len(values), np.nan) + for end in range(width - 1, len(values)): + start = end - width + 1 + first = -1 + weight_sum, denominator = 0.0, 0.0 + positive, negative, has_missing = False, False, False + for j in range(width): + if missing[start + j]: + has_missing = True + continue + if first < 0: + first = start + j + value = values[start + j] + positive = positive or value > 0 + negative = negative or value < 0 + # This equals W**2 - sum(w**2), without subtracting nearly equal numbers. + denominator += 2 * weight_sum * weights[j] + weight_sum += weights[j] + if first < 0 or (has_missing and not omit): + continue + if bias_corrected and denominator <= 0: + continue + anchor = values[first] + deltas = np.zeros(width) + scale = 0.0 + for j in range(width): + if missing[start + j]: + continue + value = values[start + j] + if positive and negative: + delta = float(value) + elif value >= anchor: + delta = float(value - anchor) + else: + delta = -float(anchor - value) + deltas[j] = delta + scale = max(scale, abs(delta)) + if not np.isfinite(scale): + continue + if scale == 0: + result[end] = 0.0 + continue + mean = 0.0 + for j in range(width): + if not missing[start + j]: + mean += (weights[j] / weight_sum) * (deltas[j] / scale) + variance = 0.0 + for j in range(width): + if not missing[start + j]: + residual = deltas[j] / scale - mean + variance += (weights[j] / weight_sum) * residual * residual + if bias_corrected: + variance = variance * weight_sum * weight_sum / denominator + result[end] = scale * np.sqrt(variance) + return result + + +_variance_numba = jit(nopython=True, cache=True)(_variance_windows) + + +def _variance_expression( + close: str | pl.Expr, + weights: npt.NDArray[np.float64], + *, + omit: bool = False, + bias_corrected: bool = True, +) -> pl.Expr: + expression = pl.col(close) if isinstance(close, str) else close + + def compute(values: pl.Series) -> pl.Series: + missing = values.is_null().to_numpy() + if values.dtype in (pl.Float32, pl.Float64): + missing = missing | values.is_nan().fill_null(False).to_numpy() + if values.dtype in (pl.Int128, pl.Null) or isinstance(values.dtype, pl.Decimal): + result = _variance_windows(values.to_list(), missing, weights, omit, bias_corrected) + else: + result = _variance_numba( + values.fill_null(0).to_numpy(), missing, weights, omit, bias_corrected + ) + return pl.Series(result) + + # Keep raw integer/Decimal values through window-local subtraction. A global + # float anchor loses later one-tick variation after a large price-level change. + return expression.map_batches(compute, return_dtype=pl.Float64) + + +def _weighted_expression( + close: str | pl.Expr, + weights: npt.NDArray[np.float64], + *, + variance: bool = False, + omit: bool = False, +) -> pl.Expr: + expression = pl.col(close) if isinstance(close, str) else close + if variance: + return _variance_expression(expression, weights) + return expression.cast(pl.Float64).map_batches( + lambda values: pl.Series(_weighted_windows(values.to_numpy(), weights, omit)), + return_dtype=pl.Float64, + ) + + +@feature( + name="linear_weighted_mean", + category="trend", + description="Finite linear weights 1..period with fixed denominator and NaN omission", + formula="nansum(price * [1..period]) / (period * (period+1) / 2)", +) +def linear_weighted_mean(close: str, period: int = 20) -> pl.Expr: + """Linear weighted mean; a missing price does not remove its denominator weight.""" + _period(period) + return _weighted_expression(close, np.arange(1, period + 1, dtype=np.float64), omit=True) + + +def _exponential_weights(period: int, decay_rate: float) -> npt.NDArray[np.float64]: + _period(period) + if ( + isinstance(decay_rate, bool) + or not isinstance(decay_rate, int | float) + or not math.isfinite(decay_rate) + or not 0 < decay_rate <= 1 + ): + raise ValueError("decay_rate must be finite and in (0, 1]") + # The common power cancels in both statistics. Newest weight 1 avoids + # needless underflow from the reference's decay**(period+1)..decay**2. + return np.power(float(decay_rate), np.arange(period - 1, -1, -1, dtype=np.float64)) + + +@feature( + name="finite_ewma", + category="trend", + description="Finite normalized exponential weighted mean; missing values propagate", + formula="sum(price * decay_rate**age) / sum(decay_rate**age)", +) +def finite_ewma(close: str, period: int = 20, decay_rate: float = 0.9) -> pl.Expr: + """Finite-window EWMA, distinct from recursively seeded ema.""" + return _weighted_expression(close, _exponential_weights(period, decay_rate)) + + +@feature( + name="finite_ewm_std", + category="statistics", + description="Finite exponential weighted standard deviation with weighted bias correction", + formula="sqrt(weighted_variance * sum(w)**2 / (sum(w)**2-sum(w**2)))", +) +def finite_ewm_std(close: str, period: int = 20, decay_rate: float = 0.9) -> pl.Expr: + """Bias-corrected finite weighted std; undefined effective degrees yield NaN.""" + return _weighted_expression(close, _exponential_weights(period, decay_rate), variance=True) + + +@feature( + name="annualized_volatility", + category="volatility", + description="Population standard deviation of supplied returns with explicit annualization", + input_type="returns", + formula="nanstd(returns, ddof=0) * sqrt(annualization_factor)", +) +def annualized_volatility( + returns: str | pl.Expr, period: int = 252, annualization_factor: float = 252.0 +) -> pl.Expr: + """Full-window population volatility; period counts supplied return observations.""" + _period(period) + if ( + isinstance(annualization_factor, bool) + or not isinstance(annualization_factor, int | float) + or not math.isfinite(annualization_factor) + or annualization_factor <= 0 + ): + raise ValueError("annualization_factor must be positive and finite") + volatility = _variance_expression(returns, np.ones(period), omit=True, bias_corrected=False) + return volatility * math.sqrt(annualization_factor) + + +@jit(nopython=True, cache=True) # type: ignore[misc] +def _drawdown_windows(values: npt.NDArray[np.float64], period: int) -> npt.NDArray[np.float64]: + result = np.full(len(values), np.nan) + for end in range(period - 1, len(values)): + peak = np.nan + largest = -np.inf + selected_peak = np.nan + selected_end = np.nan + for j in range(end - period + 1, end + 1): + value = values[j] + if np.isnan(value): + continue + peak = np.fmax(peak, value) + difference = peak - value + if difference > largest: + largest = difference + selected_peak, selected_end = peak, value + if selected_end != 0: + result[end] = (selected_peak - selected_end) / selected_end + elif selected_peak > 0: + result[end] = np.inf + return result + + +@feature( + name="finite_max_drawdown", + category="risk", + description="Pipeline absolute-drop selection with trough-price denominator", + formula="(peak - trough) / trough at first largest absolute price drop", +) +def finite_max_drawdown(close: str, period: int = 20) -> pl.Expr: + """Legacy finite drawdown definition, not portfolio/peak-denominator drawdown.""" + _period(period) + return ( + pl.col(close) + .cast(pl.Float64) + .map_batches( + lambda values: pl.Series(_drawdown_windows(values.to_numpy(), period)), + return_dtype=pl.Float64, + ) + ) + + +@feature( + name="finite_bollinger_bands", + category="volatility", + description="Finite NaN-omitting mean and population-standard-deviation bands", + formula="middle=nanmean(close); lower/upper=middle +/- k*nanstd(close,ddof=0)", +) +def finite_bollinger_bands(close: str, period: int = 20, k: float = 2.0) -> dict[str, pl.Expr]: + """Population bands requiring period rows; no smoothing or sample correction.""" + _period(period) + if isinstance(k, bool) or not isinstance(k, int | float) or not math.isfinite(k) or k < 0: + raise ValueError("k must be finite and nonnegative") + middle = finite_mean(close, period) + distance = _variance_expression(close, np.ones(period), omit=True, bias_corrected=False) * k + return {"lower": middle - distance, "middle": middle, "upper": middle + distance} + + +@jit(nopython=True, cache=True) # type: ignore[misc] +def _extreme_position( + values: npt.NDArray[np.float64], period: int, maximum: bool +) -> npt.NDArray[np.float64]: + result = np.full(len(values), np.nan) + for end in range(period - 1, len(values)): + chosen = np.nan + index = -1 + for j in range(period): + value = values[end - period + 1 + j] + if not np.isnan(value) and ( + index == -1 or (value > chosen if maximum else value < chosen) + ): + chosen, index = value, j + if index >= 0: + result[end] = 100.0 * index / (period - 1) + return result + + +@feature( + name="finite_aroon", + category="momentum", + description="Finite Aroon with first extreme ties and period-1 denominator", + input_type="HL", + normalized=True, + value_range=(0.0, 100.0), + formula="100 * first extreme position / (period-1)", +) +def finite_aroon(high: str, low: str, period: int = 25) -> dict[str, pl.Expr]: + """First-tie Aroon fields; period counts prices, not changes.""" + _period(period, minimum=2) + + def position(column: str, maximum: bool) -> pl.Expr: + return ( + pl.col(column) + .fill_nan(None) + .rank("dense") + .cast(pl.Float64) + .map_batches( + lambda values: pl.Series(_extreme_position(values.to_numpy(), period, maximum)), + return_dtype=pl.Float64, + ) + ) + + return {"down": position(low, False), "up": position(high, True)} + + +@feature( + name="fast_stochastic", + category="momentum", + description="Unsmoothed finite stochastic K with NaN-omitting extrema", + input_type="HLC", + normalized=True, + value_range=(0.0, 100.0), + formula="100 * (close - nanmin(low)) / (nanmax(high) - nanmin(low))", +) +def fast_stochastic(high: str, low: str, close: str, period: int = 14) -> pl.Expr: + """Unsmoothed K; zero ranges stay undefined rather than being replaced by zero.""" + _period(period) + upper = pl.col(high).fill_nan(None).rolling_max(period, min_samples=1) + lower = pl.col(low).fill_nan(None).rolling_min(period, min_samples=1) + numerator = _paired_difference(pl.col(close).fill_nan(None), lower) + denominator = _paired_difference(upper, lower) + return _ready(100 * numerator / denominator, period) + + +@feature( + name="finite_ichimoku", + category="trend", + description="Finite unshifted Ichimoku spans and explicit within-window chikou position", + input_type="HLC", +) +def finite_ichimoku( + high: str, + low: str, + close: str, + period: int = 52, + tenkan_length: int = 9, + kijun_length: int = 26, + chikou_position: int = 26, +) -> dict[str, pl.Expr]: + """Pipeline's unshifted fields, not a chart's displaced Ichimoku cloud. + + A zero tenkan/kijun length selects the whole window, matching Python's + legacy [-0:] slice. Chikou is an index from the oldest price in the window, + not a negative shift; its index must be in [0, period). + """ + _period(period) + for name, value in (("tenkan_length", tenkan_length), ("kijun_length", kijun_length)): + if isinstance(value, bool) or not isinstance(value, int) or not 0 <= value <= period: + raise ValueError(f"{name} must be an integer in [0, period]") + if ( + isinstance(chikou_position, bool) + or not isinstance(chikou_position, int) + or not 0 <= chikou_position < period + ): + raise ValueError("chikou_position must be an integer in [0, period)") + + def midpoint(width: int) -> pl.Expr: + return (_numeric(high).rolling_max(width) + _numeric(low).rolling_min(width)) / 2 + + tenkan = midpoint(tenkan_length or period) + kijun = midpoint(kijun_length or period) + outputs = { + "tenkan_sen": tenkan, + "kijun_sen": kijun, + "senkou_span_a": (tenkan + kijun) / 2, + "senkou_span_b": midpoint(period), + "chikou_span": pl.col(close).shift(period - 1 - chikou_position), + } + return {name: _ready(value, period) for name, value in outputs.items()} + + +@feature( + name="finite_macd_signal", + category="momentum", + description="MACD signal formed from three finite normalized exponential windows", + formula="finite_ewma(finite_ewma(fast)-finite_ewma(slow),signal)", +) +def finite_macd_signal( + close: str, fast_period: int = 12, slow_period: int = 26, signal_period: int = 9 +) -> pl.Expr: + """Finite MACD signal; length-one means use their sole observation.""" + for name, value in ( + ("fast_period", fast_period), + ("slow_period", slow_period), + ("signal_period", signal_period), + ): + try: + _period(value) + except ValueError as exc: + raise ValueError(f"{name}: {exc}") from exc + if slow_period <= fast_period: + raise ValueError("slow_period must be greater than fast_period") + + def weights(width: int) -> npt.NDArray[np.float64]: + if width == 1: + return np.ones(1) + return _exponential_weights(width, 1 - 2 / (1 + width)) + + difference = _weighted_expression(close, weights(fast_period)) - _weighted_expression( + close, weights(slow_period) + ) + return _weighted_expression(difference, weights(signal_period)) + + +@feature( + name="finite_true_range", + category="volatility", + description="Two-price true range with NaN-omitting horizontal components", + input_type="HLC", + formula="nanmax(high-low, abs(high-prev_close), abs(low-prev_close))", +) +def finite_true_range(high: str, low: str, close: str) -> pl.Expr: + """One-bar true range after its previous price; no arbitrary larger-window reduction.""" + upper, lower = pl.col(high).fill_nan(None), pl.col(low).fill_nan(None) + previous = pl.col(close).fill_nan(None).shift(1) + return _ready( + pl.max_horizontal( + _paired_difference(upper, lower), + _paired_difference(upper, previous).abs(), + _paired_difference(lower, previous).abs(), + ), + 2, + ) diff --git a/src/ml4t/engineer/features/selection.py b/src/ml4t/engineer/features/selection.py new file mode 100644 index 00000000..7608ad73 --- /dev/null +++ b/src/ml4t/engineer/features/selection.py @@ -0,0 +1,227 @@ +"""Contemporary eligible-group transforms and full-window boolean filters.""" + +import math + +import numpy as np +import numpy.typing as npt +import polars as pl + +from ml4t.engineer.features.rolling import _paired_difference, _period + + +def _value(value: str | pl.Expr) -> pl.Expr: + return pl.col(value) if isinstance(value, str) else value + + +def is_missing(value: str | pl.Expr) -> pl.Expr: + """True for native null or floating NaN, without sentinel or infinity inference.""" + + def build(expression: pl.Expr, dtype: pl.DataType) -> pl.Expr: + return ( + expression.is_null() | expression.is_nan().fill_null(False) + if dtype.is_float() + else expression.is_null() + ) + + return _value(value).pipe_with_dtype(build) + + +def _linear_edges( + values: npt.NDArray[np.float64], fractions: npt.NDArray[np.float64] +) -> npt.NDArray[np.float64]: + ordered = np.sort(values) + if not np.isfinite(ordered).all(): + raise ValueError("Quantile and percentile inputs require finite nonmissing values") + positions = (len(ordered) - 1) * np.asarray(fractions, dtype=np.float64) + a = ordered[np.floor(positions).astype(np.int64)] + b = ordered[np.ceil(positions).astype(np.int64)] + weight = positions - np.floor(positions) + opposite = np.signbit(a) != np.signbit(b) + edges = np.empty_like(positions) + # Avoid subtracting opposite finite extremes or losing small local values + # by centering every floating observation around a distant first value. + edges[opposite] = a[opposite] * (1 - weight[opposite]) + b[opposite] * weight[opposite] + edges[~opposite] = a[~opposite] + (b[~opposite] - a[~opposite]) * weight[~opposite] + return edges + + +def cross_section( + value: str | pl.Expr, + *, + method: str, + timestamp_col: str = "timestamp", + entity_col: str | list[str] = "asset", + groupby: str | list[str] | None = None, + mask: str | pl.Expr | None = None, + ascending: bool = True, + ties: str = "ordinal", + n: int | None = None, + bins: int | None = None, + min_percentile: float = 0.0, + max_percentile: float = 1.0, + ddof: int = 0, +) -> pl.Expr: + """Transform the eligible contemporary cross-section, independently per group. + + mask determines the input population, not a final screen. Null/NaN group labels, + null values and NaNs are excluded; legacy classifier sentinels must be mapped + explicitly. Ordinal ties use ascending stable entity IDs, not input row order. + Input keys must identify at most one observation per timestamp/entity. This + expression does not select source revisions or establish causal availability. + + Winsorization uses floor(n*lower)/ceil(n*upper)-1 order statistics. Quantile + edges and percentile bounds use linear interpolation. Duplicate quantile edges + raise rather than silently reducing bins. Percentile bounds are inclusive. + """ + methods = { + "rank", + "top", + "bottom", + "demean", + "zscore", + "winsorize", + "quantiles", + "percentile_between", + "peer_count", + } + if method not in methods: + raise ValueError(f"method must be one of {sorted(methods)}") + if not isinstance(ascending, bool) or ties not in {"ordinal", "min", "max", "dense", "average"}: + raise ValueError("Declare boolean ascending and a supported tie method") + if ties != "ordinal" and method != "rank": + raise ValueError("Only rank accepts a nonordinal tie method") + for name, count, applicable, minimum in ( + ("n", n, method in {"top", "bottom"}, 0), + ("bins", bins, method == "quantiles", 1), + ): + if applicable: + if isinstance(count, bool) or not isinstance(count, int) or count < minimum: + raise ValueError(f"{name} must be an integer >= {minimum}") + elif count is not None: + raise ValueError(f"{name} is not applicable to {method}") + if isinstance(ddof, bool) or not isinstance(ddof, int) or ddof < 0: + raise ValueError("ddof must be a nonnegative integer") + if ddof and method != "zscore": + raise ValueError("ddof applies only to zscore") + if ( + any( + isinstance(v, bool) or not isinstance(v, int | float) or not math.isfinite(v) + for v in (min_percentile, max_percentile) + ) + or not 0 <= min_percentile < max_percentile <= 1 + ): + raise ValueError("Declare 0 <= min_percentile < max_percentile <= 1") + if (min_percentile, max_percentile) != (0.0, 1.0) and method not in { + "winsorize", + "percentile_between", + }: + raise ValueError("Percentile bounds do not apply to this method") + entities = [entity_col] if isinstance(entity_col, str) else list(entity_col) + groups = [] if groupby is None else [groupby] if isinstance(groupby, str) else list(groupby) + if not entities or any( + not isinstance(c, str) or not c for c in [timestamp_col, *entities, *groups] + ): + raise ValueError("Declare nonempty timestamp, entity and grouping column names") + if mask is not None and not isinstance(mask, str | pl.Expr): + raise TypeError("mask must be a column name or expression") + original = _value(value) + eligible = ~is_missing(original) + if groups: + eligible &= pl.all_horizontal(~is_missing(c) for c in groups) + if mask is not None: + eligible &= _value(mask).fill_null(False) + x = pl.when(eligible).then(original) + partition = list(dict.fromkeys([timestamp_col, *groups])) + if method == "peer_count": + return pl.when(eligible).then(x.count().over([*partition, x]).cast(pl.Float64)) + if method in {"rank", "top", "bottom"}: + descending = method == "top" or (method == "rank" and not ascending) + rank = x.rank(method=ties, descending=descending).over(partition, order_by=entities) + return rank.cast(pl.Float64) if method == "rank" else (rank <= n).fill_null(False) + centered = _paired_difference(x, x.drop_nulls().first()) + if method in {"demean", "zscore"}: + # Retain typed ticks near a large offset, then scale residuals before + # moments. Opposite extreme floats need scaling before subtraction too. + fallback = centered.abs().max().is_infinite().fill_null(False) + raw_scale = x.abs().max().cast(pl.Float64) + working = pl.when(fallback).then(x / raw_scale).otherwise(centered) + scale = working.abs().max() + normalized = working / pl.when(scale > 0).then(scale).otherwise(1.0) + result = normalized - normalized.mean() + if method == "zscore": + result = result / normalized.std(ddof=ddof) + else: + result = result * scale * pl.when(fallback).then(raw_scale).otherwise(1.0) + elif method == "winsorize": + ordered = x.drop_nulls().sort() + lower = ordered.get((x.count() * min_percentile).floor().cast(pl.Int64), null_on_oob=True) + upper = ordered.get( + (x.count() * max_percentile).ceil().cast(pl.Int64) - 1, null_on_oob=True + ) + result = x.clip(lower, upper) + else: + + def coordinates(value: pl.Expr, dtype: pl.DataType) -> pl.Expr: + return ( + value.cast(pl.Float64) + if dtype.is_float() + else _paired_difference(value, value.drop_nulls().first()) + ) + + def select_quantiles(series: pl.Series) -> pl.Series: + values = series.to_numpy() + valid = ~np.isnan(values) + if not valid.any(): + return ( + pl.Series([None] * len(values), dtype=pl.Int64) + if method == "quantiles" + else pl.Series([False] * len(values), dtype=pl.Boolean) + ) + if method == "percentile_between": + lower, upper = _linear_edges( + values[valid], np.array([min_percentile, max_percentile]) + ) + return pl.Series(valid & (values >= lower) & (values <= upper)) + assert bins is not None + edges = _linear_edges(values[valid], np.linspace(0.0, 1.0, bins + 1)) + if bins > 1 and len(np.unique(edges)) != len(edges): + raise ValueError("Duplicate quantile edges; declare a different bin count") + labels = np.searchsorted(edges, values, side="left") - 1 + labels[values == edges[0]] = 0 + return pl.Series(labels, dtype=pl.Int64).set(pl.Series(~valid), None) + + result = x.pipe_with_dtype(coordinates).map_batches( + select_quantiles, return_dtype=pl.Int64 if method == "quantiles" else pl.Boolean + ) + return result.over(partition, order_by=entities) + + +def rolling_filter( + value: str | pl.Expr, + *, + period: int, + method: str, + minimum: int | None = None, +) -> pl.Expr: + """Any/all/at_least over full observation rows; missing predicates count false. + + Apply over(entity_col) to isolate histories. Leading incomplete windows are + null. Keep input eligibility and final screens separate from this history. + """ + _period(period) + if method not in {"any", "all", "at_least"}: + raise ValueError("method must be any, all or at_least") + if method == "at_least": + if isinstance(minimum, bool) or not isinstance(minimum, int) or not 0 <= minimum <= period: + raise ValueError("minimum must be an integer in [0,period]") + elif minimum is not None: + raise ValueError("minimum applies only to at_least") + + def build(expression: pl.Expr, dtype: pl.DataType) -> pl.Expr: + if dtype != pl.Boolean: + raise ValueError("rolling_filter requires boolean predicates") + return expression.fill_null(False).cast(pl.Int64) + + count = _value(value).pipe_with_dtype(build).rolling_sum(period) + threshold = 1 if method == "any" else period if method == "all" else minimum + return count >= threshold diff --git a/src/ml4t/engineer/utils/dependencies.py b/src/ml4t/engineer/utils/dependencies.py index 7389c44a..a0201a7f 100644 --- a/src/ml4t/engineer/utils/dependencies.py +++ b/src/ml4t/engineer/utils/dependencies.py @@ -115,6 +115,14 @@ def __init__(self): def _register_dependencies(self): """Register all known optional dependencies.""" + self._deps["scipy"] = DependencyInfo( + name="SciPy", + import_name="scipy", + install_cmd="uv add 'ml4t-engineer[stats]'", + purpose="Rolling paired regression inferential fields", + features=["features.paired.rolling_regression"], + ) + # ML Libraries self._deps["lightgbm"] = DependencyInfo( name="LightGBM", diff --git a/tests/compat/test_calendar_duration_runtime.py b/tests/compat/test_calendar_duration_runtime.py new file mode 100644 index 00000000..e1e4073a --- /dev/null +++ b/tests/compat/test_calendar_duration_runtime.py @@ -0,0 +1,90 @@ +"""Valid public grids remain valid in pandas2/NumPy2 legacy-coexistence environments.""" + +from datetime import date + +import polars as pl +import pytest +from ml4t.specs import FeedSpec + +from ml4t.engineer import compute_features + + +def bars(timestamp: pl.Series) -> pl.DataFrame: + return pl.DataFrame( + {"timestamp": timestamp, "asset": ["A"] * 4, "close": [100.0, 110.0, 120.0, 130.0]} + ).with_columns( + pl.col("close").alias("open"), + pl.col("close").alias("high"), + pl.col("close").alias("low"), + pl.lit(100).alias("volume"), + ) + + +def test_daily_public_feature_grid_does_not_misclassify_a_dependency_warning_as_bad_frequency(): + frame = bars(pl.Series([date(2025, 7, d) for d in (7, 8, 9, 10)])) + result = compute_features( + frame.lazy(), + [{"name": "sma", "params": {"period": 2}}], + feed_spec=FeedSpec( + time_contract_version=2, + calendar="NYSE", + entity_col="asset", + timestamp_semantics="session_label", + bar_type="time", + data_frequency="1d", + ), + ).collect() + assert result["sma"].to_list() == [None, 105.0, 115.0, 125.0] + assert result["timestamp"].equals(frame["timestamp"]) + + +@pytest.mark.parametrize( + "frequency,step", + [("1h", 3_600_000_000_000), ("30m", 1_800_000_000_000), ("1ms", 1_000_000), ("1ns", 1)], +) +def test_intraday_public_declared_grid_keeps_exact_frequency_and_rows_across_dependency_versions( + frequency, step +): + opened = 1751846400000000000 + frame = bars(pl.Series([opened + i * step for i in range(4)], dtype=pl.Datetime("ns", "UTC"))) + contract = FeedSpec( + time_contract_version=2, + calendar="24/7", + entity_col="asset", + timezone="UTC", + timestamp_semantics="event_time", + bar_type="time", + data_frequency=frequency, + bar_grid_anchor="session_open", + bar_label="left", + partial_bar_policy="exclude", + ) + result = compute_features(frame, [{"name": "sma", "params": {"period": 2}}], feed_spec=contract) + assert result["sma"].to_list() == [None, 105.0, 115.0, 125.0] + assert result["timestamp"].equals(frame["timestamp"]) + + +@pytest.mark.parametrize("frequency", ["NaT", "0h", "1month", "1M", "1Y", "1T", "1H"]) +def test_invalid_or_deprecated_ambiguous_frequencies_still_fail_before_calculation(frequency): + step = 60_000_000_000 if frequency == "1T" else 3_600_000_000_000 + frame = bars( + pl.Series( + [1751846400000000000 + i * step for i in range(4)], dtype=pl.Datetime("ns", "UTC") + ) + ) + with pytest.raises(ValueError, match="frequency"): + compute_features( + frame, + ["sma"], + feed_spec=FeedSpec( + time_contract_version=2, + calendar="24/7", + timestamp_semantics="event_time", + timezone="UTC", + bar_grid_anchor="session_open", + bar_label="left", + partial_bar_policy="exclude", + bar_type="time", + data_frequency=frequency, + ), + ) diff --git a/tests/compat/test_zipline_events.py b/tests/compat/test_zipline_events.py new file mode 100644 index 00000000..2fe070ae --- /dev/null +++ b/tests/compat/test_zipline_events.py @@ -0,0 +1,101 @@ +"""Known event weekday counts must not infer exchange closures or future knowledge.""" + +import importlib +from datetime import UTC, date, datetime + +import numpy as np +import polars as pl +import pytest + + +def expression(direction: str = "since") -> pl.Expr: + module = importlib.import_module("ml4t.engineer.features.events") + return module.event_business_days( + "event", + reference_date_col="session", + known_at_col="known", + decision_time_col="decision", + direction=direction, + ).alias("days") + + +@pytest.mark.parametrize("direction", ["since", "until"]) +@pytest.mark.parametrize("lazy", [False, True]) +def test_event_counts_use_weekdays_including_nyse_holidays(direction: str, lazy: bool) -> None: + frame = pl.DataFrame( + { + "event": [date(2025, 7, 4), date(2025, 7, 5), None, date(2025, 7, 7)], + "session": [date(2025, 7, 7)] * 4, + "known": [datetime(2025, 7, 1, tzinfo=UTC)] * 4, + "decision": [datetime(2025, 7, 7, 14, tzinfo=UTC)] * 4, + } + ) + output = (frame.lazy() if lazy else frame).select(expression(direction)) + if lazy: + output = output.collect() + events = np.array(["2025-07-04", "2025-07-05", "NaT", "2025-07-07"], dtype="datetime64[D]") + valid = ~np.isnat(events) + expected = np.full(4, np.nan) + reference = np.array("2025-07-07", dtype="datetime64[D]") + expected[valid] = ( + np.busday_count(events[valid], reference) + if direction == "since" + else np.busday_count(reference, events[valid]) + ) + np.testing.assert_allclose(output["days"].to_numpy(), expected, equal_nan=True) + assert abs(output["days"][0]) == 1 + + +def test_event_dates_are_ineligible_before_announcement_at_exact_ns_cutoff() -> None: + instant = 1751896800000000000 + frame = pl.DataFrame( + { + "event": [date(2025, 7, 10)] * 4, + "session": [date(2025, 7, 7)] * 4, + "known": pl.Series([instant - 1, instant, instant + 1, None], dtype=pl.Int64).cast( + pl.Datetime("ns", "UTC") + ), + "decision": pl.Series([instant] * 4).cast(pl.Datetime("ns", "UTC")), + } + ) + actual = frame.select(expression("until"))["days"].to_list() + assert actual == [3.0, 3.0, None, None] + + +def test_event_knowledge_comparison_preserves_timezone_and_out_of_ns_instants() -> None: + frame = pl.DataFrame( + { + "event": [date(2500, 1, 2)], + "session": [date(2500, 1, 5)], + "known": [datetime(2499, 12, 1, tzinfo=UTC)], + "decision": [datetime(2500, 1, 5, tzinfo=UTC)], + } + ) + original = frame.select(expression()) + changed = frame.with_columns(pl.col("known").dt.convert_time_zone("America/New_York")).select( + expression() + ) + assert original.equals(changed) + assert original["days"][0] == np.busday_count("2500-01-02", "2500-01-05") + + +@pytest.mark.parametrize( + "invalid", ["reference_datetime", "naive_knowledge", "naive_decision", "direction"] +) +def test_event_contract_rejects_ambiguous_time_roles(invalid: str) -> None: + frame = pl.DataFrame( + { + "event": [date(2025, 7, 4)], + "session": [date(2025, 7, 7)], + "known": [datetime(2025, 7, 1, tzinfo=UTC)], + "decision": [datetime(2025, 7, 7, tzinfo=UTC)], + } + ) + if invalid == "reference_datetime": + frame = frame.with_columns(pl.col("session").cast(pl.Datetime("us", "UTC"))) + elif invalid == "naive_knowledge": + frame = frame.with_columns(pl.col("known").dt.replace_time_zone(None)) + elif invalid == "naive_decision": + frame = frame.with_columns(pl.col("decision").dt.replace_time_zone(None)) + with pytest.raises(ValueError): + frame.select(expression("invalid" if invalid == "direction" else "since")) diff --git a/tests/compat/test_zipline_factor_recipes.py b/tests/compat/test_zipline_factor_recipes.py new file mode 100644 index 00000000..79dc9c4f --- /dev/null +++ b/tests/compat/test_zipline_factor_recipes.py @@ -0,0 +1,275 @@ +"""Thin legacy-name recipes must preserve methods, input ordering and output masks.""" + +import importlib +import math +from datetime import UTC + +import numpy as np +import polars as pl +import pytest + + +def recipe(name: str, **options): + return importlib.import_module("ml4t.engineer.compat.zipline").factor_expression( + name, **options + ) + + +@pytest.mark.parametrize( + "name,window,expected", + [ + ("Returns", 3, [np.nan, np.nan, 0.21, 0.21]), + ("DailyReturns", None, [np.nan, 0.1, 0.1, 0.1]), + ("DailyReturns", 3, [np.nan, np.nan, 0.21, 0.21]), + ("RateOfChangePercentage", 3, [np.nan, np.nan, 21, 21]), + ], +) +def test_return_recipes_translate_price_count_to_changes(name, window, expected) -> None: + frame = pl.DataFrame({"close": [100.0, 110.0, 121.0, 133.1]}) + options = {"window_length": window} if window is not None else {} + if name == "RateOfChangePercentage": + options["inputs"] = ["close"] + actual = frame.lazy().select(recipe(name, **options).alias("result")).collect() + np.testing.assert_allclose( + actual["result"].to_numpy(), expected, rtol=1e-14, atol=0, equal_nan=True + ) + + +def test_absolute_percent_change_does_not_use_signed_denominator_or_percent_scale() -> None: + frame = pl.DataFrame({"value": [-100.0, -90.0, -110.0]}) + actual = frame.select( + recipe("PercentChange", inputs=["value"], window_length=2).alias("result") + ) + np.testing.assert_allclose( + actual["result"].to_numpy(), [np.nan, 0.1, -2 / 9], rtol=1e-14, equal_nan=True + ) + zero = pl.DataFrame({"value": [-0.0, 1.0, -0.0, 2.0]}) + np.testing.assert_allclose( + zero.select(recipe("PercentChange", inputs=["value"], window_length=2)) + .to_series() + .to_numpy(), + [np.nan, np.inf, -1, np.inf], + equal_nan=True, + ) + + +@pytest.mark.parametrize( + "name", + ["EWMA", "ExponentialWeightedMovingAverage", "EWMSTD", "ExponentialWeightedMovingStdDev"], +) +@pytest.mark.parametrize( + "constructor,params", + [ + ("from_span", {"span": 3.0}), + ("from_halflife", {"halflife": 1.0}), + ("from_center_of_mass", {"center_of_mass": 1.0}), + ], +) +def test_exponential_factories_select_the_same_declared_finite_weights( + name, constructor, params +) -> None: + frame = pl.DataFrame({"close": [1.0, 3.0, 9.0]}) + actual = frame.select( + recipe(f"{name}.{constructor}", inputs=["close"], window_length=3, **params) + ).item(-1, 0) + expected = ( + math.sqrt(138 / 7) if name in {"EWMSTD", "ExponentialWeightedMovingStdDev"} else 43 / 7 + ) + assert actual == pytest.approx(expected, rel=1e-14, abs=0) + + +@pytest.mark.parametrize( + "name", + [ + "SimpleMovingAverage", + "WeightedAverageValue", + "VWAP", + "AverageDollarVolume", + "LinearWeightedMovingAverage", + "MaxDrawdown", + "RSI", + "BollingerBands", + "Aroon", + "FastStochasticOscillator", + "IchimokuKinkoHyo", + "TrueRange", + "AnnualizedVolatility", + "MACDSignal", + "MovingAverageConvergenceDivergenceSignal", + ], +) +def test_numeric_recipes_produce_lazy_expressions_without_importing_legacy_runtime(name) -> None: + import sys + + frame = pl.DataFrame( + {"close": np.linspace(10.0, 20.0, 60), "volume": np.arange(1.0, 61.0)} + ).with_columns( + (pl.col("close") + 2).alias("high"), + (pl.col("close") - 2).alias("low"), + ) + options = {} + if name in {"SimpleMovingAverage", "MaxDrawdown", "LinearWeightedMovingAverage"}: + options.update(inputs=["close"], window_length=3) + elif name == "WeightedAverageValue": + options.update(inputs=["close", "volume"], window_length=3) + elif name in {"VWAP", "AverageDollarVolume", "BollingerBands", "Aroon"}: + options["window_length"] = 3 + output = recipe(name, **options) + if isinstance(output, dict): + expressions = [value.alias(key) for key, value in output.items()] + else: + assert isinstance(output, pl.Expr) + expressions = [output.alias("result")] + result = frame.lazy().select(expressions).collect() + assert result.height == frame.height + assert not any(key == "zipline" or key.startswith("zipline.") for key in sys.modules) + + +def test_aroon_recipe_translates_legacy_low_high_input_order() -> None: + frame = pl.DataFrame({"l": [3.0, 1.0, 1.0], "h": [4.0, 5.0, 5.0]}) + output = recipe("Aroon", inputs=["l", "h"], window_length=3) + result = frame.select([v.alias(k) for k, v in output.items()]) + assert result.row(-1) == (50.0, 50.0) + + +def test_output_mask_does_not_erase_prior_input_history() -> None: + frame = pl.DataFrame({"close": [1.0, 2.0, 3.0, 4.0], "mask": [False, True, False, True]}) + result = frame.with_columns( + recipe("SimpleMovingAverage", inputs=["close"], window_length=3, mask="mask").alias( + "result" + ) + ) + assert result["result"].to_list() == [None, None, None, 3.0] + assert result.height == 4 + assert result["close"].to_list() == [1.0, 2.0, 3.0, 4.0] + + +def test_clip_recipe_delegates_bounds_and_preserves_missing_values() -> None: + frame = pl.DataFrame({"value": [float("nan"), -3.0, -1.0, 4.0]}) + result = frame.select(recipe("Clip", inputs=["value"], min_bound=-2.0, max_bound=2.0)) + np.testing.assert_allclose(result.to_series().to_numpy(), [np.nan, -2, -1, 2], equal_nan=True) + + +def test_default_annualized_recipe_keeps_price_to_returns_dependency_warmup() -> None: + frame = pl.DataFrame({"close": [100.0, 110.0, 121.0, 133.1]}) + result = frame.select(recipe("AnnualizedVolatility", window_length=3).alias("result")) + initial = result["result"].head(3) + assert (initial.is_null() | initial.is_nan()).all() + assert result["result"][-1] == pytest.approx(0, abs=1e-14) + + +@pytest.mark.parametrize( + "name,options", + [ + ("RSI", {"regression_length": 3}), + ("VWAP", {"correlation_length": 3}), + ("VWAP", {"window_length": 3, "inputs": {"close", "volume"}}), + ( + "BusinessDaysSincePreviousEvent", + { + "inputs": ["event"], + "window_length": False, + "reference_date_col": "session", + "known_at_col": "known", + "decision_time_col": "decision", + }, + ), + ], +) +def test_options_cannot_change_unrelated_factor_meaning_or_input_order(name, options) -> None: + with pytest.raises((TypeError, ValueError)): + recipe(name, **options) + + +@pytest.mark.parametrize( + "name,length,expected", + [ + ("RollingPearson", "correlation_length", 9 / math.sqrt(95)), + ("RollingSpearman", "correlation_length", 3 / math.sqrt(10)), + ], +) +def test_paired_recipes_preserve_actual_paired_series(name, length, expected) -> None: + frame = pl.DataFrame({"x": [1.0, 2.0, 3.0, 4.0], "y": [1.0, 2.0, 2.0, 4.0]}) + result = frame.select(recipe(name, inputs=["y", "x"], **{length: 4})) + assert result.item(-1, 0) == pytest.approx(expected, rel=1e-14, abs=0) + + +def test_regression_recipe_exposes_actual_named_fields() -> None: + pytest.importorskip("scipy", reason="Regression p-values require stats extra") + frame = pl.DataFrame({"x": [1.0, 2.0, 3.0, 4.0], "y": [1.0, 2.0, 2.0, 4.0]}) + result = frame.select( + recipe("RollingLinearRegression", inputs=["y", "x"], regression_length=4).alias( + "regression" + ) + ).unnest("regression") + np.testing.assert_allclose( + result.row(-1), + [0, 0.9, 9 / math.sqrt(95), 1 - 9 / math.sqrt(95), math.sqrt(0.07)], + rtol=1e-13, + atol=1e-15, + ) + + +def test_beta_recipe_preserves_allowed_missing_fraction_without_stats_extra() -> None: + gapped = pl.DataFrame({"x": [0.0, 1.0, 2.0, 3.0], "y": [1.0, None, 5.0, 7.0]}) + result = gapped.select(recipe("SimpleBeta", inputs=["y", "x"], regression_length=4)) + assert result.item(-1, 0) == pytest.approx(2.0, rel=1e-14) + + +@pytest.mark.parametrize( + "values,dtype", [(["A", "B"], pl.String), ([True, False], pl.Boolean), ([1, 2], pl.Int64)] +) +def test_latest_recipe_preserves_nonprice_input_dtypes(values, dtype) -> None: + frame = pl.DataFrame({"value": pl.Series(values, dtype=dtype)}) + result = frame.select(recipe("Latest", inputs=["value"]).alias("value")) + assert result.equals(frame) + + +@pytest.mark.parametrize( + "name,expected", [("BusinessDaysSincePreviousEvent", 1.0), ("BusinessDaysUntilNextEvent", -1.0)] +) +def test_event_recipes_use_explicit_date_and_knowledge_roles(name, expected) -> None: + from datetime import date, datetime + + frame = pl.DataFrame( + { + "event": [date(2025, 7, 4)], + "session": [date(2025, 7, 7)], + "known": [datetime(2025, 7, 1, tzinfo=UTC)], + "decision": [datetime(2025, 7, 7, tzinfo=UTC)], + } + ) + result = frame.select( + recipe( + name, + inputs=["event"], + reference_date_col="session", + known_at_col="known", + decision_time_col="decision", + ) + ) + assert result.item() == expected + + +@pytest.mark.parametrize( + "name,options", + [ + ("RSI", {"unknown": 1}), + ("RSI", {"window_length": True}), + ("RSI", {"inputs": "close"}), + ("VWAP", {"inputs": ["close"], "window_length": 3}), + ("EWMA", {"inputs": ["close"], "window_length": 3}), + ("EWMA", {"inputs": ["close"], "window_length": 3, "span": 3, "decay_rate": 0.5}), + ("SimpleMovingAverage", {"window_length": 3}), + ("SimpleMovingAverage", {"inputs": ["close"]}), + ("TrueRange", {"window_length": 3}), + ("RSI.from_span", {"span": 3}), + ("MACDSignal", {"window_length": 34}), + ("Factor", {}), + ], +) +def test_recipe_rejects_ambiguous_or_nonfunctional_requests_before_evaluation( + name, options +) -> None: + with pytest.raises((TypeError, ValueError)): + recipe(name, **options) diff --git a/tests/compat/test_zipline_factors.py b/tests/compat/test_zipline_factors.py new file mode 100644 index 00000000..09a873fc --- /dev/null +++ b/tests/compat/test_zipline_factors.py @@ -0,0 +1,361 @@ +"""Finite Pipeline definitions must not silently use recursive or sample methods.""" + +from __future__ import annotations + +import math + +import numpy as np +import polars as pl +import pytest +from polars.testing import assert_frame_equal + +from ml4t.engineer import compute_features +from ml4t.engineer.core.registry import get_registry + + +def prices(close: list[float | None], volume: list[float | None] | None = None) -> pl.DataFrame: + return pl.DataFrame( + { + "open": close, + "high": [None if x is None else x + 2 for x in close], + "low": [None if x is None else x / 2 for x in close], + "close": close, + "volume": volume if volume is not None else [10.0] * len(close), + }, + schema_overrides=dict.fromkeys(("open", "high", "low", "close", "volume"), pl.Float64), + ) + + +def calculate(frame: pl.DataFrame, name: str, **params: object) -> pl.DataFrame: + return compute_features( + frame, [{"name": name, "params": params, "output": "factor"}], calendar=None + ) + + +@pytest.mark.parametrize("lazy", [False, True]) +def test_cutler_uses_finite_changes_instead_of_wilder_recursion(lazy: bool) -> None: + frame = prices([10.0, 13.0, 12.0, 14.0, 13.0, 17.0, 16.0]) + result = compute_features( + frame.lazy() if lazy else frame, + [{"name": "rsi_cutler", "params": {"period": 3}}], + calendar=None, + ) + assert isinstance(result, pl.LazyFrame if lazy else pl.DataFrame) + actual = result.collect() if lazy else result + np.testing.assert_allclose( + actual["rsi_cutler"].to_numpy(), + [np.nan, np.nan, np.nan, 100 * 5 / 6, 50, 100 * 6 / 7, 100 * 4 / 6], + rtol=1e-14, + equal_nan=True, + ) + native = calculate(frame, "rsi", period=3)["factor"].to_numpy() + assert not np.allclose(native[4:], actual["rsi_cutler"].to_numpy()[4:]) + + +@pytest.mark.parametrize("missing", [None, float("nan")]) +def test_cutler_omits_missing_changes_without_filling_quotes(missing: float | None) -> None: + frame = prices([1.0, 2.0, missing, 4.0, 5.0, 5.0, 3.0]) + result = calculate(frame, "rsi_cutler", period=3) + np.testing.assert_allclose( + result["factor"].to_numpy(), + [np.nan, np.nan, np.nan, 100, 100, 100, 100 / 3], + equal_nan=True, + ) + assert_frame_equal(result.select(frame.columns), frame) + + +@pytest.mark.parametrize( + ("name", "close", "volume", "params", "expected"), + [ + ("finite_mean", [1.0, np.nan, 5.0], None, {}, 3.0), + ("finite_weighted_mean", [1.0, np.nan, 5.0], [1.0, 2.0, 3.0], {}, 16 / 6), + ("finite_weighted_mean", [1.0, 2.0, 5.0], [0.0, 0.0, 0.0], {}, np.nan), + ("average_dollar_volume", [1.0, np.nan, 5.0], [1.0, 2.0, 3.0], {}, 16 / 3), + ("average_dollar_volume", [1.0, np.nan, 5.0], [0.0, 0.0, 0.0], {}, 0.0), + ("linear_weighted_mean", [1.0, np.nan, 5.0], None, {}, 16 / 6), + ("finite_ewma", [1.0, 3.0, 9.0], None, {"decay_rate": 0.5}, 43 / 7), + ("finite_ewma", [1.0, 3.0, 9.0], None, {"decay_rate": 1.0}, 13 / 3), + ("finite_ewma", [1.0, np.nan, 9.0], None, {"decay_rate": 0.5}, np.nan), + ("finite_ewm_std", [1.0, 3.0, 9.0], None, {"decay_rate": 1.0}, math.sqrt(52 / 3)), + ("finite_ewm_std", [1.0, 3.0, 9.0], None, {"decay_rate": 0.5}, math.sqrt(138 / 7)), + ("finite_ewm_std", [1.0, np.nan, 9.0], None, {"decay_rate": 0.5}, np.nan), + ( + "annualized_volatility", + [1.0, 3.0, 9.0], + None, + {"annualization_factor": 9.0}, + math.sqrt(104), + ), + ("finite_max_drawdown", [100.0, 50.0, 60.0], None, {}, 1.0), + ("finite_max_drawdown", [100.0, 60.0, 50.0], None, {}, 1.0), + ], +) +def test_finite_methods_preserve_nan_denominators_weights_and_scale( + name: str, + close: list[float], + volume: list[float] | None, + params: dict[str, object], + expected: float, +) -> None: + frame = prices(close, volume) + if name == "annualized_volatility": + frame = frame.with_columns(pl.col("close").alias("returns")) + actual = calculate(frame, name, period=3, **params)["factor"].to_numpy() + np.testing.assert_allclose(actual, [np.nan, np.nan, expected], equal_nan=True, rtol=1e-14) + + +@pytest.mark.parametrize("name", ["finite_ewma", "finite_ewm_std"]) +def test_finite_weights_do_not_overflow_or_underflow_with_legacy_common_power(name: str) -> None: + frame = prices([4.0, 4.0, 4.0, 4.0]) + result = calculate(frame, name, period=4, decay_rate=1e-100)["factor"][-1] + assert result == (4.0 if name == "finite_ewma" else 0.0) + + +@pytest.mark.parametrize( + "name", ["finite_mean", "linear_weighted_mean", "finite_ewma", "rsi_cutler"] +) +def test_finite_window_never_crosses_asset_boundaries_and_lazy_matches_eager(name: str) -> None: + first = prices([10.0, 13.0, 12.0, 14.0]).with_columns(pl.lit("A").alias("asset")) + second = prices([100.0, 130.0, 120.0, 140.0]).with_columns(pl.lit("B").alias("asset")) + frame = pl.concat([first, second]).sort("close") + spec = [{"name": name, "params": {"period": 3}}] + result = compute_features(frame, spec, group_col="asset", calendar=None) + lazy = compute_features(frame.lazy(), spec, group_col="asset", calendar=None).collect() + assert_frame_equal(result, lazy) + for asset in ("A", "B"): + expected = compute_features(frame.filter(pl.col("asset") == asset), spec, calendar=None) + assert_frame_equal(result.filter(pl.col("asset") == asset), expected) + + +@pytest.mark.parametrize( + "name", + [ + "finite_mean", + "finite_weighted_mean", + "average_dollar_volume", + "linear_weighted_mean", + "finite_ewma", + "finite_ewm_std", + "annualized_volatility", + "finite_max_drawdown", + "rsi_cutler", + ], +) +@pytest.mark.parametrize("period", [0, -1, True, 1.5]) +def test_invalid_windows_fail_without_modifying_input(name: str, period: object) -> None: + frame = prices([1.0, 2.0, 3.0]).with_columns(pl.lit(0.1).alias("returns")) + before = frame.clone() + with pytest.raises((TypeError, ValueError), match="period"): + calculate(frame, name, period=period) + assert_frame_equal(frame, before) + + +@pytest.mark.parametrize("decay", [0.0, -0.1, 1.1, float("nan"), float("inf"), True]) +def test_invalid_decay_fails_before_output(decay: float) -> None: + with pytest.raises((TypeError, ValueError), match="decay_rate"): + calculate(prices([1.0, 2.0, 3.0]), "finite_ewma", period=3, decay_rate=decay) + + +def test_cutler_registry_metadata_declares_price_count_and_keeps_native_defaults() -> None: + cutler = get_registry().get("rsi_cutler") + native = get_registry().get("rsi") + assert cutler is not None and native is not None + assert cutler.lookback(period=5) == 5 + assert cutler.value_range == (0.0, 100.0) + assert cutler.normalized and not cutler.ta_lib_compatible + assert native.parameters["period"] == 14 + assert native.ta_lib_compatible + + +def test_finite_bands_use_population_std_and_nan_omission() -> None: + actual = calculate(prices([1.0, 3.0, 9.0]), "finite_bollinger_bands", period=3, k=2) + middle = 13 / 3 + distance = 2 * math.sqrt(104 / 9) + assert actual["factor_lower"][-1] == pytest.approx(middle - distance) + assert actual["factor_middle"][-1] == pytest.approx(middle) + assert actual["factor_upper"][-1] == pytest.approx(middle + distance) + missing = calculate(prices([1.0, np.nan, 9.0]), "finite_bollinger_bands", period=3, k=1) + assert missing.select("factor_lower", "factor_middle", "factor_upper").row(-1) == (1, 5, 9) + + +def test_aroon_first_extreme_tie_is_not_native_latest_tie() -> None: + frame = prices([3.5, 2.0, 4.0]).with_columns( + pl.Series("high", [4.0, 5.0, 5.0]), pl.Series("low", [3.0, 1.0, 1.0]) + ) + actual = calculate(frame, "finite_aroon", period=3) + assert actual["factor_down"][-1] == 50 + assert actual["factor_up"][-1] == 50 + assert ( + actual["factor_up"].head(2).is_null().all() + or np.isnan(actual["factor_up"].head(2).to_numpy()).all() + ) + + +def test_fast_stochastic_has_no_smoothing_or_zero_range_replacement() -> None: + frame = prices([3.5, 2.0, 4.0]).with_columns( + pl.Series("high", [4.0, 5.0, 5.0]), pl.Series("low", [3.0, 1.0, 1.0]) + ) + assert calculate(frame, "fast_stochastic", period=3)["factor"][-1] == 75 + flat = pl.DataFrame({name: [3.0] * 3 for name in ("open", "high", "low", "close", "volume")}) + assert math.isnan(calculate(flat, "fast_stochastic", period=3)["factor"][-1]) + + +def test_ichimoku_fields_use_unshifted_spans_and_position_within_window() -> None: + frame = prices([10.0, 14.0, 11.0, 20.0, 16.0, 14.0]).with_columns( + pl.Series("high", [12.0, 18.0, 14.0, 22.0, 20.0, 16.0]), + pl.Series("low", [8.0, 10.0, 9.0, 16.0, 13.0, 12.0]), + ) + actual = calculate( + frame, "finite_ichimoku", period=6, tenkan_length=2, kijun_length=3, chikou_position=2 + ) + assert actual.select(pl.col("^factor_.*$")).row(-1) == (16, 17, 16.5, 15, 11) + assert all( + actual[c].head(5).null_count() == 5 for c in actual.columns if c.startswith("factor_") + ) + + +def test_finite_macd_signal_uses_normalized_windows_not_recursive_ema() -> None: + actual = calculate( + prices([1.0, 2.0, 4.0, 8.0, 16.0]), + "finite_macd_signal", + fast_period=2, + slow_period=3, + signal_period=2, + ) + np.testing.assert_allclose( + actual["factor"].to_numpy(), [np.nan] * 3 + [0.875, 1.75], equal_nan=True + ) + + +def test_single_observation_macd_means_are_defined_instead_of_zero_weight_division() -> None: + actual = calculate( + prices([1.0, 3.0, 9.0]), + "finite_macd_signal", + fast_period=1, + slow_period=2, + signal_period=1, + ) + np.testing.assert_allclose(actual["factor"].to_numpy(), [np.nan, 0.5, 1.5], equal_nan=True) + + +def test_finite_true_range_uses_two_price_rows_and_nanmax_components() -> None: + frame = prices([10.0, 12.0, 13.0]).with_columns( + pl.Series("high", [12.0, 15.0, 14.0]), pl.Series("low", [8.0, 11.0, 12.0]) + ) + np.testing.assert_allclose( + calculate(frame, "finite_true_range")["factor"].to_numpy(), [np.nan, 5, 2], equal_nan=True + ) + + +@pytest.mark.parametrize( + ("name", "params"), + [ + ("finite_aroon", {"period": 1}), + ("finite_bollinger_bands", {"k": float("nan")}), + ("finite_ichimoku", {"period": 6, "tenkan_length": 7}), + ("finite_ichimoku", {"period": 6, "chikou_position": 6}), + ("finite_macd_signal", {"fast_period": 3, "slow_period": 3}), + ("finite_macd_signal", {"signal_period": 0}), + ], +) +def test_technical_options_reject_undefined_windows(name: str, params: dict[str, object]) -> None: + with pytest.raises(ValueError): + calculate(prices([10.0] * 60), name, **params) + + +@pytest.mark.parametrize("dtype,base", [(pl.Int64, 2**53), (pl.UInt64, 2**63), (pl.Int128, 2**126)]) +def test_cutler_preserves_integer_tick_differences_before_float_conversion(dtype, base) -> None: + frame = pl.DataFrame({"close": pl.Series([base, base + 3, base + 2, base + 4], dtype=dtype)}) + from ml4t.engineer.features.rolling import rsi_cutler + + assert frame.select(rsi_cutler("close", period=3)).item(-1, 0) == pytest.approx(100 * 5 / 6) + + +@pytest.mark.parametrize( + "name", + [ + "finite_mean", + "finite_weighted_mean", + "average_dollar_volume", + "linear_weighted_mean", + "finite_ewma", + "finite_ewm_std", + "rsi_cutler", + "finite_max_drawdown", + ], +) +def test_all_missing_window_does_not_create_an_observed_quote(name: str) -> None: + actual = calculate(prices([None] * 4), name, period=3)["factor"].to_numpy() + expected = ( + 0 + if name in {"average_dollar_volume", "linear_weighted_mean", "finite_weighted_mean"} + else np.nan + ) + np.testing.assert_allclose(actual[-1], expected, equal_nan=True) + + +@pytest.mark.parametrize("factor", [0.0, -1.0, float("nan"), float("inf"), True]) +def test_population_volatility_requires_explicit_positive_scale(factor: float) -> None: + with pytest.raises(ValueError, match="annualization_factor"): + calculate( + prices([10.0] * 4).with_columns(pl.lit(0.1).alias("returns")), + "annualized_volatility", + period=3, + annualization_factor=factor, + ) + + +@pytest.mark.parametrize("dtype,base", [(pl.Int64, 2**53), (pl.UInt64, 2**63), (pl.Int128, 2**126)]) +def test_finite_weighted_std_preserves_tiny_integer_price_variation(dtype, base) -> None: + frame = pl.DataFrame({"close": pl.Series([base, base + 1, base + 2], dtype=dtype)}) + frame = frame.with_columns( + *(pl.col("close").alias(c) for c in ("open", "high", "low")), pl.lit(1.0).alias("volume") + ) + assert calculate(frame, "finite_ewm_std", period=3, decay_rate=1.0)["factor"][-1] == 1.0 + + +def test_local_variance_does_not_lose_ticks_after_an_earlier_price_level_change() -> None: + frame = pl.DataFrame({"close": pl.Series([1, 2**63, 2**63 + 1, 2**63 + 2], dtype=pl.UInt64)}) + from ml4t.engineer.features.rolling import finite_ewm_std + + assert frame.select(finite_ewm_std("close", period=3, decay_rate=1.0)).item(-1, 0) == 1.0 + + +def test_finite_weighted_moments_do_not_overflow_representable_float_results() -> None: + from ml4t.engineer.features.rolling import finite_ewm_std, finite_ewma + + constant = pl.DataFrame({"close": [1e308] * 3}) + assert constant.select(finite_ewma("close", period=3, decay_rate=1)).item( + -1, 0 + ) == pytest.approx(1e308) + spread = pl.DataFrame({"close": [-1e300, 1e300]}) + assert spread.select(finite_ewm_std("close", period=2, decay_rate=1)).item( + -1, 0 + ) == pytest.approx(math.sqrt(2) * 1e300) + + +@pytest.mark.parametrize("name", ["finite_true_range", "fast_stochastic", "finite_aroon"]) +def test_normalized_and_range_methods_do_not_collapse_integer_price_ticks(name: str) -> None: + base = 2**63 + frame = pl.DataFrame( + { + "close": pl.Series([base + 2, base + 3, base + 4], dtype=pl.UInt64), + "high": pl.Series([base + 3, base + 4, base + 5], dtype=pl.UInt64), + "low": pl.Series([base, base + 1, base + 2], dtype=pl.UInt64), + } + ).with_columns(pl.col("close").alias("open"), pl.lit(1.0).alias("volume")) + result = calculate(frame, name, **({} if name == "finite_true_range" else {"period": 3})) + if name == "finite_true_range": + np.testing.assert_allclose(result["factor"].to_numpy(), [np.nan, 3, 3], equal_nan=True) + elif name == "fast_stochastic": + assert result["factor"][-1] == 80 + else: + assert result["factor_up"][-1] == 100 + assert result["factor_down"][-1] == 0 + + +@pytest.mark.parametrize("decay", [1e-10, 1e-50, 1e-100]) +def test_two_observation_weighted_std_retains_exact_degrees_of_freedom(decay: float) -> None: + # For any two positive weights, the unbiased variance is (x1-x0)**2/2. + actual = calculate(prices([1.0, 5.0]), "finite_ewm_std", period=2, decay_rate=decay) + assert actual["factor"][-1] == pytest.approx(math.sqrt(8), rel=1e-14, abs=0) diff --git a/tests/compat/test_zipline_relationship_features.py b/tests/compat/test_zipline_relationship_features.py new file mode 100644 index 00000000..3935878a --- /dev/null +++ b/tests/compat/test_zipline_relationship_features.py @@ -0,0 +1,164 @@ +"""Paired rolling factors must use aligned target rows, not time-trend regression.""" + +from __future__ import annotations + +import importlib + +import numpy as np +import polars as pl +import pytest +from polars.testing import assert_frame_equal + + +def paired(): + return importlib.import_module("ml4t.engineer.features.paired") + + +@pytest.mark.parametrize("method", ["pearson", "spearman", "beta"]) +@pytest.mark.parametrize("lazy", [False, True]) +def test_paired_methods_match_independent_complete_window_oracles(method: str, lazy: bool) -> None: + frame = pl.DataFrame({"x": [1.0, 3.0, 2.0, 4.0, 6.0], "y": [2.0, 8.0, 7.0, 9.0, 10.0]}) + expression = getattr(paired(), f"rolling_{method}")("y", "x", period=3).alias("result") + result = (frame.lazy() if lazy else frame).select(expression) + if lazy: + assert isinstance(result, pl.LazyFrame) + result = result.collect() + expected = [np.nan, np.nan] + for end in range(2, len(frame)): + x, y = frame["x"].to_numpy()[end - 2 : end + 1], frame["y"].to_numpy()[end - 2 : end + 1] + if method == "pearson": + expected.append(np.corrcoef(x, y)[0, 1]) + elif method == "spearman": + expected.append(np.corrcoef(np.argsort(np.argsort(x)), np.argsort(np.argsort(y)))[0, 1]) + else: + expected.append(np.cov(x, y)[0, 1] / np.var(x, ddof=1)) + np.testing.assert_allclose(result["result"].to_numpy(), expected, atol=1e-14, equal_nan=True) + + +@pytest.mark.parametrize("method", ["pearson", "spearman", "beta"]) +def test_missing_pair_is_removed_from_both_sides_and_full_row_window_remains(method: str) -> None: + frame = pl.DataFrame({"x": [1.0, 90.0, 3.0, 4.0], "y": [2.0, None, 6.0, 8.0]}) + function = getattr(paired(), f"rolling_{method}") + actual = frame.select(function("y", "x", period=4, allowed_missing=1).alias("result")) + np.testing.assert_allclose( + actual["result"].to_numpy(), [np.nan] * 3 + [2 if method == "beta" else 1], equal_nan=True + ) + strict = frame.select(function("y", "x", period=4).alias("result")).item(-1, 0) + assert strict is None or np.isnan(strict) + other_side = pl.DataFrame({"x": [1.0, None, 3.0, 4.0], "y": [2.0, 90.0, 6.0, 8.0]}) + assert other_side.select(function("y", "x", period=4, allowed_missing=1)).item( + -1, 0 + ) == pytest.approx(2 if method == "beta" else 1) + + +def test_spearman_recomputes_average_tie_ranks_in_each_window() -> None: + frame = pl.DataFrame({"x": [1.0, 1.0, 4.0, 2.0, 3.0], "y": [1.0, 2.0, 2.0, 4.0, 1.0]}) + actual = frame.select(paired().rolling_spearman("y", "x", period=3))["y"].to_numpy() + np.testing.assert_allclose(actual[2:], [0.5, 0.0, -0.5], atol=1e-14) + + +@pytest.mark.parametrize("lazy", [False, True]) +def test_rolling_regression_returns_all_five_actual_paired_fields(lazy: bool) -> None: + stats = pytest.importorskip("scipy.stats", reason="Regression p-values require stats extra") + frame = pl.DataFrame({"x": [1.0, 3.0, 2.0, 4.0, 6.0], "y": [2.0, 8.0, 7.0, 9.0, 10.0]}) + result = ( + (frame.lazy() if lazy else frame) + .select(paired().rolling_regression("y", "x", period=3).alias("regression")) + .unnest("regression") + ) + if lazy: + result = result.collect() + assert result.columns == ["alpha", "beta", "r_value", "p_value", "stderr"] + assert all(result[c].head(2).is_nan().all() for c in result.columns) + for end in range(2, frame.height): + reference = stats.linregress(frame["x"].slice(end - 2, 3), frame["y"].slice(end - 2, 3)) + np.testing.assert_allclose( + result.row(end), + [ + reference.intercept, + reference.slope, + reference.rvalue, + reference.pvalue, + reference.stderr, + ], + rtol=1e-13, + atol=0, + ) + + +def test_regression_missing_pairs_and_constant_target_are_explicit() -> None: + pytest.importorskip("scipy", reason="Regression p-values require stats extra") + frame = pl.DataFrame({"x": [1.0, None, 3.0, 4.0], "y": [3.0, 100.0, 7.0, 9.0]}) + result = frame.select( + paired().rolling_regression("y", "x", period=4, allowed_missing=1).alias("r") + ).unnest("r") + assert result["alpha"][-1] == pytest.approx(1) + assert result["beta"][-1] == pytest.approx(2) + constant = frame.with_columns(pl.lit(1.0).alias("x")) + result = constant.select(paired().rolling_regression("y", "x", period=3).alias("r")).unnest("r") + assert all(np.isnan(result[c][-1]) for c in result.columns) + + +def test_regression_has_independent_slope_intercept_pvalue_and_stderr_oracle() -> None: + pytest.importorskip("scipy", reason="Regression p-values require stats extra") + frame = pl.DataFrame({"x": [1.0, 2.0, 3.0, 4.0], "y": [1.0, 2.0, 2.0, 4.0]}) + actual = frame.select(paired().rolling_regression("y", "x", period=4).alias("r")).unnest("r") + # Sxx=5, Sxy=4.5, Syy=4.75, residual SS=.7, df=2. + # The two-sided Student-t df=2 tail is 1-|t|/sqrt(t*t+2). + expected = [0.0, 0.9, 9 / np.sqrt(95), 1 - 9 / np.sqrt(95), np.sqrt(0.07)] + np.testing.assert_allclose(actual.row(-1), expected, rtol=1e-13, atol=1e-15) + + +@pytest.mark.parametrize("method", ["pearson", "spearman", "beta", "regression"]) +def test_empty_and_singleton_pair_series_remain_empty_or_undefined(method: str) -> None: + if method == "regression": + pytest.importorskip("scipy", reason="Regression p-values require stats extra") + function = getattr(paired(), f"rolling_{method}") + for count in (0, 1): + frame = pl.DataFrame({"x": [1.0] * count, "y": [2.0] * count}) + actual = frame.select(function("y", "x", period=3).alias("r")) + assert actual.height == count + if count: + if method == "regression": + assert all(np.isnan(v) for v in actual["r"][0].values()) + else: + assert actual["r"][0] is None or np.isnan(actual["r"][0]) + + +@pytest.mark.parametrize("method", ["pearson", "spearman", "beta", "regression"]) +def test_paired_methods_do_not_cross_assets_or_mutate_inputs(method: str) -> None: + if method == "regression": + pytest.importorskip("scipy", reason="Regression p-values require stats extra") + frame = pl.DataFrame( + { + "asset": ["A", "B"] * 4, + "x": [1.0, 10.0, 3.0, 20.0, 2.0, 40.0, 4.0, 30.0], + "y": [2.0, 30.0, 8.0, 10.0, 7.0, 50.0, 9.0, 20.0], + } + ) + before = frame.clone() + expression = getattr(paired(), f"rolling_{method}")("y", "x", period=3) + combined = frame.with_columns(expression.over("asset").alias("result")) + for asset in ("A", "B"): + separate = frame.filter(pl.col("asset") == asset).with_columns(expression.alias("result")) + assert_frame_equal(combined.filter(pl.col("asset") == asset), separate) + assert_frame_equal(frame, before) + + +@pytest.mark.parametrize("method", ["pearson", "spearman", "beta", "regression"]) +@pytest.mark.parametrize( + "params", + [ + {"period": 0}, + {"period": 1}, + {"period": True}, + {"period": 3, "allowed_missing": -1}, + {"period": 3, "allowed_missing": 4}, + {"period": 3, "allowed_missing": True}, + ], +) +def test_invalid_pair_windows_fail_before_evaluation( + method: str, params: dict[str, object] +) -> None: + with pytest.raises(ValueError): + getattr(paired(), f"rolling_{method}")("y", "x", **params) diff --git a/tests/compat/test_zipline_return_factor_alignment.py b/tests/compat/test_zipline_return_factor_alignment.py new file mode 100644 index 00000000..b803ca9e --- /dev/null +++ b/tests/compat/test_zipline_return_factor_alignment.py @@ -0,0 +1,428 @@ +"""Actual price->target return->rolling factor preparation preserves keys and clocks.""" + +import importlib +from datetime import UTC, date, datetime, timedelta + +import numpy as np +import polars as pl +import pytest +from ml4t.specs import FeedSpec +from polars.testing import assert_frame_equal + +DAYS = [date(2025, 7, day) for day in (7, 8, 9, 10, 11, 14, 15, 16)] + + +def feed(**options) -> FeedSpec: + defaults = { + "time_contract_version": 2, + "calendar": None, + "entity_col": "asset", + "data_frequency": "1d", + "bar_type": "time", + "timestamp_semantics": "session_label", + } + return FeedSpec(**{**defaults, **options}) + + +def run(frame, **options): + function = importlib.import_module("ml4t.engineer.features.paired").target_return_features + defaults = { + "target": {"asset": "B"}, + "period": 3, + "method": "beta", + "price_basis": "raw", + "feed_spec": feed(), + "output": "market", + } + return function(frame, **{**defaults, **options}) + + +def panel() -> pl.DataFrame: + return pl.DataFrame( + { + "timestamp": DAYS[:4] * 2, + "asset": ["A"] * 4 + ["B"] * 4, + "close": [100.0, 120.0, 80.0, 112.0, 100.0, 110.0, 99.0, 118.8], + } + ) + + +@pytest.mark.parametrize("lazy", [False, True]) +def test_actual_price_pipeline_preserves_exact_target_keys_and_independent_beta(lazy: bool) -> None: + frame = panel().sort("timestamp", "asset", descending=[False, True]) + result = run(frame.lazy() if lazy else frame) + assert isinstance(result, pl.LazyFrame if lazy else pl.DataFrame) + if lazy: + result = result.collect() + assert_frame_equal(result.select(frame.columns), frame) + a = result.filter(pl.col("asset") == "A") + np.testing.assert_allclose( + a["market_asset_return"].to_numpy(), + [np.nan, 0.2, -1 / 3, 0.4], + rtol=1e-14, + atol=0, + equal_nan=True, + ) + np.testing.assert_allclose( + a["market_target_return"].to_numpy(), + [np.nan, 0.1, -0.1, 0.2], + rtol=1e-14, + atol=0, + equal_nan=True, + ) + assert a["market_beta"][-1] == pytest.approx(52 / 21, rel=1e-13) + assert result.filter(pl.col("asset") == "B")["market_beta"][-1] == pytest.approx(1.0, rel=1e-13) + assert result["market_available_at"].null_count() == result.height + assert result["market_price_basis"].unique().to_list() == ["raw"] + + +def test_common_clock_is_not_compressed_before_missing_tolerant_statistics() -> None: + close = [100.0, 110.0, 121.0, 108.9, 130.68, 143.748, 129.3732, 155.24784] + b = pl.DataFrame({"timestamp": DAYS, "asset": "B", "close": close}) + a = b.filter(~pl.col("timestamp").is_in(DAYS[3:5])).with_columns(pl.lit("A").alias("asset")) + frame = pl.concat([a, b]).sort("timestamp", "asset") + common = run(frame, period=4, allowed_missing=1, index_mode="common_grid") + per_asset = run(frame, period=4, allowed_missing=1, index_mode="per_asset") + at = (pl.col("asset") == "A") & (pl.col("timestamp") == DAYS[6]) + value = common.filter(at)["market_beta"][0] + assert value is None or np.isnan(value) + assert np.isfinite(per_asset.filter(at)["market_beta"][0]) + missing_exit = common.filter((pl.col("asset") == "A") & (pl.col("timestamp") == DAYS[5])) + assert missing_exit["market_asset_return"][0] is None + assert per_asset.filter((pl.col("asset") == "A") & (pl.col("timestamp") == DAYS[5]))[ + "market_asset_return" + ][0] == pytest.approx(0.188, rel=1e-14) + assert common.height == frame.height + + +def test_named_grid_fails_before_a_gap_can_become_a_one_bar_return() -> None: + frame = panel().filter(pl.col("timestamp") != DAYS[1]) + before = frame.clone() + with pytest.raises(ValueError, match="missing scheduled session"): + run(frame, calendar="NYSE") + assert_frame_equal(frame, before) + + +def test_feature_availability_uses_both_assets_and_the_actual_price_dependencies() -> None: + frame = ( + panel() + .with_columns( + (pl.col("timestamp").cast(pl.Datetime("ns", "UTC")) + timedelta(hours=16)).alias( + "known" + ) + ) + .with_columns( + pl.when((pl.col("asset") == "B") & (pl.col("timestamp") == DAYS[3])) + .then(pl.col("known") + timedelta(hours=4)) + .otherwise(pl.col("known")) + .alias("known") + ) + ) + result = run(frame, available_at_col="known") + assert result.filter(pl.col("asset") == "A")["market_available_at"][-1] == datetime( + 2025, 7, 10, 20, tzinfo=UTC + ) + earlier = run(frame.filter(pl.col("timestamp") <= DAYS[2]), method="pearson") + assert earlier["market_available_at"].null_count() == earlier.height + + +def test_custom_roles_compound_identity_and_regression_fields_survive_the_pipeline() -> None: + pytest.importorskip("scipy", reason="Regression p-values require stats extra") + frame = ( + panel() + .with_columns( + pl.struct(pl.lit("bundle").alias("namespace"), pl.col("asset").alias("sid")).alias( + "asset" + ) + ) + .rename({"timestamp": "when", "asset": "identity", "close": "quote"}) + ) + contract = FeedSpec( + time_contract_version=2, + calendar=None, + entity_col="identity", + timestamp_col="when", + price_col="quote", + timestamp_semantics="session_label", + data_frequency="1d", + ) + output = run( + frame, + feed_spec=contract, + target={"identity": {"namespace": "bundle", "sid": "B"}}, + method="regression", + ) + assert output.schema["identity"] == frame.schema["identity"] + row = output.filter(pl.col("identity").struct.field("sid") == "A")["market_regression"][-1] + assert row["beta"] == pytest.approx(52 / 21, rel=1e-13) + assert row["alpha"] == pytest.approx(-8 / 105, rel=1e-13) + + +@pytest.mark.parametrize( + "options", + [ + {"target": {"asset": "missing"}}, + {"target": {}}, + {"period": True}, + {"method": "time_trend"}, + {"price_basis": ""}, + {"calendar": "NYSE", "feed_spec": FeedSpec()}, + {"entity_col": "another"}, + {"timestamp_col": "another"}, + {"price_col": "another"}, + {"index_mode": "ffill"}, + {"window_unit": "elapsed"}, + ], +) +def test_target_contract_errors_are_atomic(options) -> None: + frame = panel() + before = frame.clone() + with pytest.raises((TypeError, ValueError)): + run(frame, **options) + assert_frame_equal(frame, before) + + +@pytest.mark.parametrize("method", ["pearson", "spearman", "beta", "regression"]) +def test_empty_frame_retains_declared_added_schema(method: str) -> None: + if method == "regression": + pytest.importorskip("scipy", reason="Regression p-values require stats extra") + for frame in (panel().clear(), panel().clear().lazy()): + result = run(frame, method=method) + result = result.collect() if isinstance(result, pl.LazyFrame) else result + assert result.height == 0 + assert result.schema["market_available_at"] == pl.Datetime("ns", "UTC") + assert result.schema["market_" + method].base_type() == ( + pl.Struct if method == "regression" else pl.Float64 + ) + + +def test_appended_future_prices_do_not_change_prior_prepared_values() -> None: + before = panel() + later = pl.DataFrame({"timestamp": [DAYS[4]] * 2, "asset": ["A", "B"], "close": [1e9, 0.001]}) + expected = run(before) + actual = run(pl.concat([before, later])).filter(pl.col("timestamp") <= DAYS[3]) + assert_frame_equal(actual, expected) + # Lazy filtering must not remove warmup or the target before preparation. + selected = ( + run(before.lazy()) + .filter((pl.col("asset") == "A") & (pl.col("timestamp") == DAYS[3])) + .collect() + ) + assert selected["market_beta"][0] == pytest.approx(52 / 21) + + +def test_adjacent_uint64_ticks_survive_actual_price_to_factor_preparation() -> None: + frame = panel().with_columns( + pl.Series("close", [2**63 + v for v in (0, 2, 6, 12, 0, 1, 3, 6)], dtype=pl.UInt64) + ) + result = run(frame) + a = result.filter(pl.col("asset") == "A") + np.testing.assert_allclose( + a["market_asset_return"].tail(3), [2 / 2**63, 4 / (2**63 + 2), 6 / (2**63 + 6)], rtol=1e-14 + ) + assert a["market_beta"][-1] == pytest.approx(2.0, rel=1e-13) + + +@pytest.mark.parametrize("bad", [None, float("nan"), float("inf"), 0.0, -1.0]) +def test_invalid_supplied_prices_raise_or_become_explicit_missing_pairs(bad) -> None: + frame = panel().with_columns( + pl.when((pl.col("asset") == "A") & (pl.col("timestamp") == DAYS[1])) + .then(pl.lit(bad)) + .otherwise(pl.col("close")) + .alias("close") + ) + with pytest.raises(ValueError, match="finite, positive"): + run(frame) + actual = run(frame, missing_price_policy="null") + a = actual.filter(pl.col("asset") == "A") + assert a["market_asset_return"][1] is None + assert a["market_asset_return"][2] is None + assert not np.isfinite(a["market_beta"][-1]) if a["market_beta"][-1] is not None else True + + +def test_multicolumn_identity_does_not_join_reused_tickers_across_namespaces() -> None: + frame = pl.concat( + [ + panel().with_columns(pl.lit("one").alias("venue")), + panel().with_columns(pl.lit("two").alias("venue")), + ] + ) + contract = FeedSpec(time_contract_version=2, calendar=None, entity_col=["venue", "asset"]) + output = run(frame, feed_spec=contract, target={"venue": "one", "asset": "B"}) + assert_frame_equal(output.select(frame.columns), frame) + assert output.filter((pl.col("asset") == "A") & (pl.col("venue") == "two"))["market_beta"][ + -1 + ] == pytest.approx(52 / 21) + + +@pytest.mark.parametrize( + "options", + [ + {"returns_periods": 0}, + {"period": 2**64}, + {"returns_periods": 2**64}, + {"output": ""}, + {"missing_price_policy": "ffill"}, + {"entity_col": []}, + {"target": {"asset": None}}, + {"target": {"asset": "B", "other": 1}}, + {"window_unit": "expected_bar"}, + {"allowed_missing": -1}, + {"feed_spec": {"timestamp_col": "timestamp", "time_col": "other"}}, + {"feed_spec": None, "entity_col": []}, + {"feed_spec": None, "price_col": "timestamp"}, + ], +) +def test_invalid_preparation_options_fail_before_mutating_input(options) -> None: + before = panel() + with pytest.raises((TypeError, ValueError)): + run(before, **options) + assert_frame_equal(before, panel()) + + +@pytest.mark.parametrize( + "kind", + [ + "duplicate", + "null_key", + "missing_column", + "string_time", + "string_price", + "collision", + "internal", + ], +) +def test_invalid_physical_frames_fail_with_explicit_errors(kind: str) -> None: + frame = panel() + if kind == "duplicate": + frame = pl.concat([frame, frame.head(1)]) + elif kind == "null_key": + frame = frame.with_columns(pl.lit(None).alias("asset")) + elif kind == "missing_column": + frame = frame.drop("close") + elif kind == "string_time": + frame = frame.with_columns(pl.col("timestamp").cast(pl.String)) + elif kind == "string_price": + frame = frame.with_columns(pl.col("close").cast(pl.String)) + else: + frame = frame.with_columns( + pl.lit(1).alias("market_beta" if kind == "collision" else "__ml4t_target_quote") + ) + with pytest.raises(ValueError): + run(frame) + + +def test_unknown_and_out_of_range_publication_instants_are_not_fabricated_or_wrapped() -> None: + frame = panel().with_columns(pl.lit(None, pl.Datetime("ns", "UTC")).alias("known")) + assert run(frame, available_at_col="known")["market_available_at"].null_count() == frame.height + naive = frame.with_columns(pl.lit(datetime(2025, 7, 7)).alias("known")) + with pytest.raises(ValueError, match="timezone-aware"): + run(naive, available_at_col="known") + wide = frame.with_columns(pl.lit(datetime(2500, 1, 1, tzinfo=UTC)).alias("known")) + with pytest.raises(ValueError, match="range"): + run(wide, available_at_col="known") + + +def test_named_calendar_and_explicit_none_precedence_reuse_the_actual_grid_validator() -> None: + frame = panel() + contract = feed(calendar="NYSE") + actual = run(frame, feed_spec=contract) + assert actual["market_beta"][-1] == pytest.approx(1.0) + sparse = frame.filter(pl.col("timestamp") != DAYS[1]) + with pytest.raises(ValueError, match="missing scheduled session"): + run(sparse, feed_spec=contract) + result = run(sparse, feed_spec=contract, calendar=None) + assert result.height == sparse.height + result = run(frame, feed_spec=None, calendar=None, entity_col="asset") + assert result.filter(pl.col("asset") == "A")["market_beta"][-1] == pytest.approx(52 / 21) + + +@pytest.mark.parametrize( + "name,method,length", + [ + ("RollingPearsonOfReturns", "pearson", "correlation_length"), + ("RollingSpearmanOfReturns", "spearman", "correlation_length"), + ("RollingLinearRegressionOfReturns", "regression", "regression_length"), + ("SimpleBeta", "beta", "regression_length"), + ], +) +@pytest.mark.parametrize("lazy", [False, True]) +def test_return_factor_recipe_uses_native_keyed_preparation_without_masking_target( + name, method, length, lazy +) -> None: + if method == "regression": + pytest.importorskip("scipy", reason="Regression p-values require stats extra") + function = importlib.import_module("ml4t.engineer.compat.zipline").return_factor_features + frame = panel() + output = function( + frame.lazy() if lazy else frame, + name, + target={"asset": "B"}, + price_basis="raw", + feed_spec=feed(), + output="market", + mask=pl.col("asset") == "A", + **{length: 3}, + ) + assert isinstance(output, pl.LazyFrame if lazy else pl.DataFrame) + output = output.collect() if lazy else output + expected = run(frame, method=method) + assert_frame_equal( + output.filter(pl.col("asset") == "A"), expected.filter(pl.col("asset") == "A") + ) + assert output.filter(pl.col("asset") == "B")["market_" + method].null_count() == 4 + assert_frame_equal(output.select(frame.columns), frame) + + +def test_return_recipe_translates_price_lookback_and_retains_structural_warmup() -> None: + function = importlib.import_module("ml4t.engineer.compat.zipline").return_factor_features + close = [100.0, 110.0, 121.0, 108.9, 130.68, 143.748, 129.3732, 155.24784] + frame = pl.DataFrame( + {"timestamp": DAYS * 2, "asset": ["A"] * 8 + ["B"] * 8, "close": close + close} + ) + result = function( + frame, + "RollingPearsonOfReturns", + target={"asset": "B"}, + returns_length=3, + correlation_length=3, + price_basis="raw", + feed_spec=feed(), + output="market", + ) + a = result.filter(pl.col("asset") == "A") + assert a["market_asset_return"][2] == pytest.approx(0.21) + assert a["market_pearson"].head(4).is_null().all() + np.testing.assert_allclose(a["market_pearson"].tail(4), 1.0, atol=1e-14) + beta = function( + frame, + "SimpleBeta", + target={"asset": "B"}, + regression_length=4, + allowed_missing_percentage=1.0, + price_basis="raw", + feed_spec=feed(), + ) + assert beta.filter(pl.col("asset") == "A")["target_beta"].head(4).is_null().all() + + +@pytest.mark.parametrize( + "name,options", + [ + ("unknown", {}), + ("SimpleBeta", {"regression_length": 3, "returns_length": 3}), + ("SimpleBeta", {"regression_length": 2}), + ("SimpleBeta", {"regression_length": 3, "allowed_missing_percentage": -1}), + ("RollingPearsonOfReturns", {"correlation_length": 3, "returns_length": 1}), + ("RollingPearsonOfReturns", {"regression_length": 3}), + ("RollingSpearmanOfReturns", {"correlation_length": 3, "allowed_missing_percentage": 0.2}), + ("RollingLinearRegressionOfReturns", {"regression_length": 3, "correlation_length": 3}), + ("SimpleBeta", {"regression_length": 3, "mask": 17}), + ], +) +def test_invalid_legacy_return_options_fail_explicitly(name, options) -> None: + function = importlib.import_module("ml4t.engineer.compat.zipline").return_factor_features + with pytest.raises((TypeError, ValueError)): + function( + panel(), name, target={"asset": "B"}, price_basis="raw", feed_spec=feed(), **options + ) diff --git a/tests/compat/test_zipline_selection.py b/tests/compat/test_zipline_selection.py new file mode 100644 index 00000000..e103656c --- /dev/null +++ b/tests/compat/test_zipline_selection.py @@ -0,0 +1,366 @@ +"""Dated eligible cross-sections preserve stable ties and original missingness.""" + +import importlib +from datetime import date + +import numpy as np +import polars as pl +import pytest +from polars.testing import assert_frame_equal + + +def selection(): + return importlib.import_module("ml4t.engineer.features.selection") + + +def panel(): + return pl.DataFrame( + { + "timestamp": [date(2025, 7, 7)] * 6, + "asset": ["D", "B", "A", "C", "E", "F"], + "value": [3.0, 1.0, 1.0, 2.0, float("nan"), 999.0], + "sector": ["x", "x", "x", "y", "x", "x"], + "eligible": [True] * 5 + [False], + } + ) + + +def calculate(frame, method, **options): + return frame.with_columns( + selection() + .cross_section("value", method=method, groupby="sector", mask="eligible", **options) + .alias("result") + ) + + +@pytest.mark.parametrize("lazy", [False, True]) +@pytest.mark.parametrize( + "ties,expected", + [ + ("ordinal", [3.0, 2.0, 1.0, 1.0]), + ("min", [3.0, 1.0, 1.0, 1.0]), + ("max", [3.0, 2.0, 2.0, 1.0]), + ("dense", [2.0, 1.0, 1.0, 1.0]), + ("average", [3.0, 1.5, 1.5, 1.0]), + ], +) +def test_rank_ties_use_stable_ids_inside_the_actual_eligible_group(lazy, ties, expected): + frame = panel() + actual = calculate(frame.lazy() if lazy else frame, "rank", ties=ties) + assert isinstance(actual, pl.LazyFrame if lazy else pl.DataFrame) + actual = actual.collect() if lazy else actual + np.testing.assert_allclose(actual["result"].head(4), expected) + assert actual["result"].tail(2).null_count() == 2 + assert_frame_equal(actual.select(frame.columns), frame) + shuffled = calculate(frame.reverse(), "rank", ties=ties).sort("asset") + assert_frame_equal(shuffled, actual.sort("asset")) + + +@pytest.mark.parametrize( + "method,expected", + [ + ("demean", [4 / 3, -2 / 3, -2 / 3, 0.0]), + ("zscore", [np.sqrt(2), -1 / np.sqrt(2), -1 / np.sqrt(2), np.nan]), + ], +) +def test_group_transforms_use_only_contemporary_eligible_values(method, expected): + output = calculate(panel(), method) + np.testing.assert_allclose(output["result"].head(4), expected, rtol=1e-14, equal_nan=True) + assert output["result"].tail(2).null_count() == 2 + future = panel().with_columns( + pl.lit(date(2025, 7, 8)).alias("timestamp"), (pl.col("value") * 100).alias("value") + ) + combined = calculate(pl.concat([panel(), future]), method).head(6) + assert_frame_equal(output, combined) + + +def test_undefined_numeric_classification_is_not_a_real_contemporary_group(): + frame = ( + panel() + .with_columns(pl.when(pl.col("sector") == "x").then(0.0).otherwise(1.0).alias("sector")) + .with_columns((pl.col("sector") / pl.col("sector")).alias("sector")) + ) + result = calculate(frame, "rank") + assert result["result"].to_list() == [None, None, None, 1.0, None, None] + + +@pytest.mark.parametrize( + "method,expected", + [ + ("top", [True, False, False, True, False, False]), + ("bottom", [False, False, True, True, False, False]), + ], +) +def test_top_bottom_use_exact_cardinality_and_stable_tie_selection(method, expected): + output = calculate(panel(), method, n=1) + assert output["result"].to_list() == expected + assert calculate(panel(), method, n=0)["result"].to_list() == [False] * 6 + + +def test_negative_group_maximum_uses_numeric_order_not_float_storage_bits(): + frame = panel().with_columns((-pl.col("value")).alias("value")) + result = calculate(frame, "top", n=1) + assert result["result"].to_list() == [False, False, True, True, False, False] + + +def test_winsorization_uses_legacy_order_statistic_cutoffs_not_linear_clipping(): + frame = pl.DataFrame( + { + "timestamp": [date(2025, 7, 7)] * 5, + "asset": list("ABCDE"), + "value": [0.0, 1.0, 2.0, 100.0, np.nan], + } + ) + result = frame.select( + selection().cross_section( + "value", method="winsorize", min_percentile=0.25, max_percentile=0.75 + ) + ) + np.testing.assert_allclose(result.to_series(), [1.0, 1.0, 2.0, 2.0, np.nan], equal_nan=True) + + +def test_quantiles_use_linear_edges_and_percentile_filter_includes_both_bounds(): + frame = pl.DataFrame( + { + "timestamp": [date(2025, 7, 7)] * 6, + "asset": list("ABCDEF"), + "value": [0.0, 1.0, 2.0, 3.0, np.nan, None], + } + ) + result = frame.with_columns( + selection().cross_section("value", method="quantiles", bins=2).alias("quantile"), + selection() + .cross_section( + "value", method="percentile_between", min_percentile=0.25, max_percentile=0.75 + ) + .alias("between"), + ) + assert result["quantile"].to_list() == [0, 0, 1, 1, None, None] + assert result["between"].to_list() == [False, True, True, False, False, False] + + +@pytest.mark.parametrize("lazy", [False, True]) +@pytest.mark.parametrize( + "values,bins,selected", + [ + ([-1e308, 0.0, 1e308], [0, 0, 1], [False, True, False]), + ([-1e308, 1e308], [0, 1], [False, False]), + ([-1e100, 0.0, 1e-100, 1e100], [0, 0, 1, 1], [False, True, True, False]), + ], +) +def test_linear_quantile_selection_preserves_finite_opposite_extremes_and_local_scale( + lazy, values, bins, selected +): + frame = pl.DataFrame( + {"timestamp": [1] * len(values), "asset": range(len(values)), "value": values} + ) + result = (frame.lazy() if lazy else frame).with_columns( + selection().cross_section("value", method="quantiles", bins=2).alias("bin"), + selection() + .cross_section( + "value", method="percentile_between", min_percentile=0.25, max_percentile=0.75 + ) + .alias("selected"), + ) + result = result.collect() if lazy else result + assert result["bin"].to_list() == bins + assert result["selected"].to_list() == selected + + +def test_integer_percentile_bounds_do_not_truncate_fractional_value_edges(): + frame = pl.DataFrame({"timestamp": [1] * 3, "asset": [1, 2, 3], "value": [-0.5, 0.25, 0.75]}) + expression = selection().cross_section( + "value", method="percentile_between", min_percentile=0, max_percentile=1 + ) + assert frame.select(expression).to_series().to_list() == [True, True, True] + assert frame.clear().select(expression).schema["value"] == pl.Boolean + + +@pytest.mark.parametrize("method,options", [("quantiles", {"bins": 2}), ("percentile_between", {})]) +def test_nonfinite_quantile_inputs_fail_explicitly_without_partial_selection(method, options): + frame = pl.DataFrame({"timestamp": [1, 1], "asset": [1, 2], "value": [1.0, float("inf")]}) + with pytest.raises(ValueError, match="finite"): + frame.with_columns(selection().cross_section("value", method=method, **options)) + tied = frame.with_columns(pl.lit(1.0).alias("value")) + with pytest.raises((ValueError, pl.exceptions.DuplicateError)): + tied.select(selection().cross_section("value", method="quantiles", bins=4)) + + +def test_peer_counts_keep_supplied_classifier_nulls_and_exclude_ineligible_members(): + frame = panel().with_columns( + pl.when(pl.col("asset") == "E").then(None).otherwise(pl.col("sector")).alias("sector") + ) + output = frame.select(selection().cross_section("sector", method="peer_count", mask="eligible")) + np.testing.assert_allclose( + output.to_series(), [3.0, 3.0, 3.0, 1.0, np.nan, np.nan], equal_nan=True + ) + + +@pytest.mark.parametrize("method", ["demean", "zscore"]) +def test_adjacent_large_integer_cross_sections_do_not_lose_tick_dispersion(method): + frame = pl.DataFrame( + { + "timestamp": [date(2025, 7, 7)] * 3, + "asset": list("ABC"), + "value": pl.Series([2**63, 2**63 + 1, 2**63 + 2], dtype=pl.UInt64), + } + ) + result = frame.select(selection().cross_section("value", method=method)).to_series() + expected = [-1.0, 0.0, 1.0] if method == "demean" else [-np.sqrt(1.5), 0.0, np.sqrt(1.5)] + np.testing.assert_allclose(result, expected, rtol=1e-14, atol=0) + + +@pytest.mark.parametrize("method", ["any", "all", "at_least"]) +def test_rolling_boolean_filters_keep_full_window_and_explicit_missing_false(method): + frame = pl.DataFrame({"value": [True, None, False, True, True]}) + kwargs = {"minimum": 2} if method == "at_least" else {} + result = frame.select( + selection().rolling_filter("value", period=3, method=method, **kwargs) + ).to_series() + expected = ( + [None, None, True, True, True] + if method == "any" + else ( + [None, None, False, False, False] + if method == "all" + else [None, None, False, False, True] + ) + ) + assert result.to_list() == expected + + +@pytest.mark.parametrize( + "options", + [ + {"method": "unknown"}, + {"method": "rank", "ties": "random"}, + {"method": "top", "n": -1}, + {"method": "top", "n": True}, + {"method": "quantiles", "bins": 0}, + {"method": "quantiles", "bins": True}, + {"method": "winsorize", "min_percentile": 0.8, "max_percentile": 0.2}, + ], +) +def test_invalid_selection_contracts_fail_before_output(options): + with pytest.raises((ValueError, TypeError)): + panel().select(selection().cross_section("value", **options)) + + +@pytest.mark.parametrize("scale", [np.ldexp(1.0, -1000), np.ldexp(1.0, 1000)]) +def test_representable_group_zscores_do_not_underflow_or_overflow_in_variance(scale): + frame = pl.DataFrame( + { + "timestamp": [date(2025, 7, 7)] * 3, + "asset": list("ABC"), + "value": [scale, 2 * scale, 3 * scale], + } + ) + actual = frame.select(selection().cross_section("value", method="zscore")).to_series() + np.testing.assert_allclose(actual, [-np.sqrt(1.5), 0.0, np.sqrt(1.5)], rtol=1e-14, atol=0) + + +@pytest.mark.parametrize( + "method,expected", + [ + ("demean", [-1e308, 1e308]), + ("zscore", [-1.0, 1.0]), + ], +) +def test_opposite_extreme_group_values_do_not_overflow_a_representable_transform(method, expected): + frame = pl.DataFrame( + {"timestamp": [date(2025, 7, 7)] * 2, "asset": list("AB"), "value": [-1e308, 1e308]} + ) + actual = frame.select(selection().cross_section("value", method=method)).to_series() + np.testing.assert_allclose(actual, expected, rtol=1e-14, atol=0) + + +@pytest.mark.parametrize( + "dtype,values,expected", + [ + (pl.Float64, [1.0, np.nan, None, np.inf], [False, True, True, False]), + (pl.Int64, [1, -1, None, 0], [False, False, True, False]), + (pl.String, ["a", "", None, "-1"], [False, False, True, False]), + ], +) +def test_native_missing_predicate_does_not_invent_classifier_sentinels(dtype, values, expected): + actual = pl.DataFrame({"x": pl.Series(values, dtype=dtype)}).select(selection().is_missing("x")) + assert actual.to_series().to_list() == expected + + +@pytest.mark.parametrize( + "method", + ["rank", "demean", "zscore", "winsorize", "quantiles", "percentile_between", "peer_count"], +) +def test_empty_all_missing_and_singleton_cross_sections_keep_defined_shapes(method): + kwargs = {"bins": 1} if method == "quantiles" else {} + frame = pl.DataFrame( + {"timestamp": [date(2025, 7, 7)], "asset": ["A"], "value": [None]}, + schema_overrides={"value": pl.Float64}, + ) + for source in (frame, frame.clear()): + result = source.select(selection().cross_section("value", method=method, **kwargs)) + assert result.height == source.height + if source.height: + assert ( + result.item() is None if method != "percentile_between" else result.item() is False + ) + present = frame.with_columns(pl.lit(3.0).alias("value")) + result = present.select(selection().cross_section("value", method=method, **kwargs)).item() + if method == "zscore": + assert np.isnan(result) + else: + assert ( + result + == { + "rank": 1.0, + "demean": 0.0, + "winsorize": 3.0, + "quantiles": 0, + "percentile_between": True, + "peer_count": 1.0, + }[method] + ) + + +@pytest.mark.parametrize( + "options", + [ + {"method": "demean", "n": 1}, + {"method": "demean", "bins": 2}, + {"method": "top", "n": 1, "ties": "dense"}, + {"method": "rank", "ascending": 1}, + {"method": "zscore", "ddof": -1}, + {"method": "rank", "ddof": 1}, + {"method": "rank", "min_percentile": 0.1}, + {"method": "winsorize", "max_percentile": np.nan}, + {"method": "rank", "entity_col": []}, + {"method": "rank", "groupby": [""]}, + {"method": "rank", "mask": 42}, + ], +) +def test_inapplicable_or_invalid_options_are_not_silently_ignored(options): + with pytest.raises((ValueError, TypeError)): + panel().select(selection().cross_section("value", **options)) + + +@pytest.mark.parametrize( + "options", + [ + {"period": 0, "method": "any"}, + {"period": 2, "method": "unknown"}, + {"period": 2, "method": "all", "minimum": 1}, + {"period": 2, "method": "at_least", "minimum": 3}, + {"period": 2, "method": "at_least", "minimum": True}, + ], +) +def test_invalid_boolean_window_contracts_fail_explicitly(options): + with pytest.raises(ValueError): + pl.DataFrame({"x": [True, False]}).select(selection().rolling_filter("x", **options)) + + +def test_boolean_window_rejects_numeric_predicates_and_respects_asset_groups(): + with pytest.raises(ValueError, match="boolean"): + pl.DataFrame({"x": [1, 0]}).select(selection().rolling_filter("x", period=2, method="any")) + frame = pl.DataFrame({"asset": ["A", "B"] * 3, "x": [True, False, True, False, True, False]}) + result = frame.select(selection().rolling_filter("x", period=2, method="all").over("asset")) + assert result.to_series().to_list() == [None, None, True, False, True, False] diff --git a/tests/core/test_registry.py b/tests/core/test_registry.py index 243de103..8addbc5b 100644 --- a/tests/core/test_registry.py +++ b/tests/core/test_registry.py @@ -555,10 +555,10 @@ def test_registry_with_real_features(): # Test category queries work momentum = registry.list_by_category("momentum") - assert len(momentum) == 31 + assert len(momentum) == 36 volatility = registry.list_by_category("volatility") - assert len(volatility) == 15 + assert len(volatility) == 18 # Test stationarity queries stationary = registry.list_normalized() diff --git a/tests/test_all_features_integration.py b/tests/test_all_features_integration.py index d17b9497..13d60d93 100644 --- a/tests/test_all_features_integration.py +++ b/tests/test_all_features_integration.py @@ -46,7 +46,7 @@ def sample_ohlcv_data(): def test_all_features_registered(): """Verify that features are actually registered.""" registry = get_registry() - assert len(registry) == 120, f"Expected 120 features, got {len(registry)}" + assert len(registry) == 136, f"Expected 136 features, got {len(registry)}" def test_all_features_compute_individually(sample_ohlcv_data): diff --git a/tests/test_examples.py b/tests/test_examples.py index d0fa5981..5286f6a8 100644 --- a/tests/test_examples.py +++ b/tests/test_examples.py @@ -38,3 +38,17 @@ def test_advertised_example_runs(script: Path) -> None: f"stdout:\n{result.stdout}\n" f"stderr:\n{result.stderr}" ) + + +@pytest.mark.parametrize("name", ["pipeline_expression_ports.py", "pipeline_migration.py"]) +def test_pipeline_examples_run_with_windows_legacy_console_encoding(name: str) -> None: + result = subprocess.run( + [sys.executable, str(EXAMPLE_DIR / name)], + cwd=EXAMPLE_DIR.parent, + env={**os.environ, "PYTHONIOENCODING": "cp1252"}, + capture_output=True, + text=True, + timeout=60, + check=False, + ) + assert result.returncode == 0, result.stderr diff --git a/tests/test_feature_panel_preparation.py b/tests/test_feature_panel_preparation.py new file mode 100644 index 00000000..6df9cbb4 --- /dev/null +++ b/tests/test_feature_panel_preparation.py @@ -0,0 +1,735 @@ +"""Actual prepared histories become causal features without revision/adjustment selection.""" + +import importlib +import json +from datetime import UTC, date, datetime + +import polars as pl +import pytest +import yaml +from ml4t.specs import FeedSpec +from polars.testing import assert_frame_equal + +from ml4t.engineer.artifacts import FeatureSpec +from ml4t.engineer.features.paired import rolling_regression +from ml4t.engineer.features.rolling import finite_mean + +DAYS = [date(2025, 7, d) for d in (7, 8, 9, 10)] + + +def prepare(data, **options): + function = importlib.import_module("ml4t.engineer.features.preparation").prepare_feature_panel + defaults = { + "features": {"average": finite_mean("close", period=3)}, + "lookbacks": {"average": 3}, + "input_columns": ["close"], + "decisions": decisions(), + "universe": universe(), + "price_basis": "raw", + "feed_spec": FeedSpec( + time_contract_version=2, + calendar=None, + entity_col="asset", + timestamp_semantics="session_label", + ), + } + return function(data, **{**defaults, **options}) + + +def history(): + return pl.DataFrame( + { + "timestamp": DAYS * 2, + "asset": ["A"] * 4 + ["B"] * 4, + "close": [1.0, 2.0, 3.0, 4.0, 10.0, 20.0, 30.0, 40.0], + "available_at": [datetime(2025, 7, day.day, 16, tzinfo=UTC) for day in DAYS] * 2, + } + ) + + +def decisions(): + return pl.DataFrame( + { + "timestamp": DAYS[2:], + "decision_time": [datetime(2025, 7, d.day, 17, tzinfo=UTC) for d in DAYS[2:]], + "effective_session": DAYS[2:], + } + ) + + +def universe(): + return pl.DataFrame( + { + "asset": ["A", "B"], + "effective_from": [DAYS[2]] * 2, + "effective_to": pl.Series([None, None], dtype=pl.Date), + "available_at": [datetime(2025, 7, 8, 12, tzinfo=UTC)] * 2, + "member": [True, False], + "sector": ["tech", "bank"], + } + ) + + +@pytest.mark.parametrize("lazy", [False, True]) +def test_dated_membership_keeps_preentry_warmup_and_original_feature_clock(lazy): + frame = history() + result = prepare(frame.lazy() if lazy else frame) + assert isinstance(result, pl.LazyFrame if lazy else pl.DataFrame) + result = result.collect() if lazy else result + assert result.select("timestamp", "asset").rows() == [ + (DAYS[2], "A"), + (DAYS[2], "B"), + (DAYS[3], "A"), + (DAYS[3], "B"), + ] + assert result["average"].to_list() == [2.0, 20.0, 3.0, 30.0] + assert result["eligible"].to_list() == [True, False, True, False] + assert result["member"].to_list() == [True, False, True, False] + assert result["sector"].to_list() == ["tech", "bank", "tech", "bank"] + assert ( + result["available_at"].to_list() + == [datetime(2025, 7, 9, 16, tzinfo=UTC)] * 2 + [datetime(2025, 7, 10, 16, tzinfo=UTC)] * 2 + ) + assert_frame_equal(frame, history()) + + +def test_next_session_decision_uses_current_membership_not_the_prior_price_session(): + requested = pl.DataFrame( + { + "timestamp": [DAYS[2]], + "decision_time": [datetime(2025, 7, 10, 13, tzinfo=UTC)], + "effective_session": [DAYS[3]], + } + ) + facts = universe().with_columns( + pl.when(pl.col("asset") == "A") + .then(pl.lit(DAYS[3])) + .otherwise(pl.col("effective_to")) + .alias("effective_to"), + pl.when(pl.col("asset") == "B") + .then(pl.lit(DAYS[3])) + .otherwise(pl.col("effective_from")) + .alias("effective_from"), + pl.lit(True).alias("member"), + ) + result = prepare(history(), decisions=requested, universe=facts) + assert result["eligible"].to_list() == [False, True] + assert result["average"].to_list() == [2.0, 20.0] + assert result["effective_session"].to_list() == [DAYS[3], DAYS[3]] + + +def test_explicit_instant_universe_clock_preserves_timezone_equivalence_and_exclusive_ns_end(): + base = int( + pl.Series([datetime(2025, 7, 10, 13, tzinfo=UTC)]) + .cast(pl.Datetime("ns", "UTC")) + .cast(pl.Int64)[0] + ) + requested = pl.DataFrame( + { + "timestamp": [DAYS[2]] * 2, + "decision_time": pl.Series([base + 2, base + 3], dtype=pl.Datetime("ns", "UTC")), + } + ) + facts = pl.DataFrame( + { + "asset": ["A"], + "effective_from": pl.Series([base + 1], dtype=pl.Datetime("ns", "America/New_York")), + "effective_to": pl.Series([base + 3], dtype=pl.Datetime("ns", "UTC")), + "available_at": [datetime(2025, 7, 9, tzinfo=UTC)], + "member": [True], + } + ) + result = prepare( + history(), decisions=requested, universe=facts, universe_time_col="decision_time" + ) + a = result.filter(pl.col("asset") == "A") + assert a["decision_time"].cast(pl.Int64).to_list() == [base + 2, base + 3] + assert a["eligible"].to_list() == [True, False] + + +def test_missing_universe_decision_key_is_not_inferred_from_the_price_or_utc_date(): + with pytest.raises(ValueError, match="effective_session"): + prepare(history(), decisions=decisions().drop("effective_session")) + assert prepare(history(), universe=universe().with_columns(pl.lit(None).alias("effective_to")))[ + "eligible" + ].to_list() == [True, False, True, False] + + +def test_future_publication_cannot_fill_or_compress_a_required_window(): + delayed = history().with_columns( + pl.when((pl.col("asset") == "A") & (pl.col("timestamp") == DAYS[1])) + .then(pl.lit(datetime(2025, 7, 10, 18, tzinfo=UTC))) + .otherwise(pl.col("available_at")) + .alias("available_at") + ) + result = prepare(delayed) + a = result.filter(pl.col("asset") == "A") + assert a["average"].null_count() == 2 + assert a["eligible"].to_list() == [False, False] + assert a["available_at"].null_count() == 2 + + +@pytest.mark.parametrize("lazy", [False, True]) +def test_explicit_known_missing_profile_uses_kernel_definition_without_changing_rows(lazy): + frame = history().with_columns( + pl.when((pl.col("asset") == "A") & (pl.col("timestamp") == DAYS[1])) + .then(None) + .otherwise(pl.col("close")) + .alias("close") + ) + strict = prepare(frame) + result = prepare(frame.lazy() if lazy else frame, missing_input_policy="known_missing") + result = result.collect() if lazy else result + assert result.select("timestamp", "asset").rows() == strict.select("timestamp", "asset").rows() + a = result.filter(pl.col("asset") == "A") + assert strict.filter(pl.col("asset") == "A")["eligible"].to_list() == [False, False] + assert a["average"].to_list() == [2.0, 3.5] + assert a["eligible"].to_list() == [True, True] + assert a["available_at"].to_list() == [ + datetime(2025, 7, 9, 16, tzinfo=UTC), + datetime(2025, 7, 10, 16, tzinfo=UTC), + ] + + +@pytest.mark.parametrize("value", [None, float("nan"), float("inf"), float("-inf")]) +def test_known_missing_profile_never_omits_unknown_future_or_invalid_infinite_inputs(value): + frame = history().with_columns( + pl.when((pl.col("asset") == "A") & (pl.col("timestamp") == DAYS[1])) + .then(pl.lit(value, dtype=pl.Float64)) + .otherwise(pl.col("close")) + .alias("close") + ) + if value is None or value != value: + frame = frame.with_columns( + pl.when((pl.col("asset") == "A") & (pl.col("timestamp") == DAYS[1])) + .then(None) + .otherwise(pl.col("available_at")) + .alias("available_at") + ) + result = prepare(frame, missing_input_policy="known_missing") + assert result.filter(pl.col("asset") == "A")["eligible"].to_list() == [False, False] + future = history().with_columns( + pl.when(pl.col("timestamp") == DAYS[1]) + .then(pl.lit(datetime(2025, 7, 10, 18, tzinfo=UTC))) + .otherwise(pl.col("available_at")) + .alias("available_at") + ) + assert not prepare(future, missing_input_policy="known_missing")["eligible"].any() + + +def test_known_missing_profile_preserves_masks_full_window_and_undefined_outputs(): + result = prepare( + history(), input_mask=pl.col("close") > 1, missing_input_policy="known_missing" + ) + assert result.filter(pl.col("asset") == "A")["average"].to_list() == [2.5, 3.0] + missing = history().with_columns(pl.lit(float("nan")).alias("close")) + result = prepare(missing, missing_input_policy="known_missing") + assert not result["eligible"].any() + short = history().filter(pl.col("timestamp") >= DAYS[1]) + result = prepare(short, missing_input_policy="known_missing") + a = result.filter(pl.col("asset") == "A") + assert a["average"].to_list() == [None, 3.0] + assert a["eligible"].to_list() == [False, True] + + +def test_custom_finite_expression_is_evaluated_per_entity_not_over_the_sorted_panel(): + result = prepare( + history(), + features={"tail_sum": pl.col("close").tail(3).sum()}, + lookbacks={"tail_sum": 3}, + ) + assert result["tail_sum"].to_list() == [6.0, 60.0, 9.0, 90.0] + + +def test_changing_only_final_screen_does_not_change_warmup_or_group_values(): + before = prepare(history()) + screened = prepare(history(), screen=pl.col("asset") == "A") + assert_frame_equal(screened, before.filter(pl.col("asset") == "A")) + masked = prepare(history(), input_mask=pl.col("close") > 1.0) + assert masked.filter((pl.col("timestamp") == DAYS[2]) & (pl.col("asset") == "A"))[ + "eligible" + ].to_list() == [False] + + +def test_unavailable_and_expired_universe_facts_do_not_default_to_membership(): + facts = universe().with_columns( + pl.when(pl.col("asset") == "A") + .then(pl.lit(datetime(2025, 7, 10, 18, tzinfo=UTC))) + .otherwise(pl.col("available_at")) + .alias("available_at") + ) + result = prepare(history(), universe=facts) + assert result.filter(pl.col("asset") == "A")["member"].null_count() == 2 + assert not result["eligible"].any() + expired = universe().with_columns(pl.lit(DAYS[3]).alias("effective_to")) + result = prepare(history(), universe=expired) + assert result.filter(pl.col("timestamp") == DAYS[3])["member"].null_count() == 2 + + +def test_appended_future_quote_and_unknown_universe_change_do_not_revise_prior_features(): + expected = prepare(history()) + future = ( + history() + .with_columns( + pl.lit(date(2025, 7, 11)).alias("timestamp"), + pl.lit(1e9).alias("close"), + pl.lit(datetime(2025, 7, 11, 16, tzinfo=UTC)).alias("available_at"), + ) + .unique(subset=["asset", "timestamp"]) + ) + future_fact = universe().with_columns( + pl.lit(date(2025, 7, 11)).alias("effective_from"), + pl.lit(datetime(2025, 7, 11, 12, tzinfo=UTC)).alias("available_at"), + pl.lit("future").alias("sector"), + ) + actual = prepare(pl.concat([history(), future]), universe=pl.concat([universe(), future_fact])) + assert_frame_equal(actual, expected) + + +def test_decision_specific_Data_snapshots_are_consumed_without_selecting_revisions(): + early = history().with_columns( + pl.lit(datetime(2025, 7, 9, 17, tzinfo=UTC)).alias("snapshot_at") + ) + late = history().with_columns( + (pl.col("close") * 2).alias("close"), + pl.lit(datetime(2025, 7, 10, 17, tzinfo=UTC)).alias("snapshot_at"), + ) + result = prepare(pl.concat([early, late]), history_decision_time_col="snapshot_at") + assert result["average"].to_list() == [2.0, 20.0, 6.0, 60.0] + with pytest.raises(ValueError, match="[Dd]uplicate"): + prepare(pl.concat([early, late])) + + +def test_ambiguous_overlapping_universe_records_require_upstream_resolution(): + facts = pl.concat( + [ + universe(), + universe().filter(pl.col("asset") == "A").with_columns(pl.lit("other").alias("sector")), + ] + ) + with pytest.raises(ValueError, match="[Aa]mbiguous|[Dd]uplicate"): + prepare(history(), universe=facts) + + +@pytest.mark.parametrize("calendar", ["omitted", None, "NYSE"]) +def test_feature_artifact_public_roundtrip_preserves_required_feed_and_eligibility_declarations( + calendar, +): + feed = FeedSpec( + time_contract_version=2, **({} if calendar == "omitted" else {"calendar": calendar}) + ) + payload = { + "artifact_id": "causal-features", + "schema": { + "timestamp_col": "when", + "entity_col": "identity", + "feature_columns": ["average"], + "available_at_col": "known", + "decision_time_col": "decision", + "eligible_col": "eligible", + "universe_time_col": "effective_session", + }, + "definition": { + "family": "financial", + "feed_spec": feed.to_dict(), + "window_unit": "observation" if calendar in (None, "omitted") else "expected_bar", + "price_basis": "split_adjusted", + "lookbacks": {"average": 3}, + "missing_input_policy": "known_missing", + }, + } + spec = FeatureSpec.from_mapping(json.loads(json.dumps(payload))) + output = json.loads(json.dumps(spec.to_dict())) + for name, value in payload["schema"].items(): + assert output["schema"][name] == value + for name, value in payload["definition"].items(): + assert output["definition"][name] == value + + +def test_adjacent_ns_observation_decision_and_publication_keys_do_not_round(): + frame = pl.DataFrame( + { + "timestamp": pl.Series([0, 1, 2], dtype=pl.Datetime("ns", "UTC")), + "asset": ["A"] * 3, + "close": [1.0, 2.0, 3.0], + "available_at": pl.Series([1, 2, 3], dtype=pl.Datetime("ns", "UTC")), + } + ) + requested = pl.DataFrame( + { + "timestamp": pl.Series([1, 2], dtype=pl.Datetime("ns", "UTC")), + "decision_time": pl.Series([2, 3], dtype=pl.Datetime("ns", "UTC")), + } + ) + facts = pl.DataFrame( + { + "asset": ["A"], + "effective_from": pl.Series([0], dtype=pl.Datetime("ns", "UTC")), + "effective_to": pl.Series([None], dtype=pl.Datetime("ns", "UTC")), + "available_at": pl.Series([0], dtype=pl.Datetime("ns", "UTC")), + "member": [True], + } + ) + result = prepare( + frame, + features={"average": finite_mean("close", period=2)}, + lookbacks={"average": 2}, + decisions=requested, + universe=facts, + universe_time_col="decision_time", + feed_spec=FeedSpec( + time_contract_version=2, + calendar=None, + entity_col="asset", + timestamp_semantics="event_time", + ), + ) + assert result["timestamp"].cast(pl.Int64).to_list() == [1, 2] + assert result["decision_time"].cast(pl.Int64).to_list() == [2, 3] + assert result["available_at"].cast(pl.Int64).to_list() == [2, 3] + assert result["average"].to_list() == [1.5, 2.5] + + +def test_instant_observation_cutoff_cannot_follow_the_actual_decision(): + frame = history().with_columns( + pl.col("timestamp").cast(pl.Datetime("ns", "UTC")).dt.offset_by("20h") + ) + requested = decisions().with_columns( + pl.col("timestamp").cast(pl.Datetime("ns", "UTC")).dt.offset_by("20h") + ) + facts = universe().with_columns( + pl.col("effective_from").cast(pl.Datetime("ns", "UTC")), + pl.col("effective_to").cast(pl.Datetime("ns", "UTC")), + ) + with pytest.raises(ValueError, match="cutoff|decision"): + prepare( + frame, + decisions=requested, + universe=facts, + universe_time_col="decision_time", + feed_spec=FeedSpec( + time_contract_version=2, calendar=None, timestamp_semantics="event_time" + ), + ) + + +def test_custom_roles_compound_ids_and_lazy_output_filter_preserve_the_real_clock(): + frame = ( + history() + .with_columns( + pl.struct(pl.lit("bundle").alias("namespace"), pl.col("asset").alias("sid")).alias( + "asset" + ) + ) + .rename({"timestamp": "when", "asset": "identity"}) + ) + facts = ( + universe() + .with_columns( + pl.struct(pl.lit("bundle").alias("namespace"), pl.col("asset").alias("sid")).alias( + "asset" + ) + ) + .rename({"asset": "identity"}) + ) + requested = decisions().rename({"timestamp": "when"}) + result = prepare( + frame.lazy(), + decisions=requested.lazy(), + universe=facts.lazy(), + feed_spec=FeedSpec( + time_contract_version=2, calendar=None, timestamp_col="when", entity_col="identity" + ), + ) + selected = result.filter( + (pl.col("identity").struct.field("sid") == "A") & (pl.col("when") == DAYS[3]) + ).collect() + assert selected["average"].to_list() == [3.0] + assert selected.schema["identity"] == frame.schema["identity"] + + +def test_missing_endpoint_or_market_coverage_does_not_carry_stale_features_forward(): + result = prepare( + history().filter(~((pl.col("asset") == "A") & (pl.col("timestamp") == DAYS[3]))) + ) + a = result.filter((pl.col("asset") == "A") & (pl.col("timestamp") == DAYS[3])) + assert a["member"][0] is True + assert a["average"][0] is None + assert a["eligible"][0] is False + empty = prepare(history().clear()) + assert empty.height == 4 + assert not empty["eligible"].any() + assert prepare(history(), decisions=decisions().clear()).is_empty() + + +@pytest.mark.parametrize( + "options", + [ + {"features": {}}, + {"lookbacks": {"other": 3}}, + {"lookbacks": {"average": True}}, + {"input_columns": "close"}, + {"input_columns": ["close", "close"]}, + {"input_columns": ["asset"]}, + {"price_basis": ""}, + {"missing_input_policy": "ignore"}, + {"input_mask": 1}, + {"screen": 1}, + {"timestamp_col": "other"}, + {"entity_col": "other"}, + {"entity_col": []}, + {"window_unit": "elapsed"}, + {"lookbacks": {"average": 2**64}}, + {"feed_spec": {"timestamp_col": "timestamp", "time_col": "other"}}, + {"features": {"average": "close"}}, + {"features": {"eligible": pl.col("close")}, "lookbacks": {"eligible": 1}}, + ], +) +def test_invalid_feature_contracts_fail_atomically(options): + before = history() + with pytest.raises((TypeError, ValueError)): + prepare(before, **options) + assert_frame_equal(before, history()) + + +@pytest.mark.parametrize( + "kind", + [ + "missing_history", + "missing_decision", + "missing_universe", + "bad_time", + "null_key", + "duplicate_key", + "duplicate_decision", + "bad_member", + "bad_interval", + "null_effective", + "classification_collision", + "reserved", + "naive_known", + "null_decision", + "range", + "missing_snapshot", + ], +) +def test_invalid_physical_inputs_do_not_produce_partial_panels(kind): + frame, requested, facts = history(), decisions(), universe() + options = {} + if kind == "missing_history": + frame = frame.drop("close") + elif kind == "missing_decision": + requested = requested.drop("decision_time") + elif kind == "missing_universe": + facts = facts.drop("member") + elif kind == "bad_time": + frame = frame.with_columns(pl.col("timestamp").cast(pl.String)) + elif kind == "null_key": + frame = frame.with_columns(pl.lit(None).alias("asset")) + elif kind == "duplicate_key": + frame = pl.concat([frame, frame.head(1)]) + elif kind == "duplicate_decision": + requested = pl.concat([requested, requested.head(1)]) + elif kind == "bad_member": + facts = facts.with_columns(pl.col("member").cast(pl.Int64)) + elif kind == "bad_interval": + facts = facts.with_columns(pl.col("effective_from").alias("effective_to")) + elif kind == "null_effective": + facts = facts.with_columns(pl.lit(None, pl.Date).alias("effective_from")) + elif kind == "classification_collision": + facts = facts.with_columns(pl.lit(1).alias("average")) + elif kind == "reserved": + frame = frame.with_columns(pl.lit(1).alias("__ml4t_feature_bad")) + elif kind == "naive_known": + frame = frame.with_columns(pl.col("available_at").dt.replace_time_zone(None)) + elif kind == "null_decision": + requested = requested.with_columns( + pl.lit(None, pl.Datetime("ns", "UTC")).alias("decision_time") + ) + elif kind == "range": + requested = requested.with_columns( + pl.lit(datetime(2500, 1, 1, tzinfo=UTC)).alias("decision_time") + ) + else: + frame = frame.with_columns(pl.lit(datetime(2025, 7, 11, tzinfo=UTC)).alias("snapshot_at")) + options["history_decision_time_col"] = "snapshot_at" + before = frame.clone() + with pytest.raises((TypeError, ValueError)): + prepare(frame, decisions=requested, universe=facts, **options) + assert_frame_equal(frame, before) + + +def test_named_schedule_checks_original_rows_and_explicit_none_remains_row_based(): + frame = history().filter(pl.col("timestamp") != DAYS[1]) + contract = FeedSpec( + time_contract_version=2, + calendar="NYSE", + entity_col="asset", + timestamp_semantics="session_label", + data_frequency="1d", + bar_type="time", + ) + with pytest.raises(ValueError, match="missing scheduled session"): + prepare(frame, feed_spec=contract) + row = prepare(frame, feed_spec=contract, calendar=None) + a = row.filter(pl.col("asset") == "A") + assert a["average"][0] is None + assert a["average"][1] == pytest.approx(8 / 3) + + +def test_unknown_availability_and_undefined_struct_outputs_cannot_become_eligible(): + unknown = history().with_columns(pl.lit(None).alias("available_at")) + result = prepare(unknown) + assert not result["eligible"].any() + assert result["available_at"].null_count() == result.height + + +def test_actual_undefined_regression_struct_fields_cannot_become_eligible(): + pytest.importorskip("scipy", reason="Regression p-values require stats extra") + expression = rolling_regression("close", pl.lit(1.0), period=2) + result = prepare(history(), features={"estimate": expression}, lookbacks={"estimate": 2}) + assert not result["eligible"].any() + + +def test_empty_history_does_not_waive_an_undeclared_default_calendar_grid(): + with pytest.raises(ValueError, match="declared bar grid"): + prepare(history().clear(), feed_spec=FeedSpec()) + + +@pytest.mark.parametrize("timezone", [None, "America/New_York"]) +def test_naive_observation_instants_require_and_use_declared_timezone(timezone): + frame = history().with_columns(pl.col("timestamp").cast(pl.Datetime("us")).dt.offset_by("12h")) + requested = ( + decisions() + .with_columns(pl.col("timestamp").cast(pl.Datetime("us")).dt.offset_by("12h")) + .with_columns(pl.col("timestamp").dt.offset_by("1h").alias("effective_at")) + ) + facts = universe().with_columns( + pl.col("effective_from").cast(pl.Datetime("us")), + pl.col("effective_to").cast(pl.Datetime("us")), + ) + options = { + "decisions": requested, + "universe": facts, + "universe_time_col": "effective_at", + "feed_spec": FeedSpec( + time_contract_version=2, + calendar=None, + timestamp_semantics="event_time", + timezone=timezone, + ), + } + if timezone is None: + with pytest.raises(ValueError, match="timezone"): + prepare(frame, **options) + else: + assert prepare(frame, **options)["eligible"].to_list() == [True, False, True, False] + + +@pytest.mark.parametrize( + "definition", + [ + {"lookbacks": {"average": 0}}, + {"lookbacks": {"average": True}}, + {"lookbacks": {"": 3}}, + {"lookbacks": {"average": "3"}}, + {"price_basis": " "}, + {"window_unit": "median_spacing"}, + {"missing_input_policy": "ignore"}, + ], +) +def test_invalid_artifact_declarations_do_not_survive_public_loading(definition): + with pytest.raises((TypeError, ValueError)): + FeatureSpec.from_mapping({"artifact_id": "invalid", "definition": definition}) + + +def test_minimal_artifacts_keep_old_wire_keys_and_source_lineage(): + spec = FeatureSpec.from_mapping( + { + "artifact_id": "old", + "definition": { + "source_artifacts": ["Data:snapshot:abc"], + "join_keys": ["timestamp", "venue", "asset"], + }, + } + ) + output = spec.to_dict() + assert set(output["schema"]) == {"timestamp_col", "entity_col", "feature_columns"} + assert set(output["definition"]) == {"family", "join_keys", "source_artifacts"} + assert output["definition"]["source_artifacts"] == ["Data:snapshot:abc"] + + +def test_persisted_compound_entity_columns_keep_canonical_schema_and_physical_ns(tmp_path): + payload = { + "artifact_id": "compound-features", + "schema": {"entity_col": ["venue", "asset"], "feature_columns": ["average"]}, + "definition": { + "feed_spec": FeedSpec( + time_contract_version=2, calendar=None, entity_col=["venue", "asset"] + ).to_dict(), + "price_basis": "raw", + "missing_input_policy": "complete", + }, + } + original = history().with_columns(pl.lit("XNYS").alias("venue")) + facts = universe().with_columns(pl.lit("XNYS").alias("venue")) + prepared = prepare( + original, + universe=facts, + feed_spec=FeedSpec.from_mapping(payload["definition"]["feed_spec"]), + ) + prepared = prepared.with_columns( + (pl.col("available_at").cast(pl.Int64) + 1) + .cast(pl.Datetime("ns", "UTC")) + .alias("available_at") + ) + path = tmp_path / "features.parquet" + prepared.write_parquet(path) + restored = pl.read_parquet(path) + assert_frame_equal(restored, prepared) + for encoding in (json.loads(json.dumps(payload)), yaml.safe_load(yaml.safe_dump(payload))): + output = FeatureSpec.from_mapping(encoding).to_dict() + assert output["schema"]["entity_col"] == ["venue", "asset"] + feed = FeedSpec.from_mapping(output["definition"]["feed_spec"]) + assert feed.resolve_calendar() is None + assert ( + restored.select(feed.timestamp_col, *feed.entity_col).rows() + == prepared.select("timestamp", "venue", "asset").rows() + ) + + +@pytest.mark.parametrize("columns", [[], [""], ["asset", "asset"], [1]]) +def test_invalid_compound_entity_schema_fails_before_persistence(columns): + with pytest.raises(ValueError, match="entity_col"): + FeatureSpec.from_mapping({"artifact_id": "invalid", "schema": {"entity_col": columns}}) + + +def test_explicit_pandas_presentation_keeps_effective_dates_exact_instants_and_stable_ids(): + pytest.importorskip("pyarrow", reason="Explicit pandas presentation requires PyArrow") + frame = prepare(history()).with_columns( + pl.when(pl.col("asset") == "A") + .then(2**63 + 1) + .otherwise(2**63 + 2) + .cast(pl.UInt64) + .alias("asset") + ) + presentation = ( + frame.select("effective_session", "asset", "average") + .to_pandas(date_as_object=True) + .set_index(["effective_session", "asset"]) + ) + assert list(presentation.index) == [ + (DAYS[2], 2**63 + 1), + (DAYS[2], 2**63 + 2), + (DAYS[3], 2**63 + 1), + (DAYS[3], 2**63 + 2), + ] + frame = frame.with_columns( + pl.Series("decision_time", [1, 2, 3, 4], dtype=pl.Datetime("ns", "UTC")) + ) + presentation = ( + frame.select("decision_time", "asset", "average") + .to_pandas() + .set_index(["decision_time", "asset"]) + ) + assert presentation.index.get_level_values("decision_time").asi8.tolist() == [1, 2, 3, 4] diff --git a/uv.lock b/uv.lock index 6525bd3a..6e203305 100644 --- a/uv.lock +++ b/uv.lock @@ -1656,6 +1656,7 @@ ml = [ { name = "shap" }, ] stats = [ + { name = "scipy" }, { name = "statsmodels" }, ] store = [ @@ -1756,6 +1757,7 @@ requires-dist = [ { name = "pyyaml", specifier = ">=6.0.0" }, { name = "ruff", marker = "extra == 'all'", specifier = ">=0.1.0" }, { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.1.0" }, + { name = "scipy", marker = "extra == 'stats'", specifier = ">=1.14.0" }, { name = "seaborn", marker = "extra == 'all'", specifier = ">=0.12.0" }, { name = "seaborn", marker = "extra == 'viz'", specifier = ">=0.12.0" }, { name = "shap", marker = "extra == 'all'", specifier = ">=0.44.0" },