Skip to content

feat(features): port finite Pipeline factors and causal selection workflows - #115

Merged
stefan-jansen merged 20 commits into
mainfrom
feat/pipeline-factor-methods
Oct 21, 2026
Merged

stefan-jansen merged 20 commits into
mainfrom
feat/pipeline-factor-methods

Conversation

@stefan-jansen

@stefan-jansen stefan-jansen commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Port finite Pipeline feature calculations and dated selection workflows to native Polars without a legacy graph runtime. Add separately named finite methods instead of changing Wilder RSI, recursive EMA, portfolio drawdown or other existing definitions.

  • Add sixteen registry-backed finite methods, paired target-price preparation and supplied known event weekday counts. Keep exact typed differences, original windows, target keys, missingness and material method choices.
  • Add eligible decision-time cross-sections, Boolean windows and causal feature-panel preparation using supplied Data/caller history and universe facts. Observation cutoff, decision instant and decision-effective universe key are distinct. Missing endpoints never carry stale values forward.
  • Add explicit factor-name/factory recipes, native custom factor/filter/classifier examples and an operation reference. No mandatory Zipline, Data, SciPy or store dependency. Regression p-values require the existing optional stats boundary.
  • Persist canonical FeedSpec, physical keys, availability, eligibility, lookbacks, price basis and explicit missing-input policy in owned feature artifacts. Preserve omitted versus null calendar selectors and compound entity columns.
  • Keep strict complete-input eligibility by default; opt into known-missing kernel rules explicitly. Unknown/future publication and infinite inputs remain ineligible. Reuse existing schedule validation before calculations.

Closes #108.
Closes #109.

Correctness and migration evidence

Actual installed Zipline Reloaded 3.2.0 comparisons include 101 finite/return-factor/default/factory profiles, 19 selection profiles, four original/custom momentum PipelineEngine journeys through the production shifted pricing loader, 55 math/summary/return/event/operator profiles and 28 classifier/filter profiles. Each original/custom journey compares 135 exact output keys, RSI and long/short selections, including known missing quotes and listing/delisting lifetimes. Actual installed ML4T Data history output preserves a causal mean under a future correction and stable ID through split/cash/rename history.

The regression tests also cover large integer/Decimal prices, variance scale, exact nanosecond knowledge boundaries, real missing scheduled bars versus explicit row mode, keyed target alignment, input masks versus final screens, undefined multi-output values, typed Parquet and JSON/YAML handoffs, and explicit pandas presentation. Deliberate producing-code controls pass all seventeen factor/panel acceptance items with no waivers. Independent mathematical expectations supplement legacy comparisons.

One legacy correctness difference is deliberate: MaximumFilter's float-bit integer ordering selects the wrong result for all-negative groups. Native top(1) retains correct numeric ordering and stable ties. Weekday event counts remain distinct from exchange schedules. RateOfChangePercentage is signed percentage points; PercentChange uses a fractional absolute denominator. The migration reference documents these differences.

Verification status

At final source bc2042430ab134bde7fe725878028ad7de3bf224, the full native suite passes 5,555 tests, with 57 configured deselections. Ruff lint/format, ty, executable documentation, strict MkDocs and commit hooks pass. The preceding 4e20f54 has an identical runtime source subtree and passes no-JIT coverage with 95.91% release-critical line coverage and 90.27% branch coverage. The final source additionally repairs actual Windows cp1252 example output and optional-SciPy executable documentation. Isolated installed Python 3.15.0rc1 with SciPy absent passes 458 focused core/compatibility/documentation cases, with 14 explicit optional-dependency skips, not a module-wide skip. Both current security audits pass.

The downloaded hosted candidate 9264c27e6d09c2db2f98dd119bd9571c60e4efd5 has exactly the reviewed source tree. Its wheel SHA256 is a96f8e4a4a015f0960094091d931817562e49344d24390dcdf2e41ad3eacdf4a. All four real PipelineEngine journeys and 55 expression/operator profiles also pass from that installed hosted artifact. Hosted supported-platform checks remain authoritative; merge waits for every applicable final-revision check, including the final Build Package job. Earlier failed and passing receipts retain their exact tested revisions.

This PR does not publish a release, run arbitrary legacy Terms, select source revisions or adjustments, implement Alphalens cleaning/analytics, or claim order/account/performance equivalence. Optional market/loading preparation and final cross-library migration acceptance remain tracked separately by #111 and #112.

AI assistance

Codex assisted with implementation, tests, migration documentation and this PR text. Producing source, regression controls, actual installed workflows and current-head checks are reviewed and verified in this owner session; automated text or import-only checks are not treated as review completion.

Select native arithmetic at schema planning time, subtract widened integer ticks before floating division, and retain float dtype/formula and logarithmic domains. Require Polars 2.0 for the native schema dispatch API. Add eager/lazy/grouped regression cases and an executable migration recipe.

Refs #107
The required dependency audit on PR113 reproduced GHSA-6q48-6c7w-9gw2 in certifi 2026.7.22. Upgrade only certifi to patched 2026.10.19; both runtime and complete contributor lock exports now pass pip-audit. Existing dependency PR106 retains the affected version and does not supply this repair.
The fresh contributor audit reproduces eight Pillow12.3 advisories published after the earlier clean local audit. Lock the patched13.0 release without ignoring vulnerabilities or changing runtime requirements. Core requirements are unchanged; focused numerical and executable documentation checks pass.
Preserve native TA-Lib defaults while providing separately registered finite weighting, Cutler RSI, population volatility and technical field definitions. Keep missing-value denominators, warmup and grouped lazy execution explicit. Refs #108.
Keep missing pairs on their original clock, expose actual regression fields through the optional stats extra, and separate civil weekdays from publication instants. Reproduce and repair integer-tick and overflow errors in finite variance and range methods without changing existing native defaults. Refs #108.
Two positive weights imply unbiased variance (x1-x0)^2/2 regardless of their ratio. Reproduce cancellation and false NaN results before accumulating the equivalent cross-weight denominator directly. Correct the earlier constant-price test assumption to zero variance. Refs #108.
Translate finite window counts, exponential constructors, named fields, paired statistics and known event roles without a legacy runtime or graph interpreter. Preserve output masks and price-to-return dependency warmup; reject unrelated options and unordered input roles. Refs #108, #109.
Copilot AI balanced review requested due to automatic review settings October 21, 2026 01:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@stefan-jansen
stefan-jansen merged commit dd26bca into main Oct 21, 2026
49 checks passed
@stefan-jansen
stefan-jansen deleted the feat/pipeline-factor-methods branch October 21, 2026 03:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prepare dated feature panels, grouped transforms and selection screens Complete reusable Pipeline factor calculations and explicit methods

2 participants