Skip to content

feat: merge schemas as numpy promotes types - #478

Merged
robinskil merged 1 commit into
mainfrom
features/numpy-type-widening
Sep 7, 2026
Merged

robinskil merged 1 commit into
mainfrom
features/numpy-type-widening

Conversation

@robinskil

Copy link
Copy Markdown
Collaborator

Summary

The default strategy refuses a boolean beside a number. A collection from numpy or xarray expects the rules of numpy.result_type.

This PR adds NumpyArrowTypeWidening, a second ArrowTypeWideningStrategy. It applies the numpy rules:

  • A boolean joins the numbers.
  • Float16 joins the floats.
  • A narrow integer beside a Float32 stays a Float32.
  • A number beside a string reads as text.
  • A date beside a timestamp is a timestamp at the finer unit.

numpy resolves a set of types at once. The strategy gathers the types of each column across every schema and resolves the set once. The file order does not change the result.

Configuration

Set BEACON_TYPE_WIDENING_STRATEGY=numpy on the server. Pass the strategy to RuntimeBuilder::with_type_widening in an embedded build. BEACON_TYPE_WIDENING_ON_CONFLICT applies under both strategies.

Limits

Arrow has no cast for four numpy rules. These pairs stay conflicts: integer beside duration, duration beside timestamp, number or text beside binary. The CSV reader parses text as the merged type. A boolean literal beside a number fails at read time.

Tests

The tests embed the numpy.promote_types table from numpy 2.5.2. They check every Arrow pair against it and the Arrow cast of every promotion.

Add `NumpyArrowTypeWidening`, a second `ArrowTypeWideningStrategy` that
follows `numpy.result_type`: a boolean joins the numbers, `Float16` joins
the floats, a narrow integer beside a `Float32` stays a `Float32`, a number
beside a string reads as text, and a date beside a timestamp is a timestamp
at the finer unit.

numpy resolves a set of types at once, and the answer differs from a chain
of pairs (`int8` + `uint8` is `int16`, `int16` + `float16` is `float32`, yet
`result_type(int8, uint8, float16)` is `float16`). The strategy gathers the
types of each column across every schema and resolves the set once, so the
listing order does not change the result. It takes one fold, as
`keep_first` does.

Four numpy rules stay behind because Arrow has no cast for them: an integer
beside a duration, a duration beside a timestamp, and a number or text
beside binary are conflicts; a time of day keeps the default chain. The CSV
reader parses text as the merged type, so a boolean literal beside a number
fails at read time under numpy where typed formats cast.

The server builds the rule from `BEACON_TYPE_WIDENING_STRATEGY` (`default`
or `numpy`) and `BEACON_TYPE_WIDENING_ON_CONFLICT`, and injects it through
`RuntimeBuilder::with_type_widening`. `OpenOptions::with_type_widening`
gives an embedder the same hook. `ArrowTypeWideningStrategy` now requires
`Debug` so a config can hold the rule.

The unit tests embed the full `numpy.promote_types` table from numpy 2.5.2
and check every Arrow pair against it, plus Arrow castability of every
promotion and the n-ary results over every permutation.
@robinskil robinskil self-assigned this Sep 7, 2026
@codecov

codecov Bot commented Sep 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.47434% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.41%. Comparing base (294e328) to head (bd596a6).

Files with missing lines Patch % Lines
...b/beacon-datafusion-ext/src/type_widening/numpy.rs 99.10% 6 Missing ⚠️
beacon-db/beacon-core/src/embedded.rs 28.57% 5 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #478      +/-   ##
==========================================
+ Coverage   83.23%   83.41%   +0.18%     
==========================================
  Files         373      374       +1     
  Lines       62526    63237     +711     
==========================================
+ Hits        52044    52750     +706     
- Misses      10482    10487       +5     
Files with missing lines Coverage Δ
beacon-db/beacon-core/src/runtime_builder.rs 92.23% <ø> (ø)
...acon-db/beacon-datafusion-ext/src/type_widening.rs 98.69% <100.00%> (+0.22%) ⬆️
beacon-server/beacon-server-config/src/lib.rs 95.77% <100.00%> (+0.99%) ⬆️
beacon-server/beacon-server/src/server/mod.rs 84.48% <100.00%> (ø)
beacon-db/beacon-core/src/embedded.rs 68.46% <28.57%> (-0.69%) ⬇️
...b/beacon-datafusion-ext/src/type_widening/numpy.rs 99.10% <99.10%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@robinskil
robinskil merged commit b27e545 into main Sep 7, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant