feat: merge schemas as numpy promotes types - #478
Merged
Merged
Conversation
Add `NumpyArrowTypeWidening`, a second `ArrowTypeWideningStrategy` that follows `numpy.result_type`: a boolean joins the numbers, `Float16` joins the floats, a narrow integer beside a `Float32` stays a `Float32`, a number beside a string reads as text, and a date beside a timestamp is a timestamp at the finer unit. numpy resolves a set of types at once, and the answer differs from a chain of pairs (`int8` + `uint8` is `int16`, `int16` + `float16` is `float32`, yet `result_type(int8, uint8, float16)` is `float16`). The strategy gathers the types of each column across every schema and resolves the set once, so the listing order does not change the result. It takes one fold, as `keep_first` does. Four numpy rules stay behind because Arrow has no cast for them: an integer beside a duration, a duration beside a timestamp, and a number or text beside binary are conflicts; a time of day keeps the default chain. The CSV reader parses text as the merged type, so a boolean literal beside a number fails at read time under numpy where typed formats cast. The server builds the rule from `BEACON_TYPE_WIDENING_STRATEGY` (`default` or `numpy`) and `BEACON_TYPE_WIDENING_ON_CONFLICT`, and injects it through `RuntimeBuilder::with_type_widening`. `OpenOptions::with_type_widening` gives an embedder the same hook. `ArrowTypeWideningStrategy` now requires `Debug` so a config can hold the rule. The unit tests embed the full `numpy.promote_types` table from numpy 2.5.2 and check every Arrow pair against it, plus Arrow castability of every promotion and the n-ary results over every permutation.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #478 +/- ##
==========================================
+ Coverage 83.23% 83.41% +0.18%
==========================================
Files 373 374 +1
Lines 62526 63237 +711
==========================================
+ Hits 52044 52750 +706
- Misses 10482 10487 +5
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The default strategy refuses a boolean beside a number. A collection from numpy or xarray expects the rules of
numpy.result_type.This PR adds
NumpyArrowTypeWidening, a secondArrowTypeWideningStrategy. It applies the numpy rules:Float16joins the floats.Float32stays aFloat32.numpy resolves a set of types at once. The strategy gathers the types of each column across every schema and resolves the set once. The file order does not change the result.
Configuration
Set
BEACON_TYPE_WIDENING_STRATEGY=numpyon the server. Pass the strategy toRuntimeBuilder::with_type_wideningin an embedded build.BEACON_TYPE_WIDENING_ON_CONFLICTapplies under both strategies.Limits
Arrow has no cast for four numpy rules. These pairs stay conflicts: integer beside duration, duration beside timestamp, number or text beside binary. The CSV reader parses text as the merged type. A boolean literal beside a number fails at read time.
Tests
The tests embed the
numpy.promote_typestable from numpy 2.5.2. They check every Arrow pair against it and the Arrow cast of every promotion.