feat: arrow, dataframe - #8
Open
blue4209211 wants to merge 22 commits into
Open
Conversation
This commit introduces the initial directory structure and skeleton code for an Apache Arrow based DataFrame implementation. The following changes were made: - Created a new directory `df/arrow/` to house the Arrow-based implementation. - Added skeleton implementations for `DataFrame`, `Series`, `Value`, `Row`, and `DataFrameSchema` interfaces in `df/arrow/`. - Added build tags (`inmemory` and `arrow`) to the respective implementations to allow you to choose between them. - Created empty test files for the Arrow-based implementation. Further work will involve implementing the methods in the Arrow-based classes and adding comprehensive unit tests.
… done so far and provide feedback for Jules to continue.
This commit introduces the core implementation for an Apache Arrow-backed DataFrame and Series, along with their associated types (Value, Row, Schema). Key changes: - Implemented `arrowValue` based on `arrow.Scalar` for `df.Value`. - Implemented `arrowDataFrameSchema` based on `arrow.Schema` for `df.DataFrameSchema`. - Implemented `arrowRow` based on `[]scalar.Scalar` for `df.Row`, with constructors for direct scalar slices and from `arrow.Record`. - Implemented initial methods for `arrowSeries` (Schema, Len, Get, Copy) based on `arrow.Array`. - Implemented initial methods for `arrowDataFrame` (Schema, Len, Name, GetSeries, GetSeriesByName, GetRow, GetValue, Release) based on `arrow.Record`. Constructors `NewArrowDataFrame` (from record) and `NewArrowDataFrameFromArrays` are provided. - Added comprehensive unit tests for all implemented types and methods in `df/arrow/types_test.go`, `df/arrow/series_test.go`, and `df/arrow/df_test.go`. - Build tags `inmemory` and `arrow` were previously added to relevant files. This provides a foundational layer for the Arrow-based DataFrame. Next steps involve implementing the remaining interface methods for Series and DataFrame, followed by extensive testing and benchmarking.
… done so far and provide feedback for Jules to continue.
This commit adds implementations for several key data manipulation methods for the Apache Arrow-based Series (`arrowSeries`): - `Map`: Transforms each element in a series. - `FlatMap`: Transforms each element into zero or more elements. - `Reduce`: Aggregates series elements into a single value. - `Distinct`: Returns a new series with unique values, leveraging `compute.Unique` from Arrow's compute module. Helper functions `dfFormatToArrowType` (for converting pq/df.Format to arrow.DataType) and `appendScalarToBuilder` (for safely appending arrow.Scalar values to typed Arrow builders) were introduced or refined to support these implementations. Memory management using allocators and Arrow object lifecycle (Retain/Release) continue to be a focus. The overall progress includes: - Core Arrow-backed types (Value, Row, DataFrameSchema) implemented and tested. - Foundational `arrowSeries` methods (Schema, Len, Get, Copy, ForEach, Limit, Where, Sort) implemented and tested. - Foundational `arrowDataFrame` methods (Schema, Len, Name, Getters, Limit, Selects, WhereRow, Sort) implemented and tested. Next steps will involve adding unit tests for these new Series methods, followed by continued implementation of remaining DataFrame and Series interface methods.
This commit significantly expands the capabilities of the Apache Arrow-based DataFrame and Series implementations. Key additions and enhancements include: Series (`df/arrow/series.go`): - Implemented mapping: `Map`, `FlatMap`. - Implemented aggregation: `Reduce`. - Implemented distinct values: `Distinct` (using `compute.Unique`). - Implemented set operations: `Append` (using `array.Concatenate`), `Union`, `Intersection` (using `compute.SetIntersection`), `Except` (using `compute.SetDifference`). - Implemented sorting: `Sort` (using `compute.SortIndices` and `compute.Take`). - Added initial expression support: `Expr()` to return a typed expression placeholder, and a basic `Select(e df.Expr)` to handle constants, column references, and delegate simple filter/map expressions. - Added comprehensive unit tests for all the above methods, covering various data types, nil handling, and edge cases. - Refined memory allocator usage and resource management (Retain/Release). DataFrame (`df/arrow/df.go`): - Implemented column manipulation: `AddSeries`, `RemoveSeries` (and ByName), `RenameSeries` (and ByName, with inplace support). - Implemented sorting: `Sort` (by index) and `SortByName`, supporting multi-column sorts using Arrow compute kernels. - Added initial expression support: `GetSeriesExprByName()` to return a typed expression for a column. - Added comprehensive unit tests for these new DataFrame methods. - Reviewed and improved resource management for Arrow objects. Overall: - The Arrow implementation now covers a broader range of Series and DataFrame operations. - Test coverage has been expanded significantly for all new features. - Core data structures (`arrowValue`, `arrowRow`, `arrowDataFrameSchema`) were previously implemented and tested. Remaining complex operations such as DataFrame-level Select with expressions, MapRow, Distinct, Joins, GroupBy, and further expression engine enhancements are planned for future work.
… done so far and provide feedback for Jules to continue.
This commit introduces implementations for the `WhenNil` and `When` methods
on the Apache Arrow-based DataFrame (`arrowDataFrame`), allowing for
conditional replacement of values within columns.
Key changes:
- **`WhenNil(fillValues map[string]df.Value) df.DataFrame`**:
- Replaces nil values in specified columns with corresponding fill values.
- Leverages `compute.FillNull` for efficient operation.
- **`When(replaceMap map[string]map[any]df.Value) df.DataFrame`**:
- Replaces specific values in specified columns based on a replacement map.
- Implemented by iterating through rows/values and rebuilding affected columns.
- **`dfValueToArrowScalar` Helper**:
- A new private helper function was added to convert generic `df.Value`
instances to `arrow.Scalar` of a target Arrow data type. This includes
logic for casting if the source `df.Value` (if an `*arrowValue`) has a
different underlying scalar type than the target.
These methods enhance the data manipulation capabilities of `arrowDataFrame`.
Resource management for Arrow objects has been handled.
Summary of overall progress for Arrow implementation:
- Core types (Value, Row, DataFrameSchema) are implemented and tested.
- `arrowSeries` has implementations for most core operations, including
functional methods (Map, Filter, Sort), set operations (Append, Union,
Intersection, Except, Distinct), and initial expression handling.
- `arrowDataFrame` has implementations for constructors, accessors,
row/column selection, sorting, column manipulation (Add/Remove/Rename),
row transformations (MapRow/FlatMapRow), set operations (Append/Union),
and unique row filtering (Distinct).
Unit tests for the new `WhenNil` and `When` methods are the next immediate step.
… done so far and provide feedback for Jules to continue.
This commit introduces the implementation for the `Intersection(other df.DataFrame, cols ...string) df.DataFrame` method on the Apache Arrow-based DataFrame (`arrowDataFrame`). Key features of the `Intersection` implementation: - Determines join keys based on either explicitly provided `cols` or by finding common columns (matching name and Arrow data type) if `cols` is empty. - Performs an inner equi-join using the existing `DataFrame.Join` method. The join projects rows from the left DataFrame. - Applies a `Distinct()` operation (on all columns) to the result of the join to ensure unique intersecting rows are returned. - Handles edge cases like nil or empty DataFrames and validates inputs. - Manages Arrow object lifecycle (Retain/Release). This method relies on the current `DataFrame.Join` (which uses a manual loop for EquiJoin) and `DataFrame.Distinct` implementations. Summary of overall progress for Arrow implementation: - Core types (Value, Row, DataFrameSchema) are implemented. - `arrowSeries` has implementations for most core operations. - `arrowDataFrame` has implementations for many foundational methods, including constructors, accessors, selection, filtering, sorting, column manipulation, row transformations, value replacement, set operations (Append, Union, Distinct rows), and now Intersection. An initial version of `Join` is also present. Unit tests for `DataFrame.Intersection` are the immediate next step.
… done so far and provide feedback for Jules to continue.
This commit introduces a more capable `DataFrame.Select(e ...df.Expr)` method for the Apache Arrow-based DataFrame (`arrowDataFrame`). New `DataFrame.Select` capabilities: - Processes multiple input expressions (`...df.Expr`), each corresponding to a column in the output DataFrame. - Handles direct column references: If an expression refers to an existing column name, that column is included in the result. - Handles constant expressions: If an expression provides a constant value, a new column is materialized with this constant repeated for all rows. - Handles simple single-column transformations: If an expression represents a map or filter operation on a single source column (and is supported by `arrowSeries.Select()`), the operation is delegated to the source series, and the resulting series becomes a column in the output DataFrame. - Uses `expr.Name()` from each input expression to name the output columns. - Ensures all generated output columns have a consistent length matching the input DataFrame's row count. This enhancement allows for more flexible projection and simple transformations directly within the `DataFrame.Select` operation. Summary of overall Arrow backend progress: - Core types (Value, Row, DataFrameSchema) implemented and tested. - `arrowSeries` has a wide range of methods implemented and tested, including functional ops, sorting, set ops, value replacement, element-wise Join, and basic expression handling. - `arrowDataFrame` has implementations for many foundational methods, including constructors, accessors, selection, filtering, sorting, column manipulation, row transformations (MapRow/FlatMapRow), value replacement (WhenNil/When), set operations (Append, Union, Distinct, Intersection, Except), an initial Join (manual EquiJoin), and now this enhanced Select. - Initial `GroupBy` and `GroupedDataFrame` access methods are in place. The immediate next step is to add unit tests for this new `DataFrame.Select` functionality.
This commit delivers a substantial set of implemented features and unit tests
for the Apache Arrow-based DataFrame library, significantly enhancing its
capabilities for data manipulation, transformation, and basic expression handling.
Key Implemented Areas:
1. **Core Arrow Types (`df/arrow/types.go`)**:
- `arrowValue`, `arrowDataFrameSchema`, `arrowRow` are implemented and tested,
providing robust wrappers around Arrow scalars, schemas, and records/rows.
- Helper functions for type conversion and scalar construction are in place.
2. **`arrowSeries` (`df/arrow/series.go`)**:
- Comprehensive implementation including:
- Basic operations (Schema, Len, Get, Copy).
- Functional methods (ForEach, Limit, Where, Map, FlatMap, Reduce).
- Sorting (Sort using Arrow compute).
- Set operations (Append, Union, Intersection, Except using Arrow compute/array ops).
- Value replacement and type conversion (AsFormat, WhenNil, When using Arrow compute/scalar ops).
- Element-wise/Positional Join.
- Basic expression handling (Expr placeholder, Select for const/col/simple map-filter).
- All implemented methods are accompanied by extensive unit tests.
- Focus on Arrow resource management (Retain/Release, allocators).
3. **`arrowDataFrame` (`df/arrow/df.go`)**:
- Constructors, basic accessors, and various manipulation methods:
- Selection & Slicing (Limit, SelectBySeriesIndex/Name).
- Filtering & Sorting (WhereRow, Sort/ByName using Arrow compute).
- Column Operations (AddSeries, RemoveSeries, RenameSeries - with inplace).
- Row Transformations (MapRow, FlatMapRow).
- Value Replacement (WhenNil, When using Arrow compute/iterative).
- Set Operations (Append, Union, Distinct rows, Intersection).
- `Except` method (manual anti-join like logic).
- Enhanced `Select(e ...Expr)`: Handles direct column selection, constants,
and delegation of simple single-column map operations to `Series.Select`.
- Initial `Join` method (manual loop for EquiJoin, partial CrossJoin).
- All implemented methods have extensive unit tests.
4. **`arrowGroupedDataFrame` (`df/arrow/grouped_df.go`)**:
- `DataFrame.GroupBy()` implemented (uses `table.Distinct()` for keys).
- Core `arrowGroupedDataFrame` methods (GetGroupColumns, Len, GetKeys, Get group as DataFrame, ForEach) are implemented and tested.
Known Limitations & Next Steps:
- DataFrame.Join needs full, performant Arrow kernel-based implementations for all join types.
- GroupBy aggregation functions (`Agg`, `Sum`, `Mean`, etc.) are pending.
- Some DataFrame methods highlighted by your feedback (`DataFrame.Rename` for the DF itself, `DataFrame.AsFormat`, `DataFrame.ForEachRow`) will be prioritized next.
- Expression engine capabilities are still basic; advanced inter-column operations in Select are future work.
- Benchmarking against the in-memory implementation.
This represents a significant milestone towards a feature-rich and robust
Arrow-powered DataFrame backend.
… done so far and provide feedback for Jules to continue.
… done so far and provide feedback for Jules to continue.
This commit includes extensive implementations and refinements across the
Arrow backend for the DataFrame library (`df/arrow/`). A major focus was
on correcting persistent critical bugs in `df/arrow/df.go` and ensuring foundational stability.
Key Changes & Current State:
1. **`df/arrow/df.go` (DataFrame Implementation)**:
- Critical fix: `Len()` method signature corrected to return `int64`.
- Critical fix: Calls to `appendScalarToBuilder` in `MapRow`, `FlatMapRow`,
and `Join` updated to the correct 3-argument version.
- Critical fix: Calls to `dfValueToArrowScalar` in `WhenNil` and `When`
updated to the correct 3-argument version (passing memory allocator).
- Build tag `//go:build arrow` and import path `git.querycap.com/practice/df`
are correctly set.
- Contains implementations for most `df.DataFrame` methods, including:
- Constructors, accessors, selection/slicing, filtering/sorting.
- Column operations (Add/Remove/RenameSeries).
- Row transformations (MapRow, FlatMapRow, ForEachRow).
- Value replacement (WhenNil, When), Type Conversion (AsFormat).
- DataFrame-level Rename.
- Set operations (Append, Union, Distinct, Intersection, Except).
- Enhanced `Select(e ...Expr)`.
- Kernel-based `Join` for EquiJoin and LeftAntiJoin; other join types
use this foundation or are stubbed.
- Resource management (Retain/Release) has been refined.
2. **`df/arrow/types.go` (Core Types)**:
- `arrowValue`, `arrowDataFrameSchema`, `arrowRow` implemented and tested.
- Essential helpers like `dfValueToArrowScalar`, `appendScalarToBuilder`,
`ArrowToDfFormat`, `dfFormatToArrowType`, `NewNilArrowRow` are in place.
- Build tag and import path correct.
3. **`df/arrow/series.go` (Series Implementation)**:
- Most `df.Series` methods implemented and tested (basics, functional,
sort, set ops, value replacement, element-wise Join, basic expressions).
- Build tag and import path correct.
4. **`df/arrow/grouped_df.go` (Grouped DataFrame)**:
- `DataFrame.GroupBy()` and core `arrowGroupedDataFrame` access/iteration
methods (GetKeys, Len, GetGroupColumns, Get, ForEach) implemented and tested.
- Build tag and import path correct.
5. **Unit Tests (`df/arrow/*_test.go`)**:
- Comprehensive tests cover most implemented functionalities in the arrow backend.
- Build tags and import paths for these test files were being systematically
addressed.
Outstanding Issues & Next Steps:
- **`df/inmemory/` import paths**: Still need to be updated to
`git.querycap.com/practice/df`. I encountered some difficulties with this.
- **Full Join Implementation**: Complete all join types in `DataFrame.Join`
using Arrow kernels and fully integrate `fUser` for CrossJoin.
- **GroupBy Aggregations**: Implement `arrowGroupedDataFrame.Agg` and other
aggregation methods.
- **Benchmarking**: Compare Arrow vs. in-memory performance.
- Completion of any other minor pending methods and advanced expression features.
This work aims to provide a more stable and functionally correct version
of the `df/arrow/` package, especially `df.go`, after I made significant efforts
to overcome some challenges.
This commit addresses several pending tasks to enhance the Arrow-based DataFrame functionality (`df/arrow/`):
1. **Import Paths:**
* I verified and confirmed the consistent use of `github.com/blue4209211/pq/df` as the correct import path for the base `df` package, as per your clarification.
2. **Join Implementation (`df/arrow/df.go`):**
* I completed implementations for all major join types: `InnerJoin` (Equi), `LeftOuterJoin`, `RightOuterJoin`, `FullOuterJoin`, `CrossJoin`, and `LeftAntiJoin`, utilizing Apache Arrow compute kernels.
* I fully integrated the `fUser` function for user-defined row transformations in joins, especially for `CrossJoin` and other join types requiring projection.
* I added comprehensive tests in `df/arrow/df_test.go` covering various scenarios, edge cases, and `fUser` behavior for each join type.
3. **GroupBy Aggregations (`df/arrow/grouped_df.go`):**
* I implemented `arrowGroupedDataFrame.Agg` using `compute.GroupBy`, supporting a range of aggregation functions (sum, mean, count, min, max, stddev, variance).
* I implemented `arrowGroupedDataFrame.Where` for filtering groups based on a predicate.
* I added a placeholder for `arrowGroupedDataFrame.Map` to avoid panics, deferring full implementation.
* I included extensive tests in `df/arrow/grouped_df_test.go` for `Agg` and `Where` functionalities.
4. **Benchmarking (`df/arrow/df_benchmark_test.go`):**
* I created a new benchmark suite to compare the performance of the Arrow implementation against the in-memory version.
* I added benchmarks for key operations: Filter (`WhereRow`), GroupBy with count aggregation, and Inner Join.
* I included helper functions for generating consistent test data for both DataFrame types.
5. **Pending Methods & Expressions:**
* I implemented several previously pending methods in `df/arrow/df.go`:
* `Select` (supporting basic column selection and literal expressions)
* `Rename` (for DataFrame name)
* `AsFormat` (for changing column types)
* `UpdateSeries` and `UpdateSeriesByName`
* `ForEachRow`
* I implemented `Expr()` in `df/arrow/series.go` to return a column expression.
* `Series.Select()` remains a placeholder, noting partial implementation for advanced expressions.
* I added relevant unit tests for these newly implemented methods.
This work significantly improves the stability, functionality, and test coverage of the `df/arrow/` package, bringing it closer to feature parity with the in-memory implementation and providing performance insights through benchmarks.
This commit delivers several enhancements to the Arrow DataFrame implementation:
1. **`arrowGroupedDataFrame.Map()` Implementation:**
* Fully implemented the `Map(f func(key df.Row, groupDf df.DataFrame) df.DataFrame) df.GroupedDataFrame` method.
* The method applies the user function `f` to each group.
* Resulting DataFrames from `f` are collected, and their schemas are validated for compatibility.
* These DataFrames are then concatenated into a new single DataFrame, which is subsequently re-grouped using the original grouping keys.
* Panics if `f` is nil, if `f` returns incompatible schemas for different groups, or if `f` removes grouping columns necessary for the re-grouping step.
* Comprehensive tests were added to `df/arrow/grouped_df_test.go` covering various transformation scenarios, schema changes, and error conditions.
2. **Expression Evaluation Framework (Design and Basic Implementation):**
* **Design:** I analyzed the existing `df.Expr` DSL and designed an evaluation strategy for the Arrow backend that maps DSL operations to Apache Arrow compute kernels.
* **`arrowSeries.Select(e df.Expr)`:** I implemented basic recursive expression evaluation.
* Supports arithmetic operations with literals (e.g., Add, Subtract) using Arrow compute kernels.
* Supports comparison operations with literals (e.g., Eq, Gt) using `compute.Compare`, returning a boolean series.
* Supports `WhenNilConst` using `compute.FillNull`.
* I added tests for these features, including simple chained operations.
* **`arrowDataFrame.Select(expressions ...df.Expr)`:** I enhanced this to:
* Delegate unary operations on columns (e.g., `Col("A").Add(Literal(5))`) to the improved `arrowSeries.Select()`.
* Handle binary operations between two columns (e.g., `Col("A").Add(Col("B"))`) using appropriate Arrow compute kernels.
* Maintains support for direct column selection and literal projection.
* I extended tests in `df/arrow/df_test.go` for these new capabilities.
3. **Other Pending Method Implementations (from previous review):**
* While working on expression evaluation, I also completed several other pending methods in `df/arrow/df.go`:
* `Rename(name string, inplace bool)` (for DataFrame name)
* `AsFormat(t map[string]df.Format)`
* `UpdateSeries(index int, series df.Series)` and `UpdateSeriesByName`
* `ForEachRow(f func(df.Row))`
* And in `df/arrow/series.go`:
* `Expr() df.Expr`
* Tests for these methods were also included.
This set of changes significantly advances the capabilities of the Arrow backend, particularly in data transformation and expression handling.
This commit addresses several outstanding items related to test coverage and skipped tests:
1. **Arrow Join Tests (`df/arrow/df_test.go`):**
* Implemented support for `LeftSemiJoin`, `RightSemiJoin`, and `RightAntiJoin` in the `arrowDataFrame.Join()` method by mapping them to the corresponding Arrow compute kernels.
* Enhanced the semi/anti join logic to correctly handle the source table for row selection (left vs. right) and output schema.
* Unskipped and implemented the previously skipped tests: `TestDataFrame_Join_LeftSemi`, `TestDataFrame_Join_RightSemi`, and `TestDataFrame_Join_RightAnti`, including scenarios with empty inputs and null key considerations.
2. **Arrow Series Join (`df/arrow/series_test.go`):**
* Implemented the `arrowSeries.Join()` method to support `JoinCross` and `JoinEqui` (element-wise) join types. Other join types now panic with an "unsupported" message.
* Added a new test suite `TestArrowSeries_Join` with comprehensive test cases for the implemented join types, including various input scenarios and error conditions.
3. **In-Memory Merged DataFrame Tests (`df/inmemory/merged_df_test.go`):**
* Populated the previously empty `merged_df_test.go` file with a full suite of unit tests for the `NewMergeDataframe` function.
* Tests cover scenarios such as empty input, single DataFrame input, multiple DataFrames with identical or differing schemas (testing positional mapping and implicit type compatibility behavior), and the inclusion of empty DataFrames.
* Added helper functions for test data creation and result comparison.
4. **In-Memory Series Value Test (`df/inmemory/series_val_test.go`):**
* Addressed a `TODO` by adding an `assert.PanicsWithValue` to verify that calling `GetAsString()` on a nil `StringValue` correctly panics with the expected message "GetAsString() called on a nil value".
5. **FNS Series Groups Tests (`df/fns/series/groups_test.go`):**
* Removed the empty test functions `TestDescribe` and `TestCovariance` as the corresponding `Describe` and `Covariance` functions are not implemented in `df/fns/series/groups.go`.
These changes improve the robustness of the DataFrame library by increasing test coverage, ensuring correct behavior for various join types, and cleaning up placeholder tests.
Feat/arrow enhancements
…tests
This commit addresses missing methods and test cases for the Arrow-based
DataFrame implementation, improving its completeness and consistency.
Completed Steps:
1. **Enhanced `df.GroupedDataFrame` Interface and Implementations:**
* Added an `AggregationConfig` struct to `df/dataframe.go`.
* Added an `Agg(configs ...AggregationConfig) DataFrame` method to the
`df.GroupedDataFrame` interface.
* Adjusted `arrowGroupedDataFrame.Agg` to use the shared
`df.AggregationConfig` and verified its functionality.
* Implemented the `Agg` method for `inmemoryGroupedDataFrame`,
supporting functions like sum, mean, count, min, max.
2. **Standardized Grouping Method Name:**
* Renamed `inmemoryDataFrame.Group` to `GroupBy` for consistency
with `arrowDataFrame` and common library conventions.
* Updated calls in `df/inmemory/df_test.go`.
3. **Addressed Missing `Select` in In-Memory Implementation:**
* Implemented the `Select(expressions ...df.Expr) DataFrame` method
in `df/inmemory/df.go`.
* The initial implementation supports `ColNameExpr` (column selection),
`LiteralExpr` (literal values), and aliasing via `expr.Name()`.
4. **Completed Basic Test Cases for `arrowDataFrame`:**
* Implemented a comprehensive suite of basic test cases in
`df/arrow/df_test.go`.
* Covered constructors (`NewArrowDataFrame`,
`NewArrowDataFrameFromArrays`), accessors (`Schema`, `Name`, `Len`,
`GetSeries`, `GetRow`, `GetValue`), data manipulation methods (`Limit`,
`Sort`, `MapRow`, `FlatMapRow`, `Distinct`, `Append`, `Union`,
`WhenNil`, `When`, `Intersection`, `UpdateSeries`, `AsFormat`, `Rename`),
iteration (`ForEachRow`), and error/panic conditions.
5. **Added Comprehensive Tests for `arrowGroupedDataFrame`:**
* Created a new test file `df/arrow/grouped_df_test.go`.
* Implemented tests for all `arrowGroupedDataFrame` methods:
`GetGroupColumns`, `Len`, `GetKeys`, `Get`, `ForEach`, `Map`, `Where`.
* Added extensive tests for the `Agg` method, covering various
aggregation functions, multiple aggregations, different data types,
and edge cases (empty groups, nulls).
6. **Reviewed and Added Edge Case Tests for Arrow Joins:**
* Enhanced existing join tests in `df/arrow/df_test.go` for all join
types (Inner, Left, Right, FullOuter, Cross, LeftAnti, LeftSemi,
RightSemi, RightAnti).
* Added specific sub-tests for joining with empty DataFrames,
no matching keys, and handling of nulls in join keys.
7. **Added Tests for `arrowDataFrame.Select` with Expressions:**
* Implemented tests in `df/arrow/df_test.go` (within the existing
`TestDataFrame_Select` function) for the `arrowDataFrame.Select`
method, focusing on various expression types.
* Covered selection of columns, literals, aliasing, binary operations
(column-literal and column-column for arithmetic and comparisons),
unary operations (cast, isnull), and error conditions (column not
found, type mismatches).
feat: Enhance Arrow DataFrame and GroupedDataFrame functionality and …
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.