feat(RFC): A richer `Expr` IR by dangotbanned · Pull Request #2572 · narwhals-dev/narwhals

dangotbanned · 2025-05-18T20:39:27Z

Will close #2571

What type of PR is this? (check all applicable)

✨ Feature

Related issues

Closes [Enh]: A richer Expr internal representation #2571
- See Related section there for more links
Search PRs: (sort:updated-desc "(expr-ir)/" in:title)

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

If you have comments or can explain your changes, please do so below

Important

See (#2571) for detail!!!!!!!

Very open to feedback

Tasks

Show 2025 May-July

dangotbanned · 2025-05-19T16:12:54Z

`FunctionFlags`, `FunctionOptions`, `Function`, `FunctionExpr`

Feel like there's more I need to understand on the correct way to propagate the flags to the top-level.

@MarcoGorelli whenever you get to this - I've left loads of notes w/ references - hoping that you'd be able to demystify the rust magic 😄

All 4 classes

They're a bit trimmed down from the polars versions, where I couldn't see a need for us to mirror everything if we wouldn't use it

narwhals/narwhals/_plan/options.py

Lines 13 to 49 in 0bada48

    
           class FunctionFlags(enum.Flag): 
        
               ALLOW_GROUP_AWARE = 1 << 0 
        
               """> Raise if use in group by 
        
               Not sure where this is disabled. 
        
               """ 
        
               INPUT_WILDCARD_EXPANSION = 1 << 4 
        
               """Appears on all the horizontal aggs. 
        
               https://github.com/pola-rs/polars/blob/e8ad1059721410e65a3d5c1d84055fb22a4d6d43/crates/polars-plan/src/plans/options.rs#L49-L58 
        
               """ 
        
               RETURNS_SCALAR = 1 << 5 
        
               """Automatically explode on unit length if it ran as final aggregation.""" 
        
               ROW_SEPARABLE = 1 << 8 
        
               """Not sure lol. 
        
               https://github.com/pola-rs/polars/pull/22573 
        
               """ 
        
               LENGTH_PRESERVING = 1 << 9 
        
               """mutually exclusive with `RETURNS_SCALAR`""" 
        
               def is_elementwise(self) -> bool: 
        
                   return self in (FunctionFlags.ROW_SEPARABLE | FunctionFlags.LENGTH_PRESERVING) 
        
               def returns_scalar(self) -> bool: 
        
                   return self in FunctionFlags.RETURNS_SCALAR 
        
               def is_length_preserving(self) -> bool: 
        
                   return self in FunctionFlags.LENGTH_PRESERVING 
        
               @staticmethod 
        
               def default() -> FunctionFlags: 
        
                   return FunctionFlags.ALLOW_GROUP_AWARE

narwhals/narwhals/_plan/options.py

Lines 52 to 108 in 0bada48

    
           class FunctionOptions(Immutable): 
        
               """ExprMetadata` but less god object. 
        
               https://github.com/pola-rs/polars/blob/3fd7ecc5f9de95f62b70ea718e7e5dbf951b6d1c/crates/polars-plan/src/plans/options.rs 
        
               """ 
        
               __slots__ = ("flags",) 
        
               flags: FunctionFlags 
        
               def is_elementwise(self) -> bool: 
        
                   return self.flags.is_elementwise() 
        
               def returns_scalar(self) -> bool: 
        
                   return self.flags.returns_scalar() 
        
               def is_length_preserving(self) -> bool: 
        
                   return self.flags.is_length_preserving() 
        
               def with_flags(self, flags: FunctionFlags, /) -> FunctionOptions: 
        
                   if (FunctionFlags.RETURNS_SCALAR | FunctionFlags.LENGTH_PRESERVING) in flags: 
        
                       msg = "A function cannot both return a scalar and preserve length, they are mutually exclusive." 
        
                       raise TypeError(msg) 
        
                   obj = FunctionOptions.__new__(FunctionOptions) 
        
                   object.__setattr__(obj, "flags", self.flags | flags) 
        
                   return obj 
        
               def with_elementwise(self) -> FunctionOptions: 
        
                   return self.with_flags( 
        
                       FunctionFlags.ROW_SEPARABLE | FunctionFlags.LENGTH_PRESERVING 
        
                   ) 
        
               @staticmethod 
        
               def default() -> FunctionOptions: 
        
                   obj = FunctionOptions.__new__(FunctionOptions) 
        
                   object.__setattr__(obj, "flags", FunctionFlags.default()) 
        
                   return obj 
        
               @staticmethod 
        
               def elementwise() -> FunctionOptions: 
        
                   return FunctionOptions.default().with_elementwise() 
        
               @staticmethod 
        
               def row_separable() -> FunctionOptions: 
        
                   return FunctionOptions.groupwise().with_flags(FunctionFlags.ROW_SEPARABLE) 
        
               @staticmethod 
        
               def length_preserving() -> FunctionOptions: 
        
                   return FunctionOptions.default().with_flags(FunctionFlags.LENGTH_PRESERVING) 
        
               @staticmethod 
        
               def groupwise() -> FunctionOptions: 
        
                   return FunctionOptions.default() 
        
               @staticmethod 
        
               def aggregation() -> FunctionOptions: 
        
                   return FunctionOptions.groupwise().with_flags(FunctionFlags.RETURNS_SCALAR)

narwhals/narwhals/_plan/common.py

Lines 149 to 172 in 0bada48

    
           class Function(ExprIR): 
        
               """Shared by expr functions and namespace functions. 
        
               https://github.com/pola-rs/polars/blob/112cab39380d8bdb82c6b76b31aca9b58c98fd93/crates/polars-plan/src/dsl/expr.rs#L114 
        
               """ 
        
               @property 
        
               def function_options(self) -> FunctionOptions: 
        
                   from narwhals._plan.options import FunctionOptions 
        
                   return FunctionOptions.default() 
        
               @property 
        
               def is_scalar(self) -> bool: 
        
                   return self.function_options.returns_scalar() 
        
               def to_function_expr(self, *inputs: ExprIR) -> FunctionExpr[Self]: 
        
                   from narwhals._plan.expr import FunctionExpr 
        
                   from narwhals._plan.options import FunctionOptions 
        
                   # NOTE: Still need to figure out how these should be generated 
        
                   # Feel like it should be the union of `input` & `function` 
        
                   PLACEHOLDER = FunctionOptions.default()  # noqa: N806 
        
                   return FunctionExpr(input=inputs, function=self, options=PLACEHOLDER)

narwhals/narwhals/_plan/expr.py

Lines 157 to 185 in 0bada48

    
           class FunctionExpr(ExprIR, t.Generic[_FunctionT]): 
        
               """**Representing `Expr::Function`**. 
        
               https://github.com/pola-rs/polars/blob/dafd0a2d0e32b52bcfa4273bffdd6071a0d5977a/crates/polars-plan/src/dsl/expr.rs#L114-L120 
        
               https://github.com/pola-rs/polars/blob/112cab39380d8bdb82c6b76b31aca9b58c98fd93/crates/polars-plan/src/dsl/function_expr/mod.rs#L123 
        
               """ 
        
               __slots__ = ("function", "input", "options") 
        
               input: Seq[ExprIR] 
        
               function: _FunctionT 
        
               """Enum type is named `FunctionExpr` in `polars`. 
        
               Mirroring *exactly* doesn't make much sense in OOP. 
        
               https://github.com/pola-rs/polars/blob/112cab39380d8bdb82c6b76b31aca9b58c98fd93/crates/polars-plan/src/dsl/function_expr/mod.rs#L123 
        
               """ 
        
               options: FunctionOptions 
        
               """Assuming this is **either**: 
        
               1. `function.function_options` 
        
               2. The union of (1) and any `FunctionOptions` in `inputs` 
        
               """ 
        
               def with_options(self, options: FunctionOptions, /) -> Self: 
        
                   options = self.options.with_flags(options.flags) 
        
                   return type(self)(input=self.input, function=self.function, options=options)

- Mentioned in (#2391 (comment)) - Needed again for #2572

MarcoGorelli · 2025-05-21T08:58:53Z

@MarcoGorelli whenever you get to this - I've left loads of notes w/ references - hoping that you'd be able to demystify the rust magic 😄

at the moment it looks like this adds a self-standing _plan, that's not integrated into the rest of Narwhals? if it's possible to integrate it with the rest to prove that this is feasible, i'd be very interested in taking a close look

* chore(typing): Add `_typing_compat.py` - Mentioned in (#2391 (comment)) - Needed again for #2572 * refactor: Reuse `TypeVar` import * refactor: Reuse `@deprecated` import * refactor: Reuse `Protocol38` import * docs: Add module-level docstring

Still need: - reprs - fix the hierarchy issue (#2572 (comment)) - Flag summing (#2572 (comment))

Resolves (#2572 (comment))

- 1 step closer to the understanding for (#2572 (comment)) - There's still some magic going on when `polars` serializes - Need to track down where `'collect_groups': 'ElementWise'` and `'collect_groups': 'GroupWise'` first appear - Seems like the flags get reduced

dangotbanned · 2025-05-22T12:40:02Z

Thanks for peeking @MarcoGorelli

at the moment it looks like this adds a self-standing _plan, that's not integrated into the rest of Narwhals? if it's possible to integrate it with the rest to prove that this is feasible, i'd be very interested in taking a close look

That is definitely the eventual goal! 🤞

Despite how quickly things have progressed, I still feel I'm a few steps behind being ready for that just yet.

General overview

I'm trying to focus on modeling these structures and how they interact:

My thought was that narwhals currently solves similar problems, but in different ways to polars.
By getting this subset of polars that we need translated from rust -> python first, we're then in a good position to make decisions on how to bridge any gaps that narwhals is occupying at the moment, if that makes sense?

So like what I have in narwhals/_plan/dummy.py is all about creating an accurate expression graph.
Consuming the graph (evaluating an expression) is a very important step - but the main difference I'm anticipating is:

Current

Expressions are based on evaluating callables
with other callables used for output names & aliases
They also optionally store a ExprMetadata object
Some backends do other things with function names, kwargs, depth
Lazy backends use more callables for handling windows

`ExprIR`

Expressions are (likely still) based on evaluating callables
All of the remaining details are either:
- Encoded into a node on the graph
  - E.g. the nodes for Expr.var, Expr.std have the name and ddof already
  - Depth can be computed anywhere by traversing the graph
- Or, can be expressed in terms of operating on a node
  - meta namespace, examples
  - WindowExpr is just another albeit complex 😅 node

I am confident we'll end up with something that's easier to maintain - but trying to integrate the two mid-solve and maintaining that branch over time seems like it'd be a real challenge 😔

`FunctionOptions`, `FunctionFlags`

The parts mentioned in (#2572 (comment)) ~~are some~~ were one of the main hurdles left.
However I think it is really just a skill issue on my part in understanding the rust code. That was all I was hoping for some help with 🙏
In narwhals terms, it is closest to (https://github.com/narwhals-dev/narwhals/blob/1b93c0ed2dc2bf47d7b8e4b4cab4c9e9cad59800/narwhals/_expression_parsing.py) but only a subset of the rules - since it only concerns functions and how they compose.

Note

Right before I was about to send this comment, I managed to fix the issue I had by updating to 1.30.0 🤦‍♂️
See (diff), which removed the flags I was having trouble finding in (0982b3a)

Example

Now all the flags I've been using are propagated in the same way as in polars! 🎉

Repro code

import polars as pl

from narwhals._plan import demo as nwd  # noqa: F811
from narwhals._plan import meta  # noqa: F811

expr_pl = (
    pl.col("a")
    .sort()
    .fill_null(1)
    .shift(1)
    .abs()
    .drop_nulls()
    .skew()
    .alias("col->sort->fill_null->shift->abs->drop_nulls->skew->alias")
)
expr_nwd = (
    nwd.col("a")
    .sort()
    .fill_null(1)
    .shift(1)
    .abs()
    .drop_nulls()
    .skew()
    .alias("col->sort->fill_null->shift->abs->drop_nulls->skew->alias")
)
roundtrip_pl = meta.polars_expr_to_dict(expr_pl)
roundtrip_nw = str(expr_nwd._ir)

roundtrip_pl

I haven't added ALLOW_EMPTY_INPUTS - so that's an expected difference between the two

>>> roundtrip_pl

{'Alias': [{'Function': {'input': [{'Function': {'input': [{'Function': {'input': [{'Function': {'input': [{'Function': {'input': [{'Sort': {'expr': {'Column': 'a'},
                   'options': {'descending': False,
                    'nulls_last': False,
                    'multithreaded': True,
                    'maintain_order': False,
                    'limit': None}}},
                 {'Literal': {'Dyn': {'Int': 1}}}],
                'function': 'FillNull',
                'options': {'check_lengths': True,
                 'flags': 'ALLOW_GROUP_AWARE | ROW_SEPARABLE | LENGTH_PRESERVING'}}},
              {'Literal': {'Dyn': {'Int': 1}}}],
             'function': 'Shift',
             'options': {'check_lengths': True,
              'flags': 'ALLOW_GROUP_AWARE | LENGTH_PRESERVING'}}}],
          'function': 'Abs',
          'options': {'check_lengths': True,
           'flags': 'ALLOW_GROUP_AWARE | ROW_SEPARABLE | LENGTH_PRESERVING'}}}],
       'function': 'DropNulls',
       'options': {'check_lengths': True,
        'flags': 'ALLOW_GROUP_AWARE | ALLOW_EMPTY_INPUTS | ROW_SEPARABLE'}}}],
    'function': {'Skew': True},
    'options': {'check_lengths': True,
     'flags': 'ALLOW_GROUP_AWARE | RETURNS_SCALAR'}}},
  'col->sort->fill_null->shift->abs->drop_nulls->skew->alias']}

roundtrip_nw

To produce this I removed the outer ", " and pasted back to ruff to format

The overall shape is very similar and the deviations from polars have been documented

Alias(
    expr=FunctionExpr(
        function=Skew(),
        input=[
            FunctionExpr(
                function=DropNulls(),
                input=[
                    FunctionExpr(
                        function=Abs(),
                        input=[
                            FunctionExpr(
                                function=Shift(n=1),
                                input=[
                                    FunctionExpr(
                                        function=FillNull(),
                                        input=[
                                            Sort(
                                                expr=Column(name="a"),
                                                options=SortOptions(
                                                    descending=False, nulls_last=False
                                                ),
                                            ),
                                            Literal(
                                                value=ScalarLiteral(
                                                    dtype=Unknown, value=1
                                                )
                                            ),
                                        ],
                                        options=FunctionOptions(
                                            flags="ALLOW_GROUP_AWARE | ROW_SEPARABLE | LENGTH_PRESERVING"
                                        ),
                                    )
                                ],
                                options=FunctionOptions(
                                    flags="ALLOW_GROUP_AWARE | LENGTH_PRESERVING"
                                ),
                            )
                        ],
                        options=FunctionOptions(
                            flags="ALLOW_GROUP_AWARE | ROW_SEPARABLE | LENGTH_PRESERVING"
                        ),
                    )
                ],
                options=FunctionOptions(flags="ALLOW_GROUP_AWARE | ROW_SEPARABLE"),
            )
        ],
        options=FunctionOptions(flags="ALLOW_GROUP_AWARE | RETURNS_SCALAR"),
    ),
    name="col->sort->fill_null->shift->abs->drop_nulls->skew->alias",
)

Can't tell if this means `FirstT` will match the entry `firstt`, but preserve the `firstt` fix (https://github.com/codespell-project/codespell#ignoring-words) (#2572 (comment))

#2572 (comment)

dangotbanned · 2025-05-23T14:23:40Z

I should've expected this, but it was a nice suprise to find we get hashable selectors for free 😄

from narwhals._plan import selectors as ndcs

>>> ndcs.matches("[^z]a")._ir == ndcs.matches("[^z]a")._ir
True

>>> ndcs.matches("[^z]a")._ir == ndcs.matches("abc")._ir
False

@MarcoGorelli regarding (#2291)

from narwhals._plan import selectors as ndcs

>>> ndcs.all()._ir == ndcs.all()._ir
True

lhs = ndcs.all()
rhs = ndcs.all().mean()

>>> lhs._ir == rhs._ir
False

>>> lhs._ir == rhs._ir.expr
True

And the same holds for the non-selectors all 🥳

from narwhals._plan import demo as nwd

lhs = nwd.all()
rhs = nwd.all().mean()
>>> lhs._ir == rhs._ir
False

>>> lhs._ir == rhs._ir.expr
True

>>> type(rhs._ir)
narwhals._plan.aggregation.Mean

Related #2572 (comment)

An experiment towards (#2572 (comment))

dangotbanned · 2025-05-28T14:59:22Z

+def test_valid_windows() -> None:
+    """Was planning to test this matched, but we seem to allow elementwise horizontal?
+
+    https://github.com/narwhals-dev/narwhals/blob/63c8e4771a1df4e0bfeea5559c303a4a447d5cc2/tests/expression_parsing_test.py#L10-L45
+    """
+    ELEMENTWISE_ERR = re.compile(r"cannot use.+over.+elementwise", re.IGNORECASE)  # noqa: N806
+    a = nwd.col("a")
+    assert a.cum_sum()
+    assert a.cum_sum().over(order_by="id")
+    with pytest.raises(InvalidOperationError, match=ELEMENTWISE_ERR):
+        assert a.cum_sum().abs().over(order_by="id")
+
+    assert (a.cum_sum() + 1).over(order_by="id")
+    assert a.cum_sum().cum_sum().over(order_by="id")
+    assert a.cum_sum().cum_sum()
+    assert nwd.sum_horizontal(a, a.cum_sum())
+    with pytest.raises(InvalidOperationError, match=ELEMENTWISE_ERR):
+        assert nwd.sum_horizontal(a, a.cum_sum()).over(order_by="a")
+
+    assert nwd.sum_horizontal(a, a.cum_sum().over(order_by="i"))
+    assert nwd.sum_horizontal(a.diff(), a.cum_sum().over(order_by="i"))
+    with pytest.raises(InvalidOperationError, match=ELEMENTWISE_ERR):
+        assert nwd.sum_horizontal(a.diff(), a.cum_sum()).over(order_by="i")
+
+    with pytest.raises(InvalidOperationError, match=ELEMENTWISE_ERR):
+        assert nwd.sum_horizontal(a.diff().abs(), a.cum_sum()).over(order_by="i")


@MarcoGorelli quick question

This is adapted from an existing test:

tests.expression_parsing_test.test_window_kind

narwhals/tests/expression_parsing_test.py

Lines 10 to 45 in 63c8e47

@pytest.mark.parametrize(

("expr", "expected"),

[

(nw.col("a"), 0),

(nw.col("a").mean(), 0),

(nw.col("a").cum_sum(), 1),

(nw.col("a").cum_sum().over(order_by="id"), 0),

(nw.col("a").cum_sum().abs().over(order_by="id"), 1),

((nw.col("a").cum_sum() + 1).over(order_by="id"), 1),

(nw.col("a").cum_sum().cum_sum().over(order_by="id"), 1),

(nw.col("a").cum_sum().cum_sum(), 2),

(nw.sum_horizontal(nw.col("a"), nw.col("a").cum_sum()), 1),

(nw.sum_horizontal(nw.col("a"), nw.col("a").cum_sum()).over(order_by="a"), 1),

(nw.sum_horizontal(nw.col("a"), nw.col("a").cum_sum().over(order_by="i")), 0),

(

nw.sum_horizontal(

nw.col("a").diff(), nw.col("a").cum_sum().over(order_by="i")

),

1,

),

(

nw.sum_horizontal(nw.col("a").diff(), nw.col("a").cum_sum()).over(

order_by="i"

),

2,

),

(

nw.sum_horizontal(nw.col("a").diff().abs(), nw.col("a").cum_sum()).over(

order_by="i"

),

2,

),

],

)

def test_window_kind(expr: nw.Expr, expected: int) -> None:

assert expr._metadata.n_orderable_ops == expected

AFAICT, all of the expressions I've needed a InvalidOperationError for shouldn't be valid.

But they aren't raising in current narwhals 🤔

1

import narwhals as nw a = nw.col("a") a.cum_sum().abs().over(order_by="id")

This error explicitly mentions abs

narwhals/narwhals/_expression_parsing.py

Lines 357 to 362 in 9bd10ad

if self.is_elementwise or self.is_filtration:

msg = (

"Cannot use `over` on expressions which are elementwise\n"

"(e.g. `abs`) or which change length (e.g. `drop_nulls`)."

)

raise InvalidOperationError(msg)

2, 3, 4

These are all raising the same as (1), but the issue seems to be that horizontal functions aren't being treated as elementwise

import narwhals as nw a = nw.col("a") nw.sum_horizontal(a, a.cum_sum()).over(order_by="a") nw.sum_horizontal(a.diff(), a.cum_sum()).over(order_by="i") nw.sum_horizontal(a.diff().abs(), a.cum_sum()).over(order_by="i")

In polars, they all seem to be elementwise but with an additional flag

https://github.com/pola-rs/polars/blob/944bf553f1111c31259b55348a7dd0a512ae51a1/crates/polars-plan/src/dsl/function_expr/mod.rs#L1388-L1392

I've done the same in this PR, but I don't think that flag would factor into this?

narwhals/narwhals/_plan/functions.py

Lines 291 to 299 in 9bd10ad

class SumHorizontal(Function):

@property

def function_options(self) -> FunctionOptions:

return FunctionOptions.elementwise().with_flags(

FunctionFlags.INPUT_WILDCARD_EXPANSION

)

def __repr__(self) -> str:

return "sum_horizontal"

`lit(Series)` was really bugging me with how it should be showing the backend

commit 9d015e2031acd5e10404da72a7d698c632446c49 Author: Marco Edward Gorelli <33491632+MarcoGorelli@users.noreply.github.com> Date: Sat May 16 09:51:39 2026 +0100 release: Bump version to 2.21.2 (#3630) commit 8674e44f657110b45aedffb8c1343d5b0505bb01 Author: Marco Edward Gorelli <33491632+MarcoGorelli@users.noreply.github.com> Date: Sat May 16 09:37:56 2026 +0100 release: Bump version to 2.21.1 (#3628) commit 739dc579adfd38a75764d6d2c09ed42b0f2e887a Author: Marco Edward Gorelli <33491632+MarcoGorelli@users.noreply.github.com> Date: Sat May 16 09:35:33 2026 +0100 ci: remove `downstream_tests_slow` (#3629) commit 1f36279bb07203628ddd18a691cf98d4c78780d3 Author: Marco Edward Gorelli <33491632+MarcoGorelli@users.noreply.github.com> Date: Sat May 16 09:17:10 2026 +0100 Revert "fix: Allow `float('nan')` as value in join for duckdb (#3555)" (#3627) This reverts commit 0d7f352. commit 1e50d020e3ca5e5c5dfe3486e53190e87c60b66c Author: Pedro <pedro.villanueva@booking.com> Date: Fri May 15 13:05:01 2026 +0200 [Enh]: Add the negation unary operator for expressions and series (#3625) negation unary operator for expressions and series commit 37ea7953f6b455736bf6ae353ca257f468b3b083 Author: Francesco Bruzzesi <42817048+FBruzzesi@users.noreply.github.com> Date: Wed May 13 10:38:42 2026 +0200 ci: Unpin "temporary" CI pins (#3618) * ci: Unpin 'temporary' CI pins * rollback formulaic and pointblank commit 4ff3a1f545924e5e38e76c4d8a71de2010a2f4d6 Author: Francesco Bruzzesi <42817048+FBruzzesi@users.noreply.github.com> Date: Tue May 12 19:11:11 2026 +0200 chore: Prepare for future pandas inplace deprecation (#3616)

This reverts commit 5dee22d.

Accidentally untracked all of these lol

That move destroyed the history

for more information, see https://pre-commit.ci

…3654)

dangotbanned mentioned this pull request May 18, 2025

[Enh]: A richer Expr internal representation #2571

Open

7 tasks

dangotbanned added enhancement New feature or request api design needs discussion labels May 18, 2025

dangotbanned added a commit that referenced this pull request May 20, 2025

chore(typing): Add _typing_compat.py

603c4e9

- Mentioned in (#2391 (comment)) - Needed again for #2572

dangotbanned mentioned this pull request May 20, 2025

chore: Add _typing_compat module #2578

Merged

10 tasks

dangotbanned commented May 21, 2025

View reviewed changes

Comment thread narwhals/_plan/common.py Outdated

dangotbanned added a commit that referenced this pull request May 21, 2025

feat: Add methods for all of functions

be6ec3e

Still need: - reprs - fix the hierarchy issue (#2572 (comment)) - Flag summing (#2572 (comment))

dangotbanned added a commit that referenced this pull request May 21, 2025

fix: Remove ExprIR from Function bases

e6ca72b

Resolves (#2572 (comment))

dangotbanned commented May 21, 2025

View reviewed changes

Comment thread narwhals/_plan/functions.py Outdated

dangotbanned mentioned this pull request May 22, 2025

Add Selector.__xor__ (symmetric difference) #2589

Open

This comment was marked as resolved.

Sign in to view

dangotbanned added a commit that referenced this pull request May 23, 2025

ci: Maybe codespell ignore FirstT

fb303f6

Can't tell if this means `FirstT` will match the entry `firstt`, but preserve the `firstt` fix (https://github.com/codespell-project/codespell#ignoring-words) (#2572 (comment))

MarcoGorelli reviewed May 23, 2025

View reviewed changes

Comment thread .pre-commit-config.yaml Outdated

dangotbanned added a commit that referenced this pull request May 23, 2025

ci: fix codespell

89c4587

#2572 (comment)

dangotbanned mentioned this pull request May 23, 2025

enh: improve when-then-otherwise to include chaining #668

Closed

dangotbanned commented May 23, 2025

View reviewed changes

Comment thread pyproject.toml

dangotbanned added a commit that referenced this pull request May 24, 2025

chore(ruff): Add tool.ruff.lint.typing-modules

7c4c6ee

Related #2572 (comment)

dangotbanned mentioned this pull request May 24, 2025

chore(ruff): Add tool.ruff.lint.typing-modules #2610

Merged

10 tasks

dangotbanned added a commit that referenced this pull request May 26, 2025

feat(DRAFT): Mock up to_compliant as an adapter

92694ce

An experiment towards (#2572 (comment))

dangotbanned commented May 28, 2025

View reviewed changes

Comment thread narwhals/_namespace.py

dangotbanned mentioned this pull request May 31, 2025

Tracking: Drop support for python 3.8 #2084

Closed

5 tasks

dangotbanned commented May 31, 2025

View reviewed changes

Comment thread tests/plan/expr_parsing_test.py Outdated

dangotbanned mentioned this pull request May 12, 2026

fix(expr-ir): Add compat for old polars #3617

Merged

13 tasks

fix(expr-ir): Add compat for old polars (#3617)

a2beffb

dangotbanned mentioned this pull request May 13, 2026

fix(expr-ir): Backport __replace__ typing for <3.13 #3620

Merged

fix(expr-ir): Backport __replace__ typing for <3.13 (#3620)

474882c

dangotbanned mentioned this pull request May 14, 2026

ci(expr-ir): Fix coverage/tests in ci #3622

Merged

ci(expr-ir): Fix coverage/tests in ci (#3622)

06666b0

dangotbanned mentioned this pull request May 14, 2026

fix(expr-ir): Add more compat for old polars #3624

Merged

2 tasks

dangotbanned added 3 commits May 14, 2026 22:17

fix(expr-ir): Add more compat for old polars (#3624)

8d491b0

refactor: Rename input -> args

1dddb85

feat: Display idomatic name in LitSeries repr

10857f9

`lit(Series)` was really bugging me with how it should be showing the backend

This was referenced May 16, 2026

ci: Improving generate_random_versions #3623

Open

feat(expr-ir): Improve lit repr #3641

Merged

dangotbanned added 3 commits May 19, 2026 11:02

feat(expr-ir): Improve lit repr (#3641)

0af706d

Revert "Squashed commit of the following:"

ee795d8

This reverts commit 5dee22d.

dangotbanned mentioned this pull request May 21, 2026

docs(expr-ir): Explain FunctionExpr #3645

Merged

dangotbanned added 3 commits May 21, 2026 18:46

docs(expr-ir): Explain FunctionExpr (#3645)

33c8d51

Merge remote-tracking branch 'upstream/main' into oh-nodes

85c9123

chore: move _plan to src/narwhals/

05c940f

dangotbanned mentioned this pull request May 21, 2026

Enable allow-redefinition for mypy #3646

Closed

dangotbanned and others added 4 commits May 21, 2026 20:11

fix: Avoid local .gitignore bug

8775915

Accidentally untracked all of these lol

ci: add .git-blame-ignore-revs

7b7bb58

That move destroyed the history

[pre-commit.ci] auto fixes from pre-commit.com hooks

08a2eda

for more information, see https://pre-commit.ci

chore: touch files that were missing

8d0ab34

dangotbanned mentioned this pull request May 24, 2026

feat(expr-ir): Add top-level select #3647

Merged

dangotbanned added 2 commits May 25, 2026 08:25

Merge branch 'main' into oh-nodes

d482fff

feat(expr-ir): Add top-level select (#3647)

edc349a

dangotbanned mentioned this pull request May 25, 2026

fix(expr-ir): Allow struct.field to select from it's own context #3654

Merged

dangotbanned added 2 commits May 25, 2026 21:56

fix(expr-ir): Allow struct.field to select from it's own context (#…

c5719de

…3654)

Merge remote-tracking branch 'upstream/main' into oh-nodes

32ebac4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(RFC): A richer `Expr` IR#2572

feat(RFC): A richer `Expr` IR#2572
dangotbanned wants to merge 518 commits into
mainfrom
oh-nodes

dangotbanned commented May 18, 2025 •

edited

Loading

Uh oh!

dangotbanned commented May 19, 2025 •

edited

Loading

Uh oh!

Uh oh!

MarcoGorelli commented May 21, 2025

Uh oh!

Uh oh!

dangotbanned commented May 22, 2025

Uh oh!

This comment was marked as resolved.

Uh oh!

dangotbanned commented May 23, 2025

Uh oh!

Uh oh!

dangotbanned May 28, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	class Selector(ExprIR):
	"""by_dtype, matches, numeric, boolean, string, categorical, datetime, all."""

	@pytest.mark.parametrize(
	("expr", "expected"),
	[
	(nw.col("a"), 0),
	(nw.col("a").mean(), 0),
	(nw.col("a").cum_sum(), 1),
	(nw.col("a").cum_sum().over(order_by="id"), 0),
	(nw.col("a").cum_sum().abs().over(order_by="id"), 1),
	((nw.col("a").cum_sum() + 1).over(order_by="id"), 1),
	(nw.col("a").cum_sum().cum_sum().over(order_by="id"), 1),
	(nw.col("a").cum_sum().cum_sum(), 2),
	(nw.sum_horizontal(nw.col("a"), nw.col("a").cum_sum()), 1),
	(nw.sum_horizontal(nw.col("a"), nw.col("a").cum_sum()).over(order_by="a"), 1),
	(nw.sum_horizontal(nw.col("a"), nw.col("a").cum_sum().over(order_by="i")), 0),
	(
	nw.sum_horizontal(
	nw.col("a").diff(), nw.col("a").cum_sum().over(order_by="i")
	),
	1,
	),
	(
	nw.sum_horizontal(nw.col("a").diff(), nw.col("a").cum_sum()).over(
	order_by="i"
	),
	2,
	),
	(
	nw.sum_horizontal(nw.col("a").diff().abs(), nw.col("a").cum_sum()).over(
	order_by="i"
	),
	2,
	),
	],
	)
	def test_window_kind(expr: nw.Expr, expected: int) -> None:
	assert expr._metadata.n_orderable_ops == expected

	if self.is_elementwise or self.is_filtration:
	msg = (
	"Cannot use `over` on expressions which are elementwise\n"
	"(e.g. `abs`) or which change length (e.g. `drop_nulls`)."
	)
	raise InvalidOperationError(msg)

	class SumHorizontal(Function):
	@property
	def function_options(self) -> FunctionOptions:
	return FunctionOptions.elementwise().with_flags(
	FunctionFlags.INPUT_WILDCARD_EXPANSION
	)

	def __repr__(self) -> str:
	return "sum_horizontal"

Conversation

dangotbanned commented May 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this? (check all applicable)

Related issues

Checklist

If you have comments or can explain your changes, please do so below

Tasks

Uh oh!

dangotbanned commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

FunctionFlags, FunctionOptions, Function, FunctionExpr

Uh oh!

Uh oh!

MarcoGorelli commented May 21, 2025

Uh oh!

Uh oh!

dangotbanned commented May 22, 2025

General overview

Current

ExprIR

FunctionOptions, FunctionFlags

Example

Uh oh!

This comment was marked as resolved.

Uh oh!

dangotbanned commented May 23, 2025

Uh oh!

Uh oh!

dangotbanned May 28, 2025

Choose a reason for hiding this comment

1

2, 3, 4

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dangotbanned commented May 18, 2025 •

edited

Loading

dangotbanned commented May 19, 2025 •

edited

Loading

`FunctionFlags`, `FunctionOptions`, `Function`, `FunctionExpr`

`ExprIR`

`FunctionOptions`, `FunctionFlags`