Repository navigation
EF-322: Native LINQ translation - #374
Closed
ajcvickers wants to merge 186 commits into
Closed
ajcvickers wants to merge 186 commits into
ajcvickers wants to merge 186 commits into
Conversation
…-358, EF-376, EF-379, EF-380 and merged upstream PRs mongodb#339, mongodb#350, mongodb#353, mongodb#355, mongodb#357, mongodb#360, mongodb#366, mongodb#367, mongodb#368
Squashes 3 commits: - EF-322: Native LINQ Overhaul (squashed: native overhaul + rebase fallout fix + $limit:0 test update) - Delete docs/failing-spec-tests.md - EF-322: drop machine-specific git pull allow-rule from .claude/settings.json
…gaps Also: EF-340, EF-343, EF-354, EF-355, EF-357, EF-358, EF-359, EF-365, EF-366, EF-408, EF-409, EF-425, EF-434, EF-353, EF-382, EF-394, EF-395, EF-396, EF-397, EF-402, EF-410, EF-413, EF-422, EF-423, EF-424, EF-427, EF-428, EF-404 Squashes 3 commits: - EF-340,EF-343,EF-354,EF-355,EF-357,EF-358,EF-359,EF-365,EF-366,EF-408,EF-409,EF-425,EF-434: fix native-query bugs classified as fixable now - EF-353,EF-382,EF-394,EF-395,EF-396,EF-397,EF-402,EF-410,EF-413,EF-422,EF-423,EF-424,EF-427,EF-428: close small native-LINQ translation gaps - EF-404: Guard native $expr field-to-field comparisons against value-converted operands
Also: EF-411 Reworded from: EF-411: native Reverse/Last/LastOrDefault for the ordered-source case
Reworded from: Only report failures in native mode when the case doesn't fail in driver mode.
…o and ThenInclude chains Also: EF-392, EF-439, EF-407 Squashes 4 commits: - EF-392: investigate native Join/LeftJoin translation; blocked by EF Core limitation (EF-439) - EF-407: re-measure and close - deep ThenInclude through an embedded hop no longer silently null - EF-392: native sibling reference Includes, reference+collection combo, and reference ThenInclude chains - Fix stale/ambiguous XML doc-comment crefs causing build warnings
…access Also: EF-215, EF-442 Squashes 2 commits: - EF-215: native translation for an enum-to-enum cast in a projection/filter - EF-442: native translation for conditional expressions and DateTime/DateTimeOffset date-part access
Also: EF-419, EF-412 Squashes 3 commits: - EF-419: Remove dead RawBsonDocument shaper branch left unreachable by SP7 Phase 1 - EF-412: native translation for a whole-root-entity leaf mixed into a projection - Ignore .superpowers/ SDD scratch workspace directory
… GroupJoin shape triage Also: EF-392, EF-444, EF-436 Squashes 3 commits: - EF-392: native single-level Join/LeftJoin translation - EF-444: native $project support for a join's whole-entity Outer/Inner leaf - EF-436: GroupJoin shapes not translated — resolve, re-tag mislabeled tests, fix EF8/EF9 LeftJoin recognition
Also: EF-421, EF-441, EF-447 Squashes 3 commits: - EF-421: native translation for correlated owned-collection element predicates - EF-441: native translation for an owned single-reference navigation entity leaf in a projection - EF-447: native translation for a constructed sub-entity leaf mixed with a computed sibling in a projection
Also: EF-448 Reworded from: EF-448: native $concat translation for string concatenation with non-string operands
…ation projected inline Also: EF-449 Reworded from: EF-449: native translation for FirstOrDefault()/First() over a reference-collection nav projected inline
Also: EF-450 Reworded from: EF-450: native translation for a collection-then-collection ThenInclude
Also: EF-451 Reworded from: EF-451: native translation for a scalar aggregate terminating directly on a bare GroupBy(key)
…x term Also: EF-452 Squashes 2 commits: - EF-452: native translation for a parameterized string.StartsWith/EndsWith/Contains regex term - Fix ambiguous Debug.Assert cref warning; re-baseline stale MQL assertions
…stinct aggregates, entity equality, Equals, nav list projection) Also: EF-453 Squashes 7 commits: - EF-453: native translation for a scalar aggregate directly terminating on a projected Distinct - EF-322: native translation for whole-entity null/self equality checks - EF-322: native translation for instance and static Equals(...) method calls - EF-322: native translation for a reference-collection-nav list projected inline - EF-322: native translation for Contains over a concatenated column and constant - EF-453: native translation for root-entity structural equality (c == local, c == null, .Equals()) - EF-453: update stale native-entity-equality guard test now that Entity_equality_local goes native
Squashes 2 commits: - EF-322: recognize EF8/EF9's internal LeftJoin shim in the driver-LINQ bridge - EF-322: admit EF8/EF9's internal LeftJoin shim unconditionally
Squashes 16 commits: - docs: add native chained-join-scope design spec and implementation plan - test: pin chained-join Where/OrderBy/Any as currently falling back to driver-LINQ - refactor: make MongoJoinScope chain-capable (Levels list, depth-1 behavior unchanged) - EF-322: build JoinScope chain metadata eagerly for every eligible leading join run - EF-322: regression test for owned-nav-keyed depth-1 join eligibility widening - feat: NativeJoinScopeTranslator.TryTranslateRootScopeOnly for chained join scopes - feat: native Where/OrderBy/ThenBy over a chained join scope (root scope only) - feat: confirm a chained join via a whole-entity-leaves-only projection over any scope index - EF-322: fix round 1 for chained-join-scope task 6 review findings - EF-322: remove leftover debug console logging from IsSingleEligibleNativeJoinScope - EF-322: verify terminal Count and the post-confirmation Take/Skip guard over a chained join scope - docs: record native chained-join Where/OrderBy/terminal support in Query/AGENTS.md - EF-322: close last-only chain-lookup gap and restore root-scope-only guard parity - EF-322: fix stale/understated doc comments from prior chained-join-scope fix rounds - EF-322: fix misleading test comment and add ThenBy/alias-disambiguation coverage - EF-322: de-duplicate the native translator, add node-dispatcher coverage
Squashes 2 commits: - EF-322: trim Query/AGENTS.md to what agents actually need - EF-322: trim the remaining AGENTS.md files
… check and trailing ThenInclude Squashes 5 commits: - EF-322: go native for a compiled query's array-parameter element access - EF-322: native translation for a reference-Include's own null check - EF-322: go native for a reference Include with a trailing collection ThenInclude - EF-322: remove stale decline test superseded by the reference-Include null-check feature - EF-322: regenerate stale MQL baselines for the reference-Include null-check feature
…e, boxed OrderBy key and OrderBy after Distinct Squashes 4 commits: - EF-322: go native for literal boolean predicates (All_true and friends) - EF-322: go native for a filtered collection Include (OrderBy/Skip/Take) - EF-322: go native for a boxed captured-variable OrderBy key - EF-322: go native for an OrderBy/ThenBy composed after a projected Distinct
Squashes 2 commits: - docs: add native join Where-over-Inner-scope design spec and implementation plan - EF-322: native Where over a single-level join's Inner scope
… arithmetic and DateTime.AddXxx Squashes 7 commits: - EF-322: go native for Where/Skip/Take composed after a projected Distinct - EF-322: go native for checked-context arithmetic - EF-322: go native for Count(pred)/LongCount/Any/All composed after a projected Distinct - EF-322: go native for DateTime/DateTimeOffset.AddXxx() - EF-322: go native for Sum/Min/Max/Average with a selector over a projected Distinct - EF-322: go native for GroupBy(key).Select(aggregate) composed after a projected Distinct - EF-322: go native for First/Single reducers composed after a projected Distinct
Squashes 10 commits: - EF-TBD: design doc for native ctor-only DTO projection support - EF-TBD: implementation plan for native ctor-only DTO projection support - EF-TBD: widen TryGetProjectionMembers with an opt-in positional-ctor path - EF-TBD: GroupBy result selector admits a ctor-only DTO natively - EF-TBD: SelectMany result selector admits a ctor-only DTO natively - EF-TBD: extract bare-body leaf-translation into a reusable local function (no behavior change) - EF-TBD: native whole-entity ctor-wrap DTO projection + decline owned-nav-entity sub-case - EF-TBD: verify Entity_passed_to_DTO_constructor_works now goes native - EF-TBD: post-review fix wave for native-ctor-only-dto-projection - EF-TBD: add missing XML param docs on TryGetProjectionMembers (fix build warning)
Also: EF-149 Squashes 8 commits: - EF-322: go native for Min/Max over an order-preserving cast selector - EF-322: go native for whole-entity Distinct() - EF-322: go native for field-to-field StartsWith/EndsWith/Contains - EF-322: go native for a bare-scalar Distinct's own identity OrderBy - EF-322: go native for a projected Distinct() as a Union/Concat/Intersect/Except operand - EF-149: go native for GroupBy(key).Select(bare scalar aggregate) - EF-322: go native for g.Select(x).Distinct().<Op>() as a GroupBy accumulator - EF-149: update stale decline test made obsolete by the bare-scalar GroupBy carve-out
Squashes 5 commits: - EF-322: recognize string-to-char-sequence projection leaves in NativeProjectionBinder - EF-322: materialize native string-to-char-sequence projection leaves in the compiled shaper - EF-322: regenerate MQL baselines now that string-sequence projections go native - EF-322: keep string-sequence projections off driver push-down and fix the mixed shaper - EF-322: regenerate stale EF8/EF9 MQL baselines for native Include/navigation work
Reworded from: EF-322: go native for the null-coalescing operator (??)
Squashes 9 commits: - EF-TBD: native translation for a nested wrapped projection leaf sourced from a join scope - EF-TBD: mixed-visitor reads a dotted document-construction member via its full path - EF-TBD: differential-correctness test for nested join-scope projection, incl. unmatched FK - EF-TBD: baseline Include_with_complex_projection separately for EF8/EF9 - EF-TBD: guard native-join-scope nested projection test for EF8/EF9 - EF-TBD: nested join-scope leaf accepts only a MongoFieldExpression member - EF-TBD: one shared lookup for a staged MongoDocumentConstructionExpression - EF-TBD: commit the nested-join-scope-projection design/plan, corrected - EF-TBD: declare the MongoArrayReduceExpression node-coverage matrix row
…nt and collection Include over a join scope Squashes 3 commits: - EF-322: go native for a bare boolean query-parameter predicate root - EF-322: native Distinct/Count over a computed projection key - EF-322: go native for a collection Include sitting over a join scope
…lect Squashes 6 commits: - EF-TBD: add pending-ordering state for GroupBy OrderBy-before-Select - EF-TBD: resolve pending GroupBy orderings in TryBindGroupProjection - EF-TBD: lower GroupOrderOp between the $group stage and its flatten $project - EF-TBD: go native for GroupBy OrderBy/ThenBy composed before the terminal Select - EF-TBD: regenerate GroupBy-OrderBy-before-Select baselines now that they go native - EF-TBD: fix wave for GroupBy OrderBy-before-Select final review findings
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… a fresh comparison
…de-owned benchmark gaps
…a linear string.Equals chain The streaming materializer's fill loop compared each element name against every mapped member in turn, so a wide entity paid O(properties) compares per element. One Expression.Switch on the name replaces the chain; the last member added for a name still wins, as it did when the chain was wrapped last-first. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An owned collection element that owned a reference or a nested collection made the whole entity ineligible for the streaming materializer, so it took the DOM shaper and allocated like driver-LINQ. The rewriter now finds the element shaper under EF's Select(IncludeExpression) selector and threads the enclosing collection plans through RewriteMaterializer, so a nested owned entity's synthesized ordinal key resolves to its enclosing element's loop counter. StreamingEligibility admits owned subtrees inside an element and still rejects a non-owned navigation there, or a collection below an owned reference (which the rewriter cannot read). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
`Orders.Include(o => o.Account)` on a model with owned navigations nests the owned auto-includes and the reference Include in one selector, and TryWalkIncludeChain rejected any embedded sibling, so the query silently fell back to driver-LINQ (and threw under NativeOnly). An embedded sibling on the Include spine is now transparent, as an embedded ThenInclude hop already was, provided no non-embedded include is nested past it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
StreamingEligibility's owned-subtree walks skip an owned type's navigation back to its owner (they recursed forever on the cyclic graph). A collection element now resets its reused locals before each element, so a missing owned reference or nullable scalar no longer reads the previous element's value. Adds a test that a reference hosted by an owned collection element beside the reference Include is never silently dropped. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tcher args to the benchmarks Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…o strings The streaming materializer's fill loop read each element name with ReadName(), which allocates a new string per element per row. It now passes a stateless INameDecoder backed by a BsonTrie of the level's mapped names, which returns the canonical string for a mapped name and decodes anything else as before. One instance per fill-loop level is shared by every row and thread. Same-sitting before/after, native, EF10, Release, profile loop (10 s per row): WholeEntityToList 10k rows 5430 KB -> 3476 KB (-36%), median 16.1 -> 15.1 ms (driver floor 3132 KB) WhereToList 10k rows 2753 KB -> 1776 KB (-35%), median 9.80 -> 9.57 ms WholeEntityTracked 10k rows 11518 KB -> 9564 KB (-17%) WideWholeEntity (400 props) 106.8 MB -> 28.5 MB (-73%), median 432 -> 389 ms (driver floor 65.3 MB) NestedWholeEntity 70.1 MB -> 56.1 MB (-20%), median 166 -> 162 ms ComplexQuery 11.5 MB -> 9.2 MB (-20%), median 50.7 -> 47.8 ms Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…n receive it A no-tracking Include fix-up called INavigationBase.SetIsLoadedWhenNoTracking for the navigation (and its inverse) on every entity it wired. EF's implementation walks the declaring type's hierarchy with several LINQ iterators to find an ILazyLoader service property, which almost no entity has, so it was an allocation-only no-op per include per row. Whether the call can do anything is fixed by the model, so it is now decided once when the fix-up is built and the call is skipped when no ILazyLoader exists in the hierarchy. An entity that does have one is still notified (new functional tests, mutation-checked: forcing the guard to false fails both lazy-loader tests). Same-sitting before/after, native, EF10, Release, profile loop (median): NestedWholeEntity 56.1 MB -> 23.7 MB (-58%), 165 -> 127 ms (-23%); driver floor 38.8 MB / 156 ms ComplexQuery 9.2 MB -> 3.8 MB (-59%), 47.4 -> 43.4 ms (-8%); driver floor 6.0 MB / 46.5 ms ReferenceInclude 10k 7.46 MB -> 3.86 MB (-48%), 120.9 -> 116.9 ms (server-bound) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ofile-harness options Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…et instead of $unwind
A native reference Include/join emitted `$lookup` then `$unwind`. On MongoDB 8.x the server
folds the `$unwind` into the `$lookup` and can no longer run it in its slot-based engine,
which costs 2-6x server time for the same documents (measured on 8.3 and 8.0; no difference
on 7.0). When the lookup matches the target's whole `_id` it returns at most one document, so
the unwind is just "take the first element": an inner join becomes
`{ $match: { <as>.0: { $exists: true } } }, { $set: { <as>: { $arrayElemAt: ["$<as>", 0] } } }`
and a left-outer one only the `$set` (an empty array gives a missing field, as `$unwind` with
preserveNullAndEmptyArrays does). Any other key (a non-`_id` foreign field, a composite key, a
navigation declared on an owned type, a dotted alias) keeps `$unwind`, as do collection joins
and every nested sub-pipeline. The driver-LINQ fallback is unchanged.
Gate covered by unit tests (mutation-checked: dropping the `_id` test fails the
non-`_id` test). 278 top-level baselines in the Northwind specification tests and nine
functional assertions were updated mechanically; no result changed.
Same-sitting before/after, native, EF10, Release, profile loop (median), server 8.3:
ReferenceInclude 10k rows 120.0 ms -> 67.4 ms (-44%), 3.86 MB; hand-written driver floor 56.3 ms / 7.33 MB
ReferenceIncludeAsync 10k 119.9 ms -> 66.3 ms (-45%)
ComplexQuery 44.1 ms -> 39.7 ms (-10%)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ings Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tten driver runs a reference, not a floor Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… a nullable date A GroupBy key or $min/$max/$avg accumulator over a date operator of a nullable date read the null as 0 where EF throws. MayBeNullBehindNonNullableType now counts date operators, so those shapes fall back to driver LINQ. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nder NativeOnly They now decline native by design, so NativeOnly (no fallback) asserts the translation failure. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rsion7 out of EF's query parameters EF's default evaluatable-expression filter only protects Random.Next and Guid.NewGuid, so the other calls were parameterized before the NonDeterministicCalls guard saw them and every row shared one value. A Mongo filter keeps every call IsNonDeterministic recognizes in the tree, so the guard throws in every query mode. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…sion7 in the non-deterministic call breaking change Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…a converted or represented property A function of such a property ran against the stored value (Name.Length over a prefixing converter, Num % 2 over a doubling one). The driver-LINQ bridge now refuses it in filters, projections and group keys, and native string search and case mapping decline any converter. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…more shapes, document the gaps Review follow-up: a string function or Length over x.P.ToString() of a property stored as that very string is no longer refused; tests cover the numeric casting allow-list and the ToString exemption; the breaking-change entry and Query AGENTS.md note the narrowing, nullable and complex-type limits. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A conditional tests its condition by truthiness, and a bool stored through a
converter ("1"/"0") is always truthy, so the first branch was always taken.
The native renderer declines such a test and the driver-LINQ bridge refuses it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rently A default-stored list tested against a converted property (Scores.Contains(x.Num)) compared the stored forms. Two stored operands that are not stored alike, and do not both hold the CLR value itself, are now refused by the driver-LINQ bridge. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…est; correct the breaking-change wording Review follow-up: (x.NFlag ?? false) ? a : b yields the stored value of a converted bool? and was not refused. The EF-466 and EF-467 breaking-change entries now state the over-refusals and the converter-instance caveat. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ent under a retained Select When Distinct or a set operation keeps the Select pushed down, the driver returns documents keyed by member name. The mixed reader looked an owned reference up under its navigation name and a scalar under its property name, so both read null. They now read under the member name. A server-evaluated case mapping there answered empty string for a null source and now throws. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… and to members the Select assigns Review follow-up: a nested owned navigation sharing a name with a root one is no longer redirected, a scalar is read by alias only when a Select assigns that member (an outer rename over a stripped Select keeps the natural name), and a Union of identical projections is not ambiguous. The breaking-change entry now says the case-mapping throw covers non-null rows and its limits. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
HasOrderPreservingStorage() (and HaveOrderPreservingStorage() for a type) declares that a property's stored form orders like its .NET value, so relational comparisons against a value, OrderBy/ThenBy and Min/Max over the bare property run on the server instead of throwing. EF's DateTimeToTicks, TimeSpanToTicks and BoolToZeroOne converters are admitted without a declaration. The declaration is an ordering claim only: Sum/Average, functions of the property and any key that mixes the property with another value (arithmetic, ??, ?:, a type-changing conversion, a comparison against another property) still throw. The refusal message names the opt-out. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A declaration vouches for the property on its own, so refuse it where several sources feed one value (Concat/Union of the property with another property or a constant, or with a differently stored declared property), and where a Sum/Average adds up another aggregate's result (GroupBy(..).Select(g => g.Max(x => x.P)).Sum()). Refusal messages only name the opt-out when it could help. A lambda parameter named like an EF query parameter is no longer taken for a value. The convention builder gains CanSetOrderPreservingStorage and fromDataAnnotation, removal honours the configuration source, and HaveOrderPreservingStorage takes the bool. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Native MQL translation for the MongoDB EF Core provider, the working branch for epic EF-322.
Draft. This supersedes #324 (closed). #324's history is unchanged on the
EF-322-Native-LINQ-rebasedbranch of the same fork as a point-in-time record. Ongoing work happens on this branch (EF-322c).What this changes
The provider gains a second query path. For the supported slice it builds the aggregation pipeline itself instead of handing a LINQ expression to the driver's LINQ v3 provider. Anything outside that slice falls back to the driver automatically. Native is the default;
UseQueryMode(MongoQueryMode.DriverLinq)restores the previous path.Under the provider's versioning rubric, changing the default path is not in itself a break: unsupported shapes fall back and results are unchanged. Cases where an observable result or behaviour did change are recorded in
BREAKING-CHANGES.md.Shape of the branch
mainatdec7e26f.EF-322: .... The first isworking's own squash commit (EF-254, EF-295, EF-221, EF-218, EF-253, EF-359, EF-358, EF-376, EF-379, EF-380 and upstream PRs EF-359: Fix Count/LongCount over embedded collections in projections #339, EF-376: Fix sibling ThenIncludes through the same target type collapsing into one $lookup #350, EF-379: Fix transitive ThenInclude hop misrouted to root-level decoy navigation #353, EF-380: Fix ThenInclude whose key selector reaches through an owned navigation #355, EF-221: Fix Equals() with mismatched numeric types throwing InvalidCastException #357, EF-218: Fix DateTimeOffset member projection without a driver change #360, EF-253: Drop redundant re-ordering by the same property in an OrderBy/ThenBy chain #366, EF-254: Fix Take(0) throwing ArgumentOutOfRangeException #367, EF-295: Fix many-to-many join entity losing composite key to duplicate shadow property shadow property #368), which is not yet onmain. The other 57 are the native work.Also: EF-...).main: +153,535 / -6,383.docs/superpowersholds the specs and implementation plans.Testing
Full solution run (UnitTests, SpecificationTests, FunctionalTests), against per-process Atlas-local containers. The tree was byte-identical when the squash was made, and the run below was on the pre-squash tip:
Reviewing
Commits are themed, so reviewing commit by commit is workable.