Conversation
# Pull Request Template for ScriptDom ## Description `JSON_ARRAYAGG` supports windowed aggregate usage via the `OVER (PARTITION BY ...)` clause in SQL Server, but ScriptDOM was missing support for parsing and generating it. This PR adds OVER clause support for `JSON_ARRAYAGG` in the TSql170 parser and script generator. **SQL syntax now supported:** ```sql SELECT JSON_ARRAYAGG(name) OVER (PARTITION BY dept) FROM employees; SELECT JSON_ARRAYAGG(name ABSENT ON NULL) OVER (PARTITION BY dept) FROM employees; SELECT JSON_ARRAYAGG(name NULL ON NULL) OVER (PARTITION BY dept) FROM employees; SELECT JSON_ARRAYAGG(name ORDER BY name NULL ON NULL RETURNING JSON) OVER (PARTITION BY dept) FROM employees; ``` ## Code Change - [x] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [x] Code changes are accompanied by appropriate unit tests - [ ] Identified and included SMEs needed to review code changes - [X] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code ## Testing - [X] Follow the [steps] All 22 TSql170Syntax tests pass across both net8.0 and net472 targets, covering all compat levels (80-170). ## Documentation - [ ] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file ## Additional Information No AST changes were needed — FunctionCall.OverClause already exists in Ast.xml. The grammar uses overClauseNoOrderBy (not overClause) since, like regular aggregates (e.g., SUM() OVER (...)), the ORDER BY within the OVER clause is not applicable for JSON_ARRAYAGG as a windowed aggregate. TSql80 parser produces 10 errors (vs 9 for other old parsers) because TSql80 does not support the OVER keyword at all. Related work items: #5003355
…ddAndUpdateTokenInfo <!-- COPILOT_AI_GENERATED_START --> **🤖 AI-Generated Pull Request 🤖** This pull request was generated by the VS Perf Rel AI Agent. Please review this AI-generated PR with extra care! For more information, visit our [wiki](https://dev.azure.com/devdiv/DevDiv/_wiki/wikis/DevDiv.wiki/49206/). Please share feedback with [TIP Insights](mailto:tipinsights@microsoft.com) --- ## Problem High memory allocations were detected in the `TSql80ParserBaseInternal.AddAndUpdateTokenInfo` method, specifically in the overload that processes collections. The method uses a `foreach` loop to iterate over an `IList<TFragmentType>` collection, which causes the allocation of a boxed `List.Enumerator` object on every invocation. Since this is a hot path in the SQL parser (called frequently during parsing operations), these repeated allocations contribute significantly to GC pressure. Performance impact: This issue appears in 0.28% of high allocation traces and allocates 3.0 MB/sec at both the 50th and 90th percentiles. ## Solution Replaced the `foreach` loop with an index-based `for` loop in the `AddAndUpdateTokenInfo` method. The new implementation iterates using an integer index from 0 to `otherCollection.Count`, accessing each element via the indexer. This eliminates the enumerator allocation while preserving identical functionality and behavior. The change is made in `/SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs` at lines 292-299. **Before:** ```csharp foreach (TFragmentType item in otherCollection) { AddAndUpdateTokenInfo(node, collection, item); } ``` **After:** ```csharp for (int i = 0; i < otherCollection.Count; i++) { TFragmentType item = otherCollection[i]; AddAndUpdateTokenInfo(node, collection, item); } ``` ## Benefits - Eliminates boxing allocation of `List.Enumerator` on hot path - Reduces GC pressure and improves parser performance - Maintains identical behavior and semantics - Minimal code change with no API modifications [Best practices wiki](https://dev.azure.com/devdiv/DevDiv/_wiki/wikis/DevDiv.wiki/24181/Garbage-collection-(GC)) | [See related failure in PRISM](https://prism.vsdata.io/failure/?eventType=allocation&failureType=dualdirection&failureHash=df809ff3-2831-8a3a-b02e-ae2c306b6d5f) Fixes: [[PerfRelAgent][Allocation][{Environment}] Failure ID df809ff3-2831-8a3a-b02e-ae2c306b6d5f](https://msdata.visualstudio.com/web/wi.aspx?pcguid=8b119ea1-2e2a-4839-8db7-8c9e8d50f6fa&id=4996528) <p><small class=\"secondary-text\">AI-generated content may be incorrect</small></p> Co-authored-by: Naresh Joshi <nareshjo@microsoft.com> <!-- COPILOT_AI_GENERATED_END --> <!-- GitOpsUserAgent=GitOps.Apps.Server.copilotswe --> Related work items: #4996528
…ECTOR_SEARCH TOP_N parameter
<!-- COPILOT_AI_GENERATED_START -->
## Summary
This PR enables the VECTOR_SEARCH table-valued function to accept 3-part and 4-part column identifiers in the TOP_N parameter, allowing queries like:
```sql
SELECT qt.qid, src.id, ann.distance
FROM QueryTable qt
CROSS APPLY
VECTOR_SEARCH(
TABLE = graphnode AS src,
COLUMN = embedding,
SIMILAR_TO = qt.qembedding,
METRIC = 'euclidean',
TOP_N = dbo.qt.top_n -- 3-part identifier now supported
) AS ann
```
Previously, the parser only accepted up to 2-part identifiers (e.g., `qt.top_n`) in the TOP_N parameter, which prevented queries using schema-qualified column references in CROSS APPLY scenarios.
## Changes Made
### Grammar Rule Update
Modified the `vectorSearchColumnReferenceExpression` rule in `SqlScriptDom/Parser/TSql/TSql170.g` to allow up to 4-part identifiers instead of the previous 2-part limit. This change enables the TOP_N parameter to accept:
- Simple identifiers: `@variable` or `columnName`
- 2-part identifiers: `table.column`
- 3-part identifiers: `schema.table.column` (the primary use case from the issue)
- 4-part identifiers: `database.schema.table.column`
The change was minimal—updating a single parameter from `multiPartIdentifier[2]` to `multiPartIdentifier[4]` in line 33971 of TSql170.g. This aligns with standard SQL Server column reference behavior in other contexts.
### Test Coverage
Added comprehensive test coverage in `Test/SqlDom/TestScripts/VectorSearchCrossApplyTests170.sql` with a test case that validates the exact query pattern from the issue. The test includes:
- A DECLARE statement setting up a VECTOR variable
- A SELECT query with CROSS APPLY using VECTOR_SEARCH
- The TOP_N parameter specified as a 3-part identifier (`dbo.qt.top_n`)
The corresponding baseline file `Test/SqlDom/Baselines170/VectorSearchCrossApplyTests170.sql` captures the expected formatted output from the parser.
Updated `Test/SqlDom/Only170SyntaxTests.cs` to register the new test case with appropriate error counts for older parser versions (TSql80 through TSql160), which still use the 2-part identifier limit and correctly report parse errors for this syntax.
## Impact
This change only affects the TSql170 (SQL Server 2025) parser and is backward compatible. The AST definition already used `ScalarExpression` for the TopN member, so no AST changes were required. Older parser versions continue to enforce the 2-part identifier limit and generate expected parse errors for 3-part identifiers.
The fix enables real-world scenarios where VECTOR_SEARCH is used in CROSS APPLY with outer references that require schema qualification, which is common in production databases with explicit schema naming conventions.
Fixes: #4843961
<p><small class=\"secondary-text\">AI-generated content may be incorrect</small></p>
reference: https://msdata.visualstudio.com/Database%20Systems/_git/DsMainDev/commit/15b0ead69fc5a8ba9eb1d4d84735c4e9d6ab5718?_a=co...
…t Subqueries
<!-- COPILOT_AI_GENERATED_START -->
### Summary
The VECTOR_SEARCH function in the TSql170 parser was incorrectly allowing subqueries in the SIMILAR_TO parameter. This fix adds validation to properly reject subqueries and throw an appropriate parse error, aligning with SQL Server 2025 implementation requirements.
### Problem
The SIMILAR_TO parameter was accepting any scalar expression, including subqueries wrapped in parentheses. This allowed invalid syntax like:
```sql
SELECT * FROM VECTOR_SEARCH(
TABLE = graphnode,
COLUMN = embedding,
SIMILAR_TO = (SELECT TOP 1 embedding FROM GTQuery), -- Should error but didn't
METRIC = 'euclidean',
TOP_N = 20
) AS ann
```
This syntax should be rejected because subqueries are not supported in the SIMILAR_TO parameter, similar to how they are restricted in other contexts.
### Solution
Added validation logic in the `vectorSearchTableReference` grammar rule (`SqlScriptDom/Parser/TSql/TSql170.g`) that checks if the SIMILAR_TO expression is a `ScalarSubquery` type. When detected, the parser now throws a SQL46098 error with the message "Subqueries are not allowed in this context. Only scalar expressions are allowed."
The validation follows the same pattern used elsewhere in the grammar for similar restrictions and is placed immediately after matching the SIMILAR_TO identifier, before assigning the expression to the result object.
### Testing
Added a comprehensive error test case in `Test/SqlDom/ParserErrorsTests.cs` within the existing `VectorSearchErrorTest170` method. The test verifies that:
- Subqueries in the SIMILAR_TO parameter are properly rejected
- The correct error code (SQL46098) is generated
- The error position is accurately reported
Valid syntax continues to work correctly, including:
- Variables: `SIMILAR_TO = @qv`
- Column references: `SIMILAR_TO = outerref.vector_col`
- Other scalar expressions that are not subqueries
### Impact
- **All 558 existing tests pass** with no regressions
- **Minimal change**: Only 12 lines added across 2 files
- **Consistent behavior**: Uses the same error code and message pattern as other subquery restrictions in the parser
- **Backward compatible**: Only rejects previously invalid syntax that should not have been allowed
Fixes: #4844065
<p><small class=\"secondary-text\">AI-generated content may be incorrect</small></p>
Co-authored-by: Leila Lali <llali@microsoft.com>
<!-- COPILOT_AI_GENERATED_END -->
<!-- GitOpsUserAgent=GitOps.Apps.Server.copilotswe -->
Related work items: #4844065
llali
requested review from
chlafreniere,
dzsquared,
kburtram and
ssreerama
as code owners
# Pull Request Template for ScriptDom ## Description Please provide a detailed description, include the link to the design specification or SQL feature document for the new TSQL syntaxes. Make sure to add links to the Github or DevDiv issue Before submitting your pull request, please ensure you have completed the following: ## Code Change - [ ] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [ ] Code changes are accompanied by appropriate unit tests - [ ] Identified and included SMEs needed to review code changes - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code ## Testing - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=to-extend-the-tests-do-the-following%3A) here to add new tests for your feature ## Documentation - [ ] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file ## Additional Information Please provide any additional information that might be helpful for the reviewers Adding release notes for 170.168.0
ssreerama
approved these changes
Feb 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please provide a detailed description. Be as descriptive as possible - include information about what is being changed,
why it's being changed, and any links to relevant issues. If this is closing an existing issue use one of the issue linking keywords to link the issue to this PR and have it automatically close when completed.
In addition, go through the checklist below and check each item as you validate it is either handled or not applicable to this change.
Code Changes