feat/nested aggregations by davidkelley · Pull Request #102 · davidkelley/searchlite

davidkelley · 2026-02-10T16:39:51Z

chore: add workspace lock symlinks for release-plz
chore: temporarily force release-plz
fix: pushed up changes to release workflow
chore: release (chore: release #56)
chore(searchlite-cli): release v0.1.2 (chore(searchlite-cli): release v0.1.2 #57)
fix: update release command
fix: updated wasm build
chore(searchlite-wasm): release v0.1.2 (chore(searchlite-wasm): release v0.1.2 #58)
chore/updated examples (chore/updated examples #60)
chore: release (chore: release #59)
remove filters final (remove filters final #62)
chore: release (chore: release #61)
perf/optimize wand and http (perf/optimize wand and http #63)
chore: release (chore: release #64)
fix: stream ndjson batches with single writer (fix: stream ndjson batches with single writer #65)
fix: default search request limit and return_stored
fix: unify doc id validation across ingest and delete paths
fix(ffi): contain panics across C boundary and stabilize ffi tests
fix: address remaining limit=0 review feedback
chore/skills (chore/skills #78)
feat: vector cap env
feat: implemented docker release
chore: release (chore: release #82)
fix: make ffi/wasm search requests atomic and align return_stored defaults (fix: make ffi/wasm search requests atomic and align return_stored defaults #73)
chore: release (chore: release #83)
feat: optional write keys (feat: optional write keys #85)
feat: multi index routing (feat: multi index routing #93)
chore: release (chore: release #94)
feat: mget & search_after (feat: mget & search_after #95)
feat/mget documentation (feat/mget documentation #100)
feat: partial update APIs (feat: partial update APIs #101)
feat: implemented the capability to return nested aggregations

davidkelley · 2026-02-10T16:42:41Z

Update based on the current feat/nested-aggregations branch state:

Implemented Option A (true nested aggregations) for issue #90.

What was added

New aggregation type: nested with request shape { "type": "nested", "path": "...", "aggs": { ... } }.
New response variant: nested bucket response with doc_count and nested aggregations.
Nested collector pipeline in core aggregation engine:
- iterates nested objects per matched parent doc,
- preserves parent/child binding for deeper nested paths,
- supports nested child aggs currently via terms and further nested blocks.
Validation updates:
- validates nested paths,
- resolves child fields relative to nested scope,
- rejects unsupported child aggregation types under nested scope.
Fast-field nested object access helper for object-scoped term collection.
Segment nested metadata fix to correctly accumulate parent/object lineage for repeated nested objects.

Performance notes (Option A)

Impact is query-time only for requests that include nested aggregations.
Normal indexing/search paths are unchanged when nested aggs are not requested.
Cost scales with:
- number of matched docs,
- nested object count per matched doc,
- terms cardinality in nested fields.
Added docs guidance to keep nested arrays bounded, apply outer filters, and use sampling/flattened facet fields for hotspot cases.

Tests, docs, and benchmark updates

Added core integration tests for:
- nested object-level counts,
- outer-filter interaction,
- invalid nested path handling,
- unsupported nested-child aggregation handling,
- parent-binding for nested-in-nested,
- metadata key/value facet binding regression.
Added HTTP integration coverage for nested aggs.
Updated request schema docs and README nested aggregation examples/perf caveats.
Added nested aggregation benchmark scenario in searchlite-core/benches/aggs.rs (aggs_nested_terms_metadata).

Validation run on this branch

cargo fmt --all
cargo clippy --all --all-features --all-targets -- -D warnings
cargo build --all --all-features
cargo test --all --all-features
cargo bench -p searchlite-core

Copilot

Pull request overview

This PR adds support for nested aggregations (nested buckets with scoped sub-aggregations) across the core engine and HTTP layer, including request/response types, validation, execution logic, schema updates, docs, and benchmarks.

Changes:

Add nested aggregation request/response types and JSON schema support.
Implement nested aggregation execution (including nested-within-nested scoping) and update nested indexing/fastfield access to support object-level binding.
Add core + HTTP tests, docs, and a benchmark for nested term aggregations.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
searchlite-http/src/lib.rs	Adds an HTTP integration test covering nested aggregations end-to-end.
searchlite-core/tests/aggregations.rs	Adds core tests for nested terms, filtering interaction, validation failures, and nested-within-nested binding.
searchlite-core/src/query/aggs/mod.rs	Implements nested aggregation collector + scoped collection and fastfield reads per nested object.
searchlite-core/src/index/segment.rs	Fixes nested collection bookkeeping to support accumulating nested objects across parent objects (enables deeper nesting).
searchlite-core/src/index/fastfields.rs	Adds `nested_str_values_at` to read values for a specific nested object index.
searchlite-core/src/api/types.rs	Introduces `NestedAggregation` and `AggregationResponse::Nested`.
searchlite-core/src/api/reader.rs	Adds nested aggregation validation + scoped field resolution for nested contexts.
searchlite-core/benches/aggs.rs	Adds a benchmark for nested terms aggregation over key/value metadata pairs.
search-request.schema.json	Adds JSON schema definition for `nested` aggregation blocks.
README.md	Documents nested aggregation semantics and provides an example.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

searchlite-core/src/query/aggs/mod.rs

searchlite-core/src/api/reader.rs

searchlite-core/src/query/aggs/mod.rs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 40541f1890

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

searchlite-core/src/index/fastfields.rs

searchlite-core/src/api/reader.rs

- stabilize nested sampling hash with fixed-width object index - validate nested agg paths against declared nested schema tree - share scoped path helpers between validation and aggregation execution - guard nested fast-field object lookups to the current doc range - remove nested finalize clone and add regression coverage

davidkelley · 2026-02-10T16:59:16Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 208ba0d6c5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

searchlite-core/src/query/aggs/mod.rs

- reject descendant terms fields inside nested scopes - require nested-in-nested paths to be direct children of current scope - add regression tests for both validation errors

davidkelley · 2026-02-26T09:53:41Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7b896b6141

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

searchlite-core/src/api/reader.rs

- block nested keyword paths at root scope for terms/collapse - block nested numeric paths at root scope for numeric aggs - add regression tests for root terms, stats, and collapse

davidkelley · 2026-02-26T10:16:37Z

@codex review

davidkelley · 2026-02-26T10:18:45Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 74c9df0179

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

searchlite-core/src/api/reader.rs

- resolve nested paths via schema tree instead of dot-splitting - validate direct-child nested scope checks against schema structure - add regression coverage for dotted nested leaf and object names

davidkelley · 2026-02-26T10:37:06Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a32f3a27d5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

searchlite-core/src/query/aggs/mod.rs

davidkelley · 2026-02-26T11:10:31Z

@codex review

chatgpt-codex-connector · 2026-02-26T11:27:50Z

Codex Review: Didn't find any major issues. Delightful!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

feat: implemented the capability to return nested aggregations

40541f1

Copilot AI review requested due to automatic review settings February 10, 2026 16:39

Copilot started reviewing on behalf of davidkelley February 10, 2026 16:40 View session

davidkelley linked an issue Feb 10, 2026 that may be closed by this pull request

#4: Nested aggregation facets #90

Closed

Copilot AI reviewed Feb 10, 2026

View reviewed changes

searchlite-core/src/query/aggs/mod.rs Outdated Show resolved Hide resolved

searchlite-core/src/api/reader.rs Outdated Show resolved Hide resolved

searchlite-core/src/api/reader.rs Outdated Show resolved Hide resolved

searchlite-core/src/query/aggs/mod.rs Outdated Show resolved Hide resolved

chatgpt-codex-connector bot reviewed Feb 10, 2026

View reviewed changes

searchlite-core/src/index/fastfields.rs Outdated Show resolved Hide resolved

searchlite-core/src/api/reader.rs Outdated Show resolved Hide resolved

chatgpt-codex-connector bot reviewed Feb 10, 2026

View reviewed changes

searchlite-core/src/query/aggs/mod.rs Show resolved Hide resolved

searchlite-core/src/query/aggs/mod.rs Show resolved Hide resolved

fix: enforce direct-child nested aggregation scopes

7b896b6

- reject descendant terms fields inside nested scopes - require nested-in-nested paths to be direct children of current scope - add regression tests for both validation errors

chatgpt-codex-connector bot reviewed Feb 26, 2026

View reviewed changes

searchlite-core/src/api/reader.rs Show resolved Hide resolved

searchlite-core/src/api/reader.rs Show resolved Hide resolved

fix: reject root-scope nested fields in fast-field validation

74c9df0

- block nested keyword paths at root scope for terms/collapse - block nested numeric paths at root scope for numeric aggs - add regression tests for root terms, stats, and collapse

chatgpt-codex-connector bot reviewed Feb 26, 2026

View reviewed changes

searchlite-core/src/api/reader.rs Show resolved Hide resolved

searchlite-core/src/api/reader.rs Outdated Show resolved Hide resolved

fix: handle dotted nested names in agg scope validation

a32f3a2

- resolve nested paths via schema tree instead of dot-splitting - validate direct-child nested scope checks against schema structure - add regression coverage for dotted nested leaf and object names

chatgpt-codex-connector bot reviewed Feb 26, 2026

View reviewed changes

searchlite-core/src/query/aggs/mod.rs Show resolved Hide resolved

davidkelley added 2 commits February 26, 2026 10:54

chore(skills): add feature-hardening skill

89a3c6d

fix(aggs): skip null slots in nested doc counts

ebaa84e

davidkelley merged commit f7d978b into main Feb 26, 2026
6 checks passed

davidkelley deleted the feat/nested-aggregations branch February 26, 2026 11:29

This was referenced Feb 26, 2026

chore: release #99

Open

feat: Fuzzy/cross-field search + exact total hits #103

Merged

feat: auto commit refresh indexes #104

Merged

fix/mget doc id response #105

Merged

Conversation

davidkelley commented Feb 10, 2026

Uh oh!

davidkelley commented Feb 10, 2026

What was added

Performance notes (Option A)

Tests, docs, and benchmark updates

Validation run on this branch

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

davidkelley commented Feb 10, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

davidkelley commented Feb 26, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

davidkelley commented Feb 26, 2026

Uh oh!

davidkelley commented Feb 26, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

davidkelley commented Feb 26, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

davidkelley commented Feb 26, 2026

Uh oh!

chatgpt-codex-connector bot commented Feb 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants