Repository navigation
Run a top-K aggregation and write the ranked rows (Step 3 of #386) - #494
Merged
Merged
Conversation
1 task done
zipdoki
force-pushed
the
feat/topk-aggregate
branch
2 times, most recently
from
August 20, 2026 02:14
37e1c4b to
7af6396
Compare
6 tasks done
…t's name A declaration may name a date bucket for the value it yields rather than for the field it reads, and a `ranges` bound refers to it by that name. `refreshAt` looked the event's value up under the bucket's name, found nothing whenever the two names differ, and fell back to counting from now — so an order that left the window a year ago was scheduled to expire a year from ingest instead. Nothing reported it. The aggregation answered SUCCESS and the message carried the wrong due time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`refreshQueue` landed on the declaration in #493 with nothing reading it. The enqueue still went to the constant, so naming a queue had no effect. Existing tests here name no queue and assert the shared one, which pins the default; the new test names one and also asserts nothing reaches the shared queue.
zipdoki
force-pushed
the
feat/topk-aggregate
branch
from
August 27, 2026 08:08
16c8202 to
51a1402
Compare
`AggregationResult` carries no field naming the item it came from, so a caller pairs a result back to its request by position. The dispatcher merged handler output as it completed, which left the same batch coming back ordered by whichever writes finished first — a three-item batch returns fully reversed when the first item is the slowest. `Graph.mutate` and `AbstractLabel.mutate` already made this choice on the mutation path. Concurrency is unchanged: `flatMapSequential` and `mergeSequential` subscribe eagerly under the same default `maxConcurrency`, so storage sees the same request pattern. `AggregationService.aggregate` had no test of its own. The new ones pin fan-out, the empty batch, request ordering, and the fact that two handlers claiming one type collapse to the last — that last one is current behavior, pinned rather than fixed.
The branch was rebased onto the squash-merged #493, where the constant is `REFRESH_QUEUE`. The rebase carried that name into `AggregationConstants`, `Group` and `EdgeSchemaSerializationTest` but left `TopkAggregationHandlerTest` and `MetadataAggControllerE2ETest` on `REFRESH_TABLE`, so the branch did not compile. Point them at the merged name.
`AggregationFunctions.kt` and `TopkAggregationHandler` both declared `parseFqn`, added in the same commit. The handler's own private copy shadows the top-level one at the only call site, so nothing reached `AggregationFunctions` and the two spellings of its failure message could drift apart unnoticed. Extract a shared one again when a second handler needs it.
`processTopk` skips the refresh on three conditions and only two were covered. The third — a top-K that sets `refreshAfterMillis` but bounds its range with fixed dates — had no test, so nothing said what a non-sliding window is supposed to do. Drop the guard and this new test is the only one that fails, which is what it is for. Also drops `returns a result for every item`, added earlier today: the ordering test asserts the same three items with `shouldContainExactly`, so it already covers membership.
Contributor
Author
|
Optimistic Merge |
zipdoki
added a commit
that referenced
this pull request
Aug 27, 2026
`AggregationSweepResult` names no item it came from, so a caller pairs a result to its request by position — the same contract `aggregate` has. `sweep` merged handler output as it completed, so a three-item batch came back fully reversed when the first item was the slowest. Same fix as #494 made on the aggregate path, and concurrency is unchanged: `flatMapSequential` subscribes eagerly under the same default `maxConcurrency`.
zipdoki
added a commit
that referenced
this pull request
Aug 27, 2026
* Recompute a ranking from a refresh message * Remove the expire payloads the refresh queue replaced The sweep design moved from an _expire table scanned by expiredAt to a refresh queue, and these three payloads never got wired to anything. AggregationExpireItemRequest has no references at all, and AggregationExpireResult is only read by AggregationsExpireResponse.from, which nothing calls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Sweep results come back in request order `AggregationSweepResult` names no item it came from, so a caller pairs a result to its request by position — the same contract `aggregate` has. `sweep` merged handler output as it completed, so a three-item batch came back fully reversed when the first item was the slowest. Same fix as #494 made on the aggregate path, and concurrency is unchanged: `flatMapSequential` subscribes eagerly under the same default `maxConcurrency`. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
zipdoki
added a commit
that referenced
this pull request
Aug 31, 2026
Steps 3 and 4 landed on main as squash merges (#494, #496) carrying review changes this branch never saw: `AggregationConstants.Topk.REFRESH_TABLE` is now `REFRESH_QUEUE`, the aggregate and sweep dispatchers emit in request order via `flatMapSequential`, and both test classes gained cases for that ordering and for a window that never slides. Took main's side wherever the two overlapped, since this branch's copies are the pre-review versions of the same work. Kept the `AggregationQueryService` bean in `GraphConfiguration`, which is Step 5 and exists only here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A group can declare a top-K, but nothing computes one. The declaration sits in metadata and no ranked row is ever written. This runs the aggregation.
TopkAggregationHandlerscans the source group with the declaration'srangesinterpolated against the event, sumsmetricper(entity, dimension key), and writes rank rows keyed byrankSource(...)under themetric_descindex. When a declaration is a sliding window andrefreshAfterMillis > 0, it also schedules a refreshfor the instant the bucket leaves the window. Consuming that refresh message is the next PR in the stack.
POST /aggregations/v1/aggregatedrives it, taking a batch of edges:{ "items": [ { "database": "graph", "table": "likes", "edge": { "source": "...", "target": "...", "...": "..." } } ] }{ "items": [ { "database": "graph", "table": "likes", "source": "...", "target": "...", "status": "OK", "error": null } ] }Status and error are reported per item, so one bad edge does not sink the rest of the batch.
Two bucket changes ride along because the window math depends on them.
Bucketnow has a test pinning how a date bucket rounds both a value and a range bound, which the refresh schedule reads directly, and its format precision is read in one place instead of being re-derived at each call site. Separately, rank keys escape|sotwo rankings can never collide on one row.
Stacked on #493 — that needs to land first.
Part of #386.
Test plan
./gradlew :core:test --tests '*BucketTest*'— date bucket rounding for values and for range bounds./gradlew :core:test --tests '*AggregationConstantsTest*'— rank key composition and|escaping./gradlew :engine:test --tests '*RankingInputsTest*'—rangesinterpolation, entity and dimension resolution./gradlew :engine:test --tests '*TopkAggregationHandlerTest*'— aggregation over the source group, rank row writes, refresh scheduling./gradlew :server:test --tests '*MetadataAggControllerE2ETest*'—POST /aggregations/v1/aggregateover HTTP./gradlew spotlessKotlinCheck build— formatting and full buildAI Assistance