perf(manager): preaggregate cached CVE subquery - #2438
Open
pindo696 wants to merge 1 commit into
Open
Conversation
Postgres planner misestimated row count due to workspaces selectivity, choosing a nested loop join instead of hash join on the cve_account_granular_cache table. This led to multiple rows to be compared and filtered out. By moving GROUP BY into the inner subquery and isolating group filtering into a separate IN subquery, we reduce the number of rows joined, which leads to more performatiove query and planner can (hopefully) better estimate the number of rows as well.
Reviewer's GuideRefactors the vulnerabilities manager CVE query to support a pre-aggregated cached path that aggregates per‑CVE metrics and RHEL versions inside the granular cache subquery, adjusts joins and filters accordingly, and preserves the existing aggregation path for non‑cached queries. Entity relationship diagram for pre-aggregated granular cache subqueryerDiagram
CveAccountGranularCache {
int cve_id
int group_set_id
int operating_system_id
int systems_affected_rpmdnf
int systems_affected_edge
int systems_status_divergent
}
SystemGroupSet {
int id
jsonb groups
}
OperatingSystem {
int id
text rhel_version
}
CveAccountGranularCache ||--|| SystemGroupSet : group_set_id
CveAccountGranularCache }o--|| OperatingSystem : operating_system_id
%% group_set_subquery selects SystemGroupSet.id filtered by groups
%% _granular_cached_count_subquery aggregates per CveAccountGranularCache.cve_id and uses IN on group_set_id
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
Author
|
@sourcery-ai review |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- Consider removing the commented-out
group_set_subquery = group_set_subquery.offset(0)line or adding a brief explanation if it’s intentionally left as a placeholder, to avoid confusion about dead/debug code. - The special handling of
CVE_RHEL_VERSION(being filtered inside the cached subquery but outside for non-cached) is now spread across multiple places; it may be worth centralizing this logic or clearly documenting it to reduce the risk of the two paths diverging in future changes.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- Consider removing the commented-out `group_set_subquery = group_set_subquery.offset(0)` line or adding a brief explanation if it’s intentionally left as a placeholder, to avoid confusion about dead/debug code.
- The special handling of `CVE_RHEL_VERSION` (being filtered inside the cached subquery but outside for non-cached) is now spread across multiple places; it may be worth centralizing this logic or clearly documenting it to reduce the risk of the two paths diverging in future changes.
## Individual Comments
### Comment 1
<location path="manager/vulnerabilities_handler.py" line_range="183" />
<code_context>
- fn.COALESCE(BusinessRisk.name, DEFAULT_BUSINESS_RISK).alias("business_risk"),
- fn.COALESCE(CveAccountData.status_id, 0).alias("status_id"),
- CveAccountData.status_text.alias("status_text")))
+ def _full_query(rh_account_id, join_type, count_subquery, pre_aggregated=False):
+ if pre_aggregated:
+ systems_affected = (fn.COALESCE(count_subquery.c.systems_affected_rpmdnf_, 0) +
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Align advisory-availability semantics between pre-aggregated and non-pre-aggregated modes.
In the non‑pre‑aggregated path, `advisory_available` uses `Bool_Or(...)` over joined rows, but in the pre‑aggregated path you `COALESCE(count_subquery.c.advisory_available_, ...)`. Since `_granular_cached_count_subquery` now does `Bool_Or(advisory_available_column)`, this should usually match, but the behavior can diverge if the subquery returns NULL or no rows (e.g., after `WHERE` filtering). Consider making both paths structurally consistent and explicitly aligning how "no rows" vs "all false" are handled so cached and uncached results stay in sync.
Suggested implementation:
```python
if pre_aggregated:
systems_affected = (fn.COALESCE(count_subquery.c.systems_affected_rpmdnf_, 0) +
fn.COALESCE(count_subquery.c.systems_affected_edge_, 0)).alias("systems_affected")
systems_status_divergent = fn.COALESCE(count_subquery.c.systems_status_divergent_, 0).alias("systems_status_divergent")
advisory_available = fn.Bool_Or(
fn.COALESCE(
count_subquery.c.advisory_available_,
fn.COALESCE(CveMetadata.advisories_list, '[]') != SQL("'[]'")
)
).alias("advisory_available")
rhel_versions = fn.COALESCE(count_subquery.c.rhel_versions_, "{}").alias("rhel_versions")
else:
systems_affected = fn.Sum((fn.COALESCE(count_subquery.c.systems_affected_rpmdnf_, 0) +
fn.COALESCE(count_subquery.c.systems_affected_edge_, 0))).alias("systems_affected")
systems_status_divergent = fn.Sum(fn.COALESCE(count_subquery.c.systems_status_divergent_, 0)).alias("systems_status_divergent")
advisory_available = fn.Bool_Or(
fn.COALESCE(
count_subquery.c.advisory_available_,
fn.COALESCE(CveMetadata.advisories_list, '[]') != SQL("'[]'")
)
).alias("advisory_available")
```
1. This change assumes `_granular_cached_count_subquery` is already aggregating `advisory_available_` with `Bool_Or` so that using `Bool_Or` again in the pre-aggregated branch is idempotent (single-row aggregation). If that assumption is incorrect and `advisory_available_` is no longer aggregated in the cached subquery, you may want to keep the pre-aggregated path as a plain `COALESCE` without `Bool_Or`.
2. If you need stricter handling of the “no rows” case (e.g., explicitly falling back to `CveMetadata.advisories_list` when the join produces no count-subquery rows), you’ll need to verify the join type (`join_type`) and possibly adjust the join or add additional `COALESCE` around the outer `Bool_Or` to enforce a default value.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| fn.COALESCE(BusinessRisk.name, DEFAULT_BUSINESS_RISK).alias("business_risk"), | ||
| fn.COALESCE(CveAccountData.status_id, 0).alias("status_id"), | ||
| CveAccountData.status_text.alias("status_text"))) | ||
| def _full_query(rh_account_id, join_type, count_subquery, pre_aggregated=False): |
There was a problem hiding this comment.
suggestion (bug_risk): Align advisory-availability semantics between pre-aggregated and non-pre-aggregated modes.
In the non‑pre‑aggregated path, advisory_available uses Bool_Or(...) over joined rows, but in the pre‑aggregated path you COALESCE(count_subquery.c.advisory_available_, ...). Since _granular_cached_count_subquery now does Bool_Or(advisory_available_column), this should usually match, but the behavior can diverge if the subquery returns NULL or no rows (e.g., after WHERE filtering). Consider making both paths structurally consistent and explicitly aligning how "no rows" vs "all false" are handled so cached and uncached results stay in sync.
Suggested implementation:
if pre_aggregated:
systems_affected = (fn.COALESCE(count_subquery.c.systems_affected_rpmdnf_, 0) +
fn.COALESCE(count_subquery.c.systems_affected_edge_, 0)).alias("systems_affected")
systems_status_divergent = fn.COALESCE(count_subquery.c.systems_status_divergent_, 0).alias("systems_status_divergent")
advisory_available = fn.Bool_Or(
fn.COALESCE(
count_subquery.c.advisory_available_,
fn.COALESCE(CveMetadata.advisories_list, '[]') != SQL("'[]'")
)
).alias("advisory_available")
rhel_versions = fn.COALESCE(count_subquery.c.rhel_versions_, "{}").alias("rhel_versions")
else:
systems_affected = fn.Sum((fn.COALESCE(count_subquery.c.systems_affected_rpmdnf_, 0) +
fn.COALESCE(count_subquery.c.systems_affected_edge_, 0))).alias("systems_affected")
systems_status_divergent = fn.Sum(fn.COALESCE(count_subquery.c.systems_status_divergent_, 0)).alias("systems_status_divergent")
advisory_available = fn.Bool_Or(
fn.COALESCE(
count_subquery.c.advisory_available_,
fn.COALESCE(CveMetadata.advisories_list, '[]') != SQL("'[]'")
)
).alias("advisory_available")- This change assumes
_granular_cached_count_subqueryis already aggregatingadvisory_available_withBool_Orso that usingBool_Oragain in the pre-aggregated branch is idempotent (single-row aggregation). If that assumption is incorrect andadvisory_available_is no longer aggregated in the cached subquery, you may want to keep the pre-aggregated path as a plainCOALESCEwithoutBool_Or. - If you need stricter handling of the “no rows” case (e.g., explicitly falling back to
CveMetadata.advisories_listwhen the join produces no count-subquery rows), you’ll need to verify the join type (join_type) and possibly adjust the join or add additionalCOALESCEaround the outerBool_Orto enforce a default value.
pindo696
marked this pull request as ready for review
August 4, 2026 09:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Postgres planner misestimated row count due to workspaces selectivity, choosing a nested loop join instead of hash join on the cve_account_granular_cache table. This led to multiple rows to be compared and filtered out. By moving GROUP BY into the inner subquery and isolating group filtering into a separate IN subquery, we reduce the number of rows joined, which leads to more performatiove query and planner can (hopefully) better estimate the number of rows as well.
Secure Coding Practices Checklist GitHub Link
Secure Coding Checklist
Summary by Sourcery
Optimize CVE vulnerabilities query planning by pre-aggregating cached counts and isolating group and OS filters into the cached subquery.
Enhancements: