fix(nb-36647): drop the image from Job-collapsed backoff fingerprints - #582
Open
mayankpande88 wants to merge 3 commits into
Open
fix(nb-36647): drop the image from Job-collapsed backoff fingerprints#582mayankpande88 wants to merge 3 commits into
mayankpande88 wants to merge 3 commits into
Conversation
Contributor
|
📦 Image Tags Updated |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the fingerprinting logic for the image_pull_backoff matcher. For Job-owned Pods, the failing image is now excluded from the fingerprint to prevent fingerprint fanout in one-Job-per-image workflows, while image-based discrimination is retained for non-Job workloads. Unit tests have been added to verify these behaviors. Feedback was provided to optimize FingerprintFn by avoiding the unconditional call to firstFailingImage(obj) when the owner is a Job, which prevents unnecessary heap allocations and CPU overhead on a hot path.
mayankpande88
added a commit
that referenced
this pull request
Aug 20, 2026
Review feedback on #582: the image was computed and then discarded for every Job-owned Pod, walking all container statuses for a result the fingerprint never uses. Hoist the job-owned check and compute the image only when it is part of the identity. No behaviour change.
Contributor
|
📦 Image Tags Updated |
Follow-up to #580, caught verifying that change on the dev cluster. #580 collapsed a Job owner to its family so repeated runs would share a fingerprint, but kept the failing image in the hash. For a one-Job-per- image creator the image IS the per-run identity, so it re-forked the fingerprint immediately: after #580 shipped, every trivy-image-scan Job in nudgebee-agent-dev still had its own fingerprint, each pulling a different image under scan. Drop the image only when we collapsed a Job owner. Ordinary workloads keep it, so a single typo'd container is still its own Finding. The failing image remains in the evidence blocks either way. Measured on 30d of prod: job-owned 3961 -> 9 fingerprints. Every other owner kind is unchanged (deployment 35, daemonset 8, statefulset 3) — the image component only ever forked identity for Job-owned Pods.
Review feedback on #582: the image was computed and then discarded for every Job-owned Pod, walking all container statuses for a result the fingerprint never uses. Hoist the job-owned check and compute the image only when it is part of the identity. No behaviour change.
mayankpande88
force-pushed
the
fix/nb-36647-imagepull-drop-image-for-job-family
branch
from
August 20, 2026 14:11
094b03e to
da3f60f
Compare
Contributor
|
📦 Image Tags Updated |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Follow-up to #580, found by verifying that change on the dev cluster rather than trusting the projection in its PR body.
#580 collapsed a Job owner to its family so repeated runs of one logical job would share a fingerprint — but kept the failing image in the hash. For a one-Job-per-image creator the image is the per-run identity, so it re-forked the fingerprint immediately and the collapse achieved nothing.
Observed in
nudgebee-agent-dev~25 minutes after #580 rolled out — every scan Job still had its own fingerprint:This drops the image from the fingerprint only when a Job owner was collapsed. Ordinary workloads keep it, so a single typo'd container is still its own Finding. The failing image stays in the evidence blocks either way — it just no longer forks the identity.
Whose noise this actually fixes — read this before the numbers
30 days of production
kubernetes_api_serverevents come from exactly two places: our own tenants (Nudgebee + theiteration-*demo tenants) and one customer, "mohan's Org" /blinq-k8s. Every other tenant with a registered k8s account emits viaprometheus/Azure_Monitor_Alert/pagerduty_webhook— they are not on the Go agent yet. So the population this PR measurably affects is us plus one customer.For that customer,
image_pull_backoff_reporterhas 1 fingerprint in 30 days. The headline below is our own image scanner running against our own dev registry.That is still worth doing — it is a real defect and our demo tenants are what prospects look at — but it should not be read as customer-facing impact. The customer-facing part of this work is the
ConfigurationChangefix in #580, which is validated on blinq's own application workloads (blinq-prediction-hasura-graph: 618 events, 618 fingerprints, max occurrence 1) and rests on a structural cause —resourceVersionadvances on every write in every cluster — rather than on our sample.The one genuine customer benefit here is indirect: 25 of blinq's 27
job_failurefingerprints arekube-bench-scan-*in thenudgebee-agentnamespace — our own agent's scanner generating noise inside their cluster. The job-family collapse takes that to 1. Their two real jobs (blinq-api-healthchecks, CronJob-owned;blinq-clob-indexer-cleardb, no generated suffix) are untouched.cd runner && make validate— exit 0, 37 packages, no failures.Correction to #580
That PR claimed
image_pull_backoff 4,008 -> 72, labelled "excl. image component". The caveat was doing all the work: I measured the collapse with the image excluded and then shipped code that included it. The ConfigurationChange and job_failure numbers in #580 were not affected and are confirmed working on dev —kube-system/calico-typhaproduced 4 config-change events over 11 minutes on a single fingerprint, occurrence 1 -> 4, against a 14-day pre-fix baseline of 1,871 chains all stuck at occurrence 1.Risk
Two different unpullable images within one scan-Job family now collapse to one Finding with a climbing occurrence count instead of one Finding each. That is the intended grouping — "the scanner cannot pull images" is one problem — and the per-image detail is preserved in evidence.
JobFamilyis calibrated on a thin sample. It strips-<digits>/-<8+ hex>/-<uuid>tails, and the five job-name families in the data aretrivy-image-scan,kube-bench-scan,pinot-segment-push-v2,grade-astropy--astropyandblinq-api-healthchecks— three of them ours. The only customer job names available to test against are two, and neither exercises the heuristic. A customer runningetl-20240101/etl-20240102gets the intended grouping; one running genuinely unrelatedreport-a1b2c3d4andreport-deadbeefwould see them merged. There is no data to bound that today. If it becomes a problem, the containment is to restrict stripping to CronJob-owned Jobs plus an explicit prefix allowlist.Part of nudgebee/nudgebee-enterprise#36647