Investigation: Expand test failure dashboard

[yulianedyalkova]

We have a Grafana dashboard (creds) that was supposed to help us monitor flakes and figure out which ones need to get prioritized sooner. Sadly, it's currently not very actionable as it only shows the failed case and we noticed that the same test can fail for many different reasons. So the stats don't help us much.

Questions to Answer

• What does crust-gather output contain? What fields/signals are most useful for root cause categorization?
• How can we consume this data in our Grafana dashboard?

[or4dx]

Crust gather collects a fairly detailed snapshot of the current state of the workload cluster excluding the following;

• Secrets - crust-gather excludes secret contents by default
• Lease resources - excluded to reduce noise (leader election objects)
• EndpointSlice and ControllerRevision - excluded from WC snapshots to reduce size
• MC Node resources - excluded from MC snapshots (the MC manages itself; node collection would try to create debug pods on the long-lived MC which is blocked by policy)
• Cluster state after teardown - the snapshot is taken before cluster deletion; if teardown itself fails there is no second snapshot

Also for our management cluster the snapshot is scoped to the workload's clusters namespace in the management cluster. As a result of this I believe the value that can be extracted from crust gather to extend our dashboard include;

• Events by reason in the failure window
• Sum / max container restartCount in the target namespace
• Pods by terminal waiting.reason
• CAPI custom resource conditions False (with reason)
• Controller ERROR/WARN line counts, bucketed by controller

We should be able to convert the crust-gather data into custom metrics we persist in a some sort of data source(like mimir,postgres). I am actively looking to how this is done so I following existing standard for storing the custom metrics we create from the crust-gather archive.

[or4dx]

Following up on the investigation above: we've built a dedicated Go service, kepler-collector, that implements what was proposed and answers both of the questions above with real data rather than assumptions.

How it consumes the crust-gather data

kepler-collector runs on a schedule, picks up each new crust-gather archive as it's pushed to the registry, and reads exactly the signals identified in the earlier comment, events by reason, pod waiting reasons, CAPI/CAPA condition failures, and container restart counts. It turns this evidence into a Record for each failure, classifies it (infra failure vs. app failure, and which specific component or app is responsible), and resolves a real owning team using the app's own metadata and its GitHub CODEOWNERS file.

How it gets into the dashboard

Each record is sent two ways over OTLP, aggregate counts and rates go to Mimir as metrics, and a detailed, searchable log line per failure goes to Loki. This means the dashboard can now show not just that a test failed, but why it failed, which specific app or component was responsible, and which team should be notified, which is exactly the gap the original dashboard had.

Where we are now

The pipeline is built and working end-to-end against real failure archives. We've been refining the classification logic as new real failures come in, so that failures get attributed to a specific team and cause rather than a generic bucket, and we've fixed several real bugs (like the collector reporting the wrong management cluster) that only surfaced once we looked at genuine data. The next step is to keep validating this against new real failures and start wiring the classified data into the Grafana dashboard and, longer term, an alert (e.g. a Slack notification).

[or4dx]

Update on repo setup

kepler-collector is now a real git repo, pushed to giantswarm/kepler-collector (private).

Since then, added the scaffolding the repo was missing (checked against template-app and the closer real analog cluster-api-events):

• .dockerignore, .nancy-ignore, a generated values.schema.json for the Helm chart, and Chart.yaml metadata (home/icon/team annotations, per current GS chart-metadata conventions)
• A CircleCI pipeline (devctl gen circleci) to build/push the container image to gsoci and publish the chart to giantswarm-catalog

This is all on branch add-repo-scaffolding, pushed, not yet merged.

Currently investigating: the first real PR triggered a wall of golangci-lint goconst failures (repeated string literals) — mostly noise from idiomatic literal repetition in test files, since this is the first time this repo's pre-commit CI has actually run. Working out the right fix before merging.

Still blocked: devctl repo setup (branch protection / Renovate access) 403s — needs an org owner, not something we can self-serve.

[or4dx]

Update: CircleCI pipeline now actually working

Since the last update, kepler-collector moved from "repo exists" to "CI is actually building and pushing real artifacts":

• Removed the pre-commit CI workflow entirely (confirmed not required, and cluster-api-events doesn't run it either); fixed the real golangci-lint findings it surfaced (.golangci.yml with goconst.ignore-tests, plus genuine constant extractions — a shared CAPI API version string, two coincidentally-identical-but-unrelated "unknown" fallbacks).
• Got CircleCI itself actually watching the repo and building (devctl gen circleci-generated pipeline: image build/push to gsoci, chart package/push to giantswarm-catalog).
• Fixed two real chart issues surfaced by the pipeline's own validators: abs's HasTeamLabel check (Chart.yaml's team annotation wasn't propagated into a rendered label) and kube-linter's no-read-only-root-fs check (added readOnlyRootFilesystem: true + a /tmp emptyDir, genuinely needed since crust-gather and the OCI-pull fallback both write temp files).

New, separate scope identified (not started): the chart-test job (app-test-suite/ATS) needs an actual tests/ats/ pytest suite doesn't exist yet. Not blocking this branch's builds, but will block real tagged releases (push-chart-release requires the chart tests to pass).

I intend to collaborate with @njuettner on this tomorrow.

[or4dx]

Update: Deployed kepler-collector to gazelle-cicdprod

Deployed the kepler-collector app (giantswarm-catalog chart) to the gazelle-cicdprod WC, as a HelmRelease/OCIRepository rather than an App CR (#2039, #2043, #2044).

What's in place (cicdprod/apps/kepler-collector/):

helm-release.yaml + oci-repository.yaml — pulls oci://gsociprivate.azurecr.io/charts/giantswarm/kepler-collector
configmap.yaml — overrides image.repository (private ACR mirror), ociAuth.existingSecret, imagePullSecrets
Two encrypted secrets committed for audit/recovery: chart-pull creds (kepler-pull-secret.enc.yaml) and the app's OCI runtime creds (kepler-collector-registry-credentials.enc.yaml) — a dedicated read-only credential, kept isolated from the crust-gather push credential
New .sops.yaml rule extending the existing cicdprod PGP key to the apps/ path

I also needed to add an upstream chart fix because the kepler-collector had no imagePullSecrets support at all (its default image lives in a private registry, but neither the pod spec nor the ServiceAccount template had a way to attach pull creds), causing ErrImagePullBackOff. Added imagePullSecrets to the chart (now v0.2.0), wired onto the ServiceAccount (pods inherit it automatically).

Verified live: OCIRepository/HelmRelease both Ready: True on chart 0.2.0, ServiceAccount has the correct imagePullSecrets, ociAuth.existingSecret matches the actual (unprefixed, manually-created) secret name on the WC.

I also reviewed the dasboard and I can confirm that the metrics and logs are coming through. I need to take a closer look at the quality of data being sent through since I had used some test data in the past that caused some repetition in the Recent Failures table.

I am proceeding to move this task from in progress.