Repository navigation
ci: pilot weekly mutation testing on cluster lifecycle and EKS cleanup checks - #8195
Conversation
…p checks A scheduled drift sensor for test quality, not a PR gate. Every Monday the new Mutation Testing workflow runs gremlins (exact pin v0.6.0) on pkg/cli/lifecycle (create/delete orchestration) and internal/maintenance/eksinventory (the fail-closed EKS cleanup verifier). Each job publishes the package's mutation score, its surviving mutants and the step's wall-clock time as a job summary, and keeps the JSON report as an artifact. summarize-mutation-report.sh fails closed: a missing or malformed report, or one in which gremlins examined no mutant, is an error rather than a green run with an empty score. The workflow also runs once on any pull request that changes it, which serves as its own first measurement. Fixes #7849 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codex review |
@coderabbitai review |
|
|
@cursor review |
Action not completed
|
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 30b537bc-4bb5-452d-92fe-19032e65e1c5) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
✅MegaLinter analysis: Success✅ Linters with no issuesactionlint, bash-exec, git_diff, hadolint, jscpd, jsonlint, lychee, markdown-table-formatter, markdownlint, prettier, prettier, shellcheck, shfmt, stylelint, syft, trivy-sbom, trufflehog, v8r, v8r, yamllint See detailed reports in MegaLinter artifacts
|
@coderabbitai full review |
Action performed
|
First mutation-testing run: scores, run times and survivor triageThis is the pilot's first measurement, as promised in the description: run 41797062828 at
Cost. The two jobs run in parallel on Survivor triage. I read each survivor, grouped them, and filed the notable ones:
On a PR-time gate (acceptance criterion 4). About 8 to 12 minutes per package for a full run is too slow to gate every PR, and the survivor lists show the sensor needs test work first. My recommendation is to keep it weekly. Revisit a changed-lines-only gate once #8197 and #8198 raise the baselines. 🤖 Generated with Claude Code |
|
Readiness at
Marking ready. 🤖 Generated with Claude Code |
devantler
left a comment
There was a problem hiding this comment.
🤖 Generated by the Agentic Engineer (interactive Claude Code session)
Self-review (fallback — CodeRabbit, Codex and Cursor Bugbot unavailable)
Reviewed commit: 3f44ca7
- CodeRabbit:
@coderabbitai full reviewwas requested at this head at 15:17:22Z. Its reply at 15:23Z ("Full review finished") and its refreshed summary carry no verdict thatcoderabbit-comment-verdict.shorcoderabbit-summary-verdict.shaccepts (bothNONE no-verdict), and there is no review object at this head. So no recognizable review artifact was produced. The earlier request at 08:07:44Z was rate-limited ("Review limit reached"). - Codex:
@codex reviewat this head was refused at 08:07:43Z with "You have reached your Codex usage limits for code reviews". The same refusal came again in this repository at 15:27:49Z (ksail#8180). That is a usage limit, with no retry window. - Cursor Bugbot: "Bugbot couldn't run - usage limit reached" at 08:07:41Z on this PR. That is a spend limit, which only the maintainer can lift.
Scope: the whole diff against main (.github/workflows/mutation-testing.yaml, .github/scripts/summarize-mutation-report.sh and its test), read independently of its author. I checked it for correctness, security and the repository's review guidelines.
- Correctness. The summarizer fails closed on a missing, empty, non-numeric or vacuous report. "Examined" is killed + lived + timed out, so a run whose mutants were all uncovered is refused. Formatting is done in jq, so the locale cannot change the decimal separator. The survivor table lists only
LIVEDmutants. The workflow runs the summarizer's self-test before trusting it. The PR's own run produced real scores for both packages: 669/74/19/22 and 61/20/0/44. - Security.
permissions: {}at the top level, withcontents: readper job. There are no secrets,persist-credentials: false, and all third-party actions are pinned by SHA. Gremlins is installed at an exact version through the Go checksum database. Thepull_requesttrigger runs only code on the PR's own head, with a read-only token. No untrusted${{ }}input reaches arun:script; the matrix values are literals passed throughenv:. - Workflow contracts. Runs on
mainnever cancel in progress. There is a 90-minute job timeout and a 75-minute step timeout, and both measured steps took under 13 minutes. - Findings: one nit, inline. It does not block.
Verdict: no P0/P1 findings
|
You have reached your Codex usage limits. You can see your limits in the Codex usage dashboard. |
Ready to merge at
|
Correction to my comment above: CodeRabbit did review this head. Its summary comment (6625890582) records the review range ending at 🤖 Generated with Claude Code |

What
This is a pilot of scheduled mutation testing as a drift sensor for test quality, not a pull-request gate. It responds to #7849: agents write most of KSail's code and the tests that check it, so line coverage cannot tell a test that pins behaviour from one that only executes it.
v0.6.0throughGREMLINS_VERSIONand installed withgo install …@v0.6.0, which the Go checksum database verifies. The alternative checked,avito-tech/go-mutesting(MIT, v2.3.1), is also maintained. Gremlins was chosen for its JSON report with efficacy and coverage figures, and for its coverage-guided runs, which test only covered mutants.pkg/cli/lifecycle: cluster create/delete orchestration (unit suite about 1.4 s).internal/maintenance/eksinventory: the fail-closed EKS cleanup verifier, which decides whether a paid trial left anything behind (about 0.7 s)..github/workflows/mutation-testing.yaml.workflow_dispatch, and once on any PR that changes the sensor (its self-test).contents: readonly, no secrets and no caches written..github/scripts/summarize-mutation-report.shfails on a missing, empty or non-gremlins report, and on a report in which gremlins examined no mutants. Its formatting is locale-independent.Validation
.github/scripts/summarize-mutation-report.test.shcovers a scored report (counts row, survivor listed, killed mutant not listed), a clean report, a vacuous, an empty, a missing and a malformed report, and wrong usage. It passes under the host locale andLC_ALL=C, and the workflow runs it before trusting the script.shellcheckis clean.actionlintis clean,zizmorreports no findings, andgo test ./internal/ciharness/passes. That includes the workflow-wide contracts, such as "runs on main never cancels in progress".Fixes #7849
🤖 Generated with Claude Code