Skip to content

ci: pilot weekly mutation testing on cluster lifecycle and EKS cleanup checks - #8195

Merged
devantler merged 1 commit into
mainfrom
claude/ci-mutation-testing-pilot-7849
Nov 4, 2026
Merged

devantler merged 1 commit into
mainfrom
claude/ci-mutation-testing-pilot-7849

Conversation

@devantler

Copy link
Copy Markdown
Contributor

What

This is a pilot of scheduled mutation testing as a drift sensor for test quality, not a pull-request gate. It responds to #7849: agents write most of KSail's code and the tests that check it, so line coverage cannot tell a test that pins behaviour from one that only executes it.

  • Tool: go-gremlins/gremlins, Apache-2.0, maintained (v0.6.0 in December 2025; commits through 2026). It is pinned to the exact version v0.6.0 through GREMLINS_VERSION and installed with go install …@v0.6.0, which the Go checksum database verifies. The alternative checked, avito-tech/go-mutesting (MIT, v2.3.1), is also maintained. Gremlins was chosen for its JSON report with efficacy and coverage figures, and for its coverage-guided runs, which test only covered mutants.
  • Packages (where a silent regression is costly):
    • pkg/cli/lifecycle: cluster create/delete orchestration (unit suite about 1.4 s).
    • internal/maintenance/eksinventory: the fail-closed EKS cleanup verifier, which decides whether a paid trial left anything behind (about 0.7 s).
  • New workflow .github/workflows/mutation-testing.yaml.
    • Runs weekly (Monday 04:23 UTC), on workflow_dispatch, and once on any PR that changes the sensor (its self-test).
    • Has a job per package with contents: read only, no secrets and no caches written.
    • Each job publishes a per-package mutation score (killed / lived / timed out / not covered / not viable, test efficacy, mutant coverage), the list of surviving mutants and the wall-clock time of the mutation step as a job summary. It keeps the JSON report for 30 days.
  • No vacuous green. .github/scripts/summarize-mutation-report.sh fails on a missing, empty or non-gremlins report, and on a report in which gremlins examined no mutants. Its formatting is locale-independent.

Validation

  • .github/scripts/summarize-mutation-report.test.sh covers a scored report (counts row, survivor listed, killed mutant not listed), a clean report, a vacuous, an empty, a missing and a malformed report, and wrong usage. It passes under the host locale and LC_ALL=C, and the workflow runs it before trusting the script. shellcheck is clean.
  • actionlint is clean, zizmor reports no findings, and go test ./internal/ciharness/ passes. That includes the workflow-wide contracts, such as "runs on main never cancels in progress".
  • First measurement. This PR's own run of the workflow is the pilot's first run. I will add its scores, run times and the triage of notable surviving mutants (filed issues, or reasons for equivalent mutants) to this PR before it is promoted.

Fixes #7849

🤖 Generated with Claude Code

…p checks

A scheduled drift sensor for test quality, not a PR gate. Every Monday the
new Mutation Testing workflow runs gremlins (exact pin v0.6.0) on
pkg/cli/lifecycle (create/delete orchestration) and
internal/maintenance/eksinventory (the fail-closed EKS cleanup verifier).

Each job publishes the package's mutation score, its surviving mutants and
the step's wall-clock time as a job summary, and keeps the JSON report as an
artifact. summarize-mutation-report.sh fails closed: a missing or malformed
report, or one in which gremlins examined no mutant, is an error rather than
a green run with an empty score. The workflow also runs once on any pull
request that changes it, which serves as its own first measurement.

Fixes #7849

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

@codex review

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

@coderabbitai review

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

@devantler

Copy link
Copy Markdown
Contributor Author

@cursor review

@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@cursor

cursor Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 30b537bc-4bb5-452d-92fe-19032e65e1c5)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-11-04T08:07:37.838109Z
We couldn’t complete this review. Comment @codex review to try again.
3f44ca7 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Adds a Bash script that validates Gremlins JSON reports and prints mutation metrics and any surviving mutants as Markdown. Adds tests for valid reports, error cases, and argument handling. Adds a GitHub Actions workflow that runs Gremlins on two packages weekly, manually, and for selected pull request changes. The workflow publishes mutation summaries and uploads reports.

Merge Risk: Minimal · up to 3f44c

The scheduled mutation-testing pilot is mergeable with normal checks. The suggested additional fixture covers a report shape the pinned tool cannot produce.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely identifies the weekly mutation-testing pilot and the two targeted areas: cluster lifecycle and EKS cleanup checks.
Description check Passed The description directly explains the scheduled mutation-testing workflow, targeted packages, reporting, validation, and pilot objectives.
Linked Issues check Passed Directly linked issue #7849 has four acceptance criteria. Three are coding requirements. One is an operational step.

1. Scheduled workflow with an exact tool pin [#7849]: Met. `.github/workflows/muta…
Out of Scope Changes check Passed The whole-PR diff against the merge base adds three files. Each file serves #7849.

- .github/workflows/mutation-testing.yaml: the scheduled and manual Gremlins workflow. It contains the pin, the pe…
Full details: Linked Issues check

Explanation

Directly linked issue #7849 has four acceptance criteria. Three are coding requirements. One is an operational step.

  1. Scheduled workflow with an exact tool pin [#7849]: Met. .github/workflows/mutation-testing.yaml sets on.schedule cron 23 4 * * 1. It sets GREMLINS_VERSION: v0.6.0 and installs with go install "github.com/go-gremlins/gremlins/cmd/gremlins@${GREMLINS_VERSION}". The matrix covers pkg/cli/lifecycle and internal/maintenance/eksinventory, which matches the packages named in the issue.

  2. Per-package mutation score, and failure when no mutants are examined [#7849]: Met. The Publish the mutation score step runs summarize-mutation-report.sh under set -euo pipefail and appends its stdout to $GITHUB_STEP_SUMMARY. The script exits 1 when killed + lived + timed_out == 0, when the report is missing, empty, or malformed, or when a numeric field is not a number. A non-zero exit fails the job. summarize-mutation-report.test.sh covers scored, clean, vacuous, empty, missing, malformed, and usage cases. The test file has checks for each case, and the workflow runs it before the mutation step.

  3. Runtime and cost recorded [#7849]: Met for runtime. The mutation step records elapsed-seconds. The summary step prints the wall-clock time per package. Cost is not computed as a separate figure. Runtime is the recorded input for cost.

  4. Triage of the first run's surviving mutants [#7849]: Not a coding requirement. It depends on the first run's output and on issue filing. The PR description states this is deferred until after the first run. This assessment does not count it against the PR.

Residual uncertainty: The gremlins v0.6.0 CLI flags (--workers, --output) and JSON field names (mutants_killed, files[].mutations[].status, and others) cannot be verified against upstream source with the available tools. The test fixture uses the same names, so it does not prove they match gremlins output. If they differ, the summarizer fails closed on every run. This does not produce a vacuous green result.

Full details: Out of Scope Changes check

Explanation

The whole-PR diff against the merge base adds three files. Each file serves #7849.

  • .github/workflows/mutation-testing.yaml: the scheduled and manual Gremlins workflow. It contains the pin, the per-package jobs, the score publication, and the report artifact.
  • .github/scripts/summarize-mutation-report.sh: the fail-closed report summarizer. It is called by the workflow.
  • .github/scripts/summarize-mutation-report.test.sh: tests for the summarizer. It is run by the workflow.

The workflow uses read-only permissions, no secrets, and no cache writes. This matches the issue's pilot scope. The PR description cites internal/ciharness contract tests. The changed-file summary does not list changes to internal/ciharness, so no out-of-scope source change exists there. No change touches product code or unrelated workflows.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 2 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share
Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between e6c8845 and 3f44ca7.

Files selected for processing: 3
  • .github/scripts/summarize-mutation-report.sh
  • .github/scripts/summarize-mutation-report.test.sh
  • .github/workflows/mutation-testing.yaml

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Action performed

Full review finished.

@devantler

Copy link
Copy Markdown
Contributor Author

First mutation-testing run: scores, run times and survivor triage

This is the pilot's first measurement, as promised in the description: run 41797062828 at 3f44ca7cf1 (gremlins v0.6.0, coverage-guided).

Package Killed Lived Timed out Not covered Test efficacy Mutator coverage Mutation step Whole job
pkg/cli/lifecycle 61 20 0 44 75.31 % 64.80 % 7 min 42 s 12 min 31 s
internal/maintenance/eksinventory 669 74 19 22 90.04 % 97.12 % 12 min 14 s 13 min 41 s

Cost. The two jobs run in parallel on ubuntu-latest, so one run takes about 14 minutes of wall-clock time and about 26 runner-minutes. This is a public repository, so GitHub-hosted minutes cost nothing. The real cost is runner contention: the weekly slot (Monday 04:23 UTC) sits outside working hours. Both mutation steps stay well under their 75-minute step timeout.

Survivor triage. I read each survivor, grouped them, and filed the notable ones:

  • ksail#8197: pkg/cli/lifecycle. 18 of its 20 survivors negate a condition in the standalone target resolution in simple.go, covering kubeconfig precedence, the eks.yaml-name guard and the ValidateStandaloneAWSTarget early returns. The branches run, but no test case makes the two candidate answers differ.
  • ksail#8198: internal/maintenance/eksinventory. A successful read that then returns early, before its rows are checked (ownership.go:1156, 1318, 931, capacity_accounting.go:328). Also the terminated-instance skip, the count of incomplete observations, the shape of the quota row and request, and the resource classification.
  • Explained, not filed (the reasons are in ksail#8198): about 45 boundary mutants on defensive size caps, where one side refuses and fails closed and the other accepts one more item of data that is already validated; capacity hints such as make(map, n+1); one redundant clause, verification.go:118, which its sibling clauses already cover; and one error-to-error swap, capacity.go:105. The 19 timed-out mutants (mostly in cmd/main.go) count as detected.

On a PR-time gate (acceptance criterion 4). About 8 to 12 minutes per package for a full run is too slow to gate every PR, and the survivor lists show the sensor needs test work first. My recommendation is to keep it weekly. Revisit a changed-lines-only gate once #8197 and #8198 raise the baselines.

🤖 Generated with Claude Code

@devantler

Copy link
Copy Markdown
Contributor Author

Readiness at 3f44ca7cf1:

  • CI: every check at this head is green, including the required gate and both 🧬 Mutate jobs (run 41797062828).
  • Review: CodeRabbit's full review, requested at 15:17Z, finished at 15:23Z ("Full review finished"). Its refreshed summary covers all three files, e6c8845cec… → 3f44ca7cf16b6e9c5a08b5630772e9306dfe0f80, and its verdict is "No actionable comments were generated in the recent review". There are no review threads.
    • Note for whoever checks this with coderabbit-summary-verdict.sh: the helper reports NONE no-verdict here. CodeRabbit now writes the verdict in a separate recent_review_status block, outside the recent_review block the helper reads.
  • Tried as a user:
    • The pilot ran for real on this PR. Its scores, run times and survivor triage are in the comment above, with follow-ups ksail#8197 and ksail#8198.
    • summarize-mutation-report.test.sh passes locally under both the host locale and LC_ALL=C.
  • Acceptance criteria of Pilot scheduled mutation testing on KSail's critical packages #7849:
    1. Scheduled, pinned workflow: met.
    2. Per-package score, failing on a vacuous run: met.
    3. First run's survivors triaged into issues or explained: met.
    4. Run time and cost recorded: met.

Marking ready.

🤖 Generated with Claude Code

@devantler
devantler marked this pull request as ready for review November 4, 2026 15:44

@devantler devantler left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Self-review (fallback — CodeRabbit, Codex and Cursor Bugbot unavailable)

Reviewed commit: 3f44ca7

  • CodeRabbit: @coderabbitai full review was requested at this head at 15:17:22Z. Its reply at 15:23Z ("Full review finished") and its refreshed summary carry no verdict that coderabbit-comment-verdict.sh or coderabbit-summary-verdict.sh accepts (both NONE no-verdict), and there is no review object at this head. So no recognizable review artifact was produced. The earlier request at 08:07:44Z was rate-limited ("Review limit reached").
  • Codex: @codex review at this head was refused at 08:07:43Z with "You have reached your Codex usage limits for code reviews". The same refusal came again in this repository at 15:27:49Z (ksail#8180). That is a usage limit, with no retry window.
  • Cursor Bugbot: "Bugbot couldn't run - usage limit reached" at 08:07:41Z on this PR. That is a spend limit, which only the maintainer can lift.

Scope: the whole diff against main (.github/workflows/mutation-testing.yaml, .github/scripts/summarize-mutation-report.sh and its test), read independently of its author. I checked it for correctness, security and the repository's review guidelines.

  • Correctness. The summarizer fails closed on a missing, empty, non-numeric or vacuous report. "Examined" is killed + lived + timed out, so a run whose mutants were all uncovered is refused. Formatting is done in jq, so the locale cannot change the decimal separator. The survivor table lists only LIVED mutants. The workflow runs the summarizer's self-test before trusting it. The PR's own run produced real scores for both packages: 669/74/19/22 and 61/20/0/44.
  • Security. permissions: {} at the top level, with contents: read per job. There are no secrets, persist-credentials: false, and all third-party actions are pinned by SHA. Gremlins is installed at an exact version through the Go checksum database. The pull_request trigger runs only code on the PR's own head, with a read-only token. No untrusted ${{ }} input reaches a run: script; the matrix values are literals passed through env:.
  • Workflow contracts. Runs on main never cancel in progress. There is a 90-minute job timeout and a 75-minute step timeout, and both measured steps took under 13 minutes.
  • Findings: one nit, inline. It does not block.

Verdict: no P0/P1 findings

Comment thread .github/workflows/mutation-testing.yaml Resolved
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits. You can see your limits in the Codex usage dashboard.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Ready to merge at 3f44ca7cf1: required checks green, no unresolved threads, local-review-verdict.sh GREEN at this head, no conflict, and the first mutation run triaged. CodeRabbit, Codex and Bugbot gave no usable review at this head, so the review is a local round. My merge attempt was refused by the session's permission classifier ("Merge Without Review"). I did not retry it in another form. It needs a maintainer merge or a bot review:

gh pr merge 8195 --repo devantler-tech/ksail --squash --match-head-commit 3f44ca7cf16b6e9c5a08b5630772e9306dfe0f80

@devantler
devantler merged commit 971c41f into main Nov 4, 2026
69 checks passed
@devantler
devantler deleted the claude/ci-mutation-testing-pilot-7849 branch November 4, 2026 16:08
@github-project-automation github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Nov 4, 2026
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Correction to my comment above: CodeRabbit did review this head. Its summary comment (6625890582) records the review range ending at 3f44ca7cf1 and "No actionable comments were generated in the recent review". The summary-verdict helper on main, fixed in monorepo#4812, reads it as GREEN; the stale copy I ran earlier could not parse the new layout. So this head has a CodeRabbit green and a GREEN local round.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

Pilot scheduled mutation testing on KSail's critical packages

1 participant