Skip to content

test(configuration): pin 10-14 save, endpoint and response-cap rules that survived mutation - #2421

Merged
freeqaz merged 2 commits into
mainfrom
mutant-sweep-1014b
Oct 14, 2026
Merged

freeqaz merged 2 commits into
mainfrom
mutant-sweep-1014b

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The 10-14 landings added save-time and endpoint rules, each with tests. A mutation sweep checked whether those tests would catch a regression in each rule's conditions. The rules covered:

The sweep generated 252 mutants: flipped comparisons, dropped regex alternatives, removed early returns and off-by-one caps. Main's tests killed 171 of them. These 81 survived:

  • Model selection: openai/ with no model, root tts.summaryModel, fallbacks that is not a list, and an entry that holds only a models policy (Codex projection).
  • Preset variables: an entry primary object whose provider waits on a variable; apiKey SecretRefs whose default env alias, source name or source body waits on one; an entry fallback.
  • canSpell: overlapping literals, and the 64-character literal limit on a Preset Agent ID.
  • openrouter:auto/:free: only the exact selectors read as openrouter.
  • The Gateway password: ${NAME} followed by more text, ${9X}, and an object with no id.
  • Sensitive keys: secret, api_key, encryptKey, privateKey, serviceAccountRef, the file/path/dir/env exemptions, the synology manifest fields, and the first refused key being the one named.
  • Media and token counts: tools.media.video, and the count words (min, num, total, context, input, completion, singular token).
  • Preset option names: a name that holds a variable.
  • URL authority: backslashes, leading space and control characters, scheme case, a tab inside the scheme, and a scheme alone.
  • Size: the size check before the content rules on Agent provisioning.
  • IPv4 endpoint: the message, uppercase schemes, and the spelling refusal coming first in the renderer.
  • Response caps: a response of exactly 4 MiB, declared or streamed, for both Docker and ChatGPT.

Change

Test rows only, in six files, next to the existing rows, with no new helpers:

  • configuration-controller.test.mjs: rows in the model selection, Preset model provider, OpenRouter, roster, key slot and sensitive slot tables. One Preset-only option-name subtest. Agent provisioning added to the size-first test, through the controller, because the provision route's body limit refuses a 1 MiB document first. Sensitive-slot subtests are numbered so names stay unique.
  • url-authority.test.mjs: seven refused spellings, one accepted spelling and one Preset variable spelling.
  • kubernetes-compute.test.mjs: the IPv4 endpoint message is now pinned, HTTPS://192.0.2.1/v1 is accepted, and an entry whose only model setting is models is checked in the Codex projection.
  • profile-renderer.test.mjs: a spelling refusal for an IPv4 host.
  • docker-compute.test.mjs: a /version reply of exactly 4 MiB is read whole.
  • service-account-driver.test.mjs: a ChatGPT reply of exactly 4 MiB, declared or streamed, is read, and one byte more is refused.

Verification

  • Mutation sweep: main's tests killed 171 of 252 mutants; with these rows they kill 231. Each of the 60 newly killed mutants is killed by a new row, and every new row passes on main.
  • The 21 survivors are listed below:
    • 14 are equivalent. Examples:
    • 6 change a verdict only on contrived input. One is the deploy-time IPv4 check in pluginRuntimeSnapshot. Only an openai source base_url stored before fix(credential-sources): refuse rewritten endpoint spellings and non-dns openai hosts #2378 reaches it, and no pure-node test drives such a source.
    • 1 is realistic but deferred to deployment by design. A bare fallback such as fast gets a different save message under the mutant. Main lets a bare primary or fallback through at save, and every deployment refuses it. Board finding 1335 asks whether save should refuse it.
  • Each touched file passed 20 of 20 runs pinned to 2 CPUs (taskset -c 20,21) (configuration-controller, url-authority, docker-compute, service-account-driver, kubernetes-compute, profile-renderer).
  • eslint and prettier --check are clean on the touched files. Prettier was shown to fail on them before formatting.
  • The Docker test runs without a daemon: it uses a loopback server behind a redirected socket.

Risks

Test-only. The 4 MiB bodies add about a second to docker-compute and service-account-driver. None of the touched files is touched by #2415, #2417 or #2418.

…utation

Rows for the model selection, fallback, Preset variable, credential slot,
sensitive key, URL authority, size-first, IPv4 endpoint and response cap
rules whose conditions no test exercised.
@clawsweeper

clawsweeper Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review blocked

Automated review did not run, so no review verdict was produced.

Reason: The input-safety check rejected material in this revision. No detected value, path, or scanner output is reproduced here.

Automatic review is on hold, including after source changes. Request a fresh re-review after resolving the failure; maintainers can also release the hold through an intentional scanner-policy update.

Next step: If this is a genuine credential, remove and rotate it. If it is an intentional test fixture, a maintainer must review and qualify it.

View the workflow run.

@freeqaz
freeqaz marked this pull request as ready for review October 14, 2026 12:46
@freeqaz
freeqaz requested a review from a team as a code owner October 14, 2026 12:46
@freeqaz
freeqaz merged commit 09cd520 into main Oct 14, 2026
46 of 48 checks passed
@freeqaz
freeqaz deleted the mutant-sweep-1014b branch October 14, 2026 13:07

This branch had an error being deployed

1 failed deployment
integration-qa-pr — 5467b5dd Deployed Oct 14, 2026 by freeqaz via QA advisory (kubernetes) #870
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant