Repository navigation
Description
🤖 Generated by the Agentic Engineer
User Story
As a maintainer or agent finishing a Platform change,
I want changes that leave every production input unchanged to complete their required validation without redeploying production,
So that CI-only improvements do not occupy the deployment lane or wait on unrelated production convergence.
Evidence and hypothesis
- The protected candidate for ci(policy): avoid rerunning unrelated mutation fixtures #5065,
dedc6033bb3b39eaed40a99c0f993d4581e4bf39, differs from its baseb8b64944a29ff1bfdd5215def01423640b6a32d8in exactly one file:scripts/tests/test-validate-kyverno-fixture-evaluation.sh(56 changed lines). This is the actual candidate-to-base comparison, not the PR's older file list. - Protected CI run 39172454843 nevertheless entered Deploy to Prod at 22:51:55 UTC on 2026-10-18. It completed successfully; the completed baseline records 38m07s required feedback, 15m43s deployment and 53m00s occupied job time.
- The broad
k8schange classification currently selects both validation and the merge-group production deployment. That classification intentionally includes CI regression scripts. Sharing the selector makes a validation requirement imply a production change. - Hypothesis: distinguishing validation selection from the complete production-input change set removes unnecessary deployment work without changing which checks run or which actual production changes require deployment and acceptance. Savings are not proven yet.
Part of #4979. Related: #4569 (parallel validation scheduling, which deliberately preserves existing admission).
Acceptance Criteria
- Record the completed baseline run's feedback, deployment duration and summed occupied runner time separately. Do not call occupied time a billed-cost measurement. Baseline receipt.
- Trace and test the complete production-input closure before defining a separate selector: rendered manifests, dependency pins, publication inputs, bootstrap/deployment helpers, workflow/action definitions, authorization, and generated or embedded inputs. A filename or extension alone is not proof that an input is irrelevant.
- Add RED/GREEN regression cases showing a genuine CI-test-only change with unchanged production inputs does not select deployment, while independently mutating each production-input family still does. Include generated/embedded inputs and merge-group comparisons.
- Missing comparison data, unknown inputs or failed classification cannot silently skip deployment: require it or fail closed with a named reason.
- Preserve all existing required validation families, strict selected-job result checks, signature/trust checks, queue-membership checks, and production acceptance/heal behavior for actual deployment candidates. Do not skip checks, weaken assertions, raise timeouts or relax protected-branch gates.
- Validate an actual protected test-only candidate: every required check succeeds, production input equality is demonstrated, and deployment is intentionally unselected. Also prove a real production-input candidate still takes the complete deployment/acceptance path.
- Measure at least three comparable successful post-delivery runs against the recorded baseline, reporting feedback time and occupied runner time separately with sample limitations. Keep the outcome issue open until those observations exist.
Refinement and priority rationale
This addresses delivery lead time for every contributor, avoids needless production activity, and can release the serialized deployment lane. It is a candidate for high priority, but must be refined before Ready: map the production-input closure, identify affected fixtures and downstream consumers, and estimate end-to-end implementation, review and protected-queue proof. No savings estimate or readiness claim is assigned from this single observation.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- StatusShow more project fields🏃🏻♂️ In Progress
Activity
devantler commentedon Oct 18, 2026
The motivating protected run has now completed: 39172454843, exact candidate
dedc6033bb3b39eaed40a99c0f993d4581e4bf39, is successful across all 18 terminal jobs. #5065 merged that same commit at 23:07:58 UTC on 2026-10-18, and the merged source head matches the evaluated source head.Baseline from complete native job metadata:
The actual candidate/base diff remains one CI regression script. This measures the avoidable-deployment hypothesis's current obstacle; it does not yet prove the full production-input closure is unchanged or establish that all 15m43s can be recovered. The input-closure analysis, fail-closed regression proof and post-delivery measurements remain required. The issue stays open in Backlog pending that refinement and delivery.
devantler commentedon Oct 19, 2026
One concrete production-input counterexample for this refinement: the current deploy action directly executes
scripts/tests/test-pvc-prune-safety.shas a live persistence-retirement guard before publication. A file being underscripts/tests/therefore does not establish that it is CI-only.Include a mutation of that actual reachable script in the production-input classifier's negative controls: it must require the protected deployment path, even when every rendered manifest is unchanged. Trace actual entrypoints and their reachable scripts/configuration before granting any no-deploy classification. This does not establish the complete closure or justify skipping deployment for the motivating fixture; those remain unproved.
devantler commentedon Nov 13, 2026
Started 13 November 2026, in an interactive session the maintainer asked for. This issue now sits under #6203 (delivery throughput); it was under #4979, whose subject is deploys that fail for reasons outside the change.
Why now, and why high priority. Over the 27 days to 13 November, 244 of 633 production deploys (39%) published manifests identical to
main. They came from 243 pull requests:Each deploy took the production lane for about 11 minutes, and about 40 minutes on 13 November, when jobs waited for runners. On a busy day the lane then handles about 36 deploys while 36 to 45 arrive, so the queue only grows. Without the deploys that change nothing in production, 22 to 27 arrive and the queue drains, even with runners scarce.
Refinement of the open criteria.
scripts/tests/, four setup scripts and nine Go program directories. That list alone is not proof, so the deploy job will run from a working tree that contains only the declared production inputs. A file outside the list is then absent when the deploy runs, and a missed dependency fails that deploy loudly. It cannot be skipped silently.Rationale for the planning fields. Priority High: the largest measured gain for delivery inside this repository, for the maintainer, both agent lanes and every tenant. Effort High: it changes the production deploy job and the required check, and needs a real queue proof for both a skipped and a deployed change.
ci(deploy): harden the production input classifier before it gates an…
ci(deploy): keep the classifier step from failing on its own summary
devantler commentedon Nov 13, 2026
Two requirements for step 2, found in review of the decision record (platform#6207).
Step 1 (platform#6209) only reports whether a queued change touches a production input, and only the merge-queue deploy runs from the list. Step 2 makes the deploy follow that answer. Two things must be true in the same delivery, or the change is unsafe or pointless:
Every deploy path runs from the list. Recovery (
Heal Prod), manual (CD) and rebuild deploys still use the whole checkout. Once a change outside the list may merge without deploying, a file it changed could still reach production through one of those, without the proof its own change would have carried. So before any deploy is skipped, those deploys run from the same list, and the trace starts from their jobs as well as the merge-queue deploy job.The post-merge observer judges by the same list.
scripts/resolve-prod-convergencedecides whether production is behindmainby thek8spaths filter (--filter k8s, the default).k8sbut touches no production input will then merge without deploying.main, so the skipped deploy would happen anyway, one job later.So step 2 is one change with these parts:
wait-for-production-turnanddeploy-prodrun whenneeds.changes.outputs.k8s == 'true' && needs.changes.outputs.production != 'false'.ci-required-checksaccepts a skipped deploy only formerge_group+k8s == 'true'+production == 'false', and still requires success in every other case.resolve-prod-convergencejudges "a deploy input changed" byscripts/production-inputs/inputs.txt.validate-merge-group-heal,validate-dr-signing).Done when a real test-only change merges without a deploy, the observer reads production as converged after it, and a real manifest change still deploys.
Not started. It waits for step 1 to merge and for about twenty clean deploys from the restricted working tree.
Small things to pick up in the same step, from the reviews of step 1:
configWordinscripts/production-inputs/trace.go) does not match a quoted value. Accept quotes.restrictverifies tracked files only. An ignored or untracked file in an unlisted directory would stay in the working tree; a fresh checkout has none.fix(deploy): list five inputs the trace could not see, and teach it t…
test(deploy): see git calls in JavaScript and trace the configuration…
fix(deploy): keep the directory ksail.yaml names in the deploy workin…