Skip to content

Avoid production redeployment when only CI validation inputs change #5077

Description

@devantler

🤖 Generated by the Agentic Engineer

User Story

As a maintainer or agent finishing a Platform change,
I want changes that leave every production input unchanged to complete their required validation without redeploying production,
So that CI-only improvements do not occupy the deployment lane or wait on unrelated production convergence.

Evidence and hypothesis

  • The protected candidate for ci(policy): avoid rerunning unrelated mutation fixtures #5065, dedc6033bb3b39eaed40a99c0f993d4581e4bf39, differs from its base b8b64944a29ff1bfdd5215def01423640b6a32d8 in exactly one file: scripts/tests/test-validate-kyverno-fixture-evaluation.sh (56 changed lines). This is the actual candidate-to-base comparison, not the PR's older file list.
  • Protected CI run 39172454843 nevertheless entered Deploy to Prod at 22:51:55 UTC on 2026-10-18. It completed successfully; the completed baseline records 38m07s required feedback, 15m43s deployment and 53m00s occupied job time.
  • The broad k8s change classification currently selects both validation and the merge-group production deployment. That classification intentionally includes CI regression scripts. Sharing the selector makes a validation requirement imply a production change.
  • Hypothesis: distinguishing validation selection from the complete production-input change set removes unnecessary deployment work without changing which checks run or which actual production changes require deployment and acceptance. Savings are not proven yet.

Part of #4979. Related: #4569 (parallel validation scheduling, which deliberately preserves existing admission).

Acceptance Criteria

  • Record the completed baseline run's feedback, deployment duration and summed occupied runner time separately. Do not call occupied time a billed-cost measurement. Baseline receipt.
  • Trace and test the complete production-input closure before defining a separate selector: rendered manifests, dependency pins, publication inputs, bootstrap/deployment helpers, workflow/action definitions, authorization, and generated or embedded inputs. A filename or extension alone is not proof that an input is irrelevant.
  • Add RED/GREEN regression cases showing a genuine CI-test-only change with unchanged production inputs does not select deployment, while independently mutating each production-input family still does. Include generated/embedded inputs and merge-group comparisons.
  • Missing comparison data, unknown inputs or failed classification cannot silently skip deployment: require it or fail closed with a named reason.
  • Preserve all existing required validation families, strict selected-job result checks, signature/trust checks, queue-membership checks, and production acceptance/heal behavior for actual deployment candidates. Do not skip checks, weaken assertions, raise timeouts or relax protected-branch gates.
  • Validate an actual protected test-only candidate: every required check succeeds, production input equality is demonstrated, and deployment is intentionally unselected. Also prove a real production-input candidate still takes the complete deployment/acceptance path.
  • Measure at least three comparable successful post-delivery runs against the recorded baseline, reporting feedback time and occupied runner time separately with sample limitations. Keep the outcome issue open until those observations exist.

Refinement and priority rationale

This addresses delivery lead time for every contributor, avoids needless production activity, and can release the serialized deployment lane. It is a candidate for high priority, but must be refined before Ready: map the production-input closure, identify affected fixtures and downstream consumers, and estimate end-to-end implementation, review and protected-queue proof. No savings estimate or readiness claim is assigned from this single observation.

Activity

  1. moved this to 📥 Backlog in 🌊 Project Boardon Oct 18, 2026
  2. devantler commented on Oct 18, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The motivating protected run has now completed: 39172454843, exact candidate dedc6033bb3b39eaed40a99c0f993d4581e4bf39, is successful across all 18 terminal jobs. #5065 merged that same commit at 23:07:58 UTC on 2026-10-18, and the merged source head matches the evaluated source head.

    Baseline from complete native job metadata:

    • Creation to successful required aggregation: 38m07s (22:29:36–23:07:43 UTC).
    • Production deployment job: 15m43s (22:51:55–23:07:38 UTC).
    • Summed occupied non-skipped job time: 53m00s. This is runner occupancy, not billed cost.

    The actual candidate/base diff remains one CI regression script. This measures the avoidable-deployment hypothesis's current obstacle; it does not yet prove the full production-input closure is unchanged or establish that all 15m43s can be recovered. The input-closure analysis, fail-closed regression proof and post-delivery measurements remain required. The issue stays open in Backlog pending that refinement and delivery.

  3. devantler commented on Oct 19, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    One concrete production-input counterexample for this refinement: the current deploy action directly executes scripts/tests/test-pvc-prune-safety.sh as a live persistence-retirement guard before publication. A file being under scripts/tests/ therefore does not establish that it is CI-only.

    Include a mutation of that actual reachable script in the production-input classifier's negative controls: it must require the protected deployment path, even when every rendered manifest is unchanged. Trace actual entrypoints and their reachable scripts/configuration before granting any no-deploy classification. This does not establish the complete closure or justify skipping deployment for the motivating fixture; those remain unproved.

  4. moved this from 📥 Backlog to 🏃🏻‍♂️ In Progress in 🌊 Project Boardon Nov 13, 2026
  5. self-assigned this
    on Nov 13, 2026
  6. updated Priority High and Effort High on Nov 13, 2026
  7. devantler commented on Nov 13, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer (interactive Claude Code session)

    Started 13 November 2026, in an interactive session the maintainer asked for. This issue now sits under #6203 (delivery throughput); it was under #4979, whose subject is deploys that fail for reasons outside the change.

    Why now, and why high priority. Over the 27 days to 13 November, 244 of 633 production deploys (39%) published manifests identical to main. They came from 243 pull requests:

    What the pull request touched Pull requests
    Only tests, documentation or fixtures 38
    Only scripts the deploy never references 71
    The CI or CD workflow file, plus such scripts 75
    Go module files or shared Go packages 33
    A file the deploy action references directly 26

    Each deploy took the production lane for about 11 minutes, and about 40 minutes on 13 November, when jobs waited for runners. On a busy day the lane then handles about 36 deploys while 36 to 45 arrive, so the queue only grows. Without the deploys that change nothing in production, 22 to 27 arrive and the queue drains, even with runners scarce.

    Refinement of the open criteria.

    • Closure. A static trace from the deploy action reaches 56 of about 820 non-test script files, 12 files under scripts/tests/, four setup scripts and nine Go program directories. That list alone is not proof, so the deploy job will run from a working tree that contains only the declared production inputs. A file outside the list is then absent when the deploy runs, and a missed dependency fails that deploy loudly. It cannot be skipped silently.
    • First delivery. Every workflow file and the Go module files stay production inputs; narrowing them is a later step with its own proof. Replaying the actual list (ci(deploy): run the queue deploy from the declared production inputs #6209) over the squash commits of the 629 merges that deployed in the window, 74 (12%) touch nothing on it and would stop deploying. That is fewer than the first two rows suggest, because the list keeps all workflows and because this replay compares each merged commit with its parent and does not rely on the pull request's file list.
    • Fail closed. The selector answers "deploy" for a missing base, an unreadable list, an unknown path or any error, and says why.

    Rationale for the planning fields. Priority High: the largest measured gain for delivery inside this repository, for the maintainer, both agent lanes and every tenant. Effort High: it changes the production deploy job and the required check, and needs a real queue proof for both a skipped and a deployed change.

  8. devantler commented on Nov 13, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer (interactive Claude Code session)

    Two requirements for step 2, found in review of the decision record (platform#6207).

    Step 1 (platform#6209) only reports whether a queued change touches a production input, and only the merge-queue deploy runs from the list. Step 2 makes the deploy follow that answer. Two things must be true in the same delivery, or the change is unsafe or pointless:

    Every deploy path runs from the list. Recovery (Heal Prod), manual (CD) and rebuild deploys still use the whole checkout. Once a change outside the list may merge without deploying, a file it changed could still reach production through one of those, without the proof its own change would have carried. So before any deploy is skipped, those deploys run from the same list, and the trace starts from their jobs as well as the merge-queue deploy job.

    The post-merge observer judges by the same list.

    • scripts/resolve-prod-convergence decides whether production is behind main by the k8s paths filter (--filter k8s, the default).
    • A change that matches k8s but touches no production input will then merge without deploying.
    • The observer would read production as BEHIND after each such merge and redeploy main, so the skipped deploy would happen anyway, one job later.

    So step 2 is one change with these parts:

    1. The recovery, manual and rebuild deploys run from the list; the trace covers the steps of their jobs.
    2. wait-for-production-turn and deploy-prod run when needs.changes.outputs.k8s == 'true' && needs.changes.outputs.production != 'false'.
    3. ci-required-checks accepts a skipped deploy only for merge_group + k8s == 'true' + production == 'false', and still requires success in every other case.
    4. resolve-prod-convergence judges "a deploy input changed" by scripts/production-inputs/inputs.txt.
    5. The contract tests that pin the deploy selector and the required result are updated to the new rule, not loosened (validate-merge-group-heal, validate-dr-signing).

    Done when a real test-only change merges without a deploy, the observer reads production as converged after it, and a real manifest change still deploys.

    Not started. It waits for step 1 to merge and for about twenty clean deploys from the restricted working tree.

    Small things to pick up in the same step, from the reviews of step 1:

    • The trace's rule for a single-word directory name in a configuration (configWord in scripts/production-inputs/trace.go) does not match a quoted value. Accept quotes.
    • restrict verifies tracked files only. An ignored or untracked file in an unlisted directory would stay in the working tree; a fresh checkout has none.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Fields

Priority

High

Effort

High

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions