You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Reduce CI critical-path time while preserving validationΒ #5495
Recent successful merge-queue runs take 28β29 minutes, delaying delivery and recovery while each change waits for the next deployment.
Baseline
Runs 40639023721, 40641879084 and 40644713318 took 1,732, 1,684 and 1,751 seconds from first job start to last completion. Deployment took 989β1,044 seconds. The publication-generator validation job took 557β709 seconds, and repository guards took 528β608 seconds. These are measurements, not a demonstrated optimization.
Acceptance
Measure individual validation helpers and the dependency path before choosing a change. Identify redundant work or independent, isolated fixtures that can run concurrently with bounded resource use.
Preserve every required check, negative control, trust boundary and deployment health check. Failures, timeouts and incomplete background work must still fail the required aggregate.
Keep logs attributable to each helper and retain an equivalent serial path for comparison and diagnosis.
Demonstrate a material improvement on comparable runs, report validation and total critical-path time separately, then verify the protected default-branch run and production recovery.
Use existing execution capacity; changing paid capacity or financial settings is outside this change.
Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.
Create Plan
π§ͺ Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your .coderabbit.yaml:
issue_enrichment:
auto_enrich:
enabled: false
π¬ Have feedback or questions? Drop into our discord!
The bounded publication runner improved the matched local nine-fixture execution from 742s serial to 406s with two isolated workers. In the protected run, the publication validation job took 451s versus baseline job times of 557β709s; its actual fixture runner took 390s. All nine fixtures completed successfully.
The overall readiness result is not yet an improvement. A fresh analysis of the actual workflow dependencies and complete native job timestamps puts the required-gate path at 2,742s, versus 1,657β1,762s for the same ready-to-required-completion measure in the three baseline runs. Its longest observed path was change detection β Helm post-renderer validation β production deployment β required aggregate: 1,594s ready-to-start waiting, 1,147s execution, plus a one-second transition. The post-renderer job alone waited 865s before its 192s execution. Deployment waiting can also include the protected concurrency lock; these timestamps do not establish a hosting-capacity cause.
The later queue-membership observer extends the full run to 3,107s but intentionally does not block the required gate or merging. It is excluded from the next readiness optimization target. This issue stays open: preserve all validators and deployment checks while reducing avoidable waiting or repeated execution within existing capacity.
The measurement workflow has shipped in #5727. Its first main invocation passed all four arms at the same source: Go 4/8 workers took 556.95/520.90 seconds, and publication 2/4 workers took 600.16/354.39 seconds. All nine publication fixtures passed in each arm. The delivery receipt records CPU cost, exact named outcome reconciliation and the stdout-record gaps. One pair does not establish whole-pipeline causality or retire the adverse samples.
The lookup repair I mentioned earlier is on hold: #5739 is already in the protected queue, and the related shared library has another delivery owner. I removed my empty, unmodified repair worktree rather than creating a competing implementation. The failed #5750 publication fixture is retained and has not been blindly rerun.
The main Go profile points to the credential-refresh regression package as the remaining long pole: 466 seconds with four workers and 434 seconds with eight; the next slowest package took 53/63 seconds. Profiling that fixture's process and execution costs is a stronger next candidate than increasing worker limits again. No validation or capacity change is made by this observation. This issue stays open for reliability and outcome verification.
Profile of the slowest validator, and why runner wait is the bigger target. Nothing was changed by this reading.
The Go long pole has no sleeps to remove. I traced one case of the refresh-flux-ghcr-auth package locally, which is macOS rather than the CI runner, so the proportions are indicative only.
Measure, one case
Reading
Wall time
8.05 s, of which 6.66 s was CPU
Fake commands started
561, of which 502 were kubectl
Startup of one fake command
about 4 ms, the same as an empty Go program
Share of traced time in renew_sync_lease
about 35%, from 121 calls
The fake commands are not the cost, and the test's sync interval is already 0.
The case is CPU-bound, which fits 8 workers barely beating 4 on these runners (434 s against 466 s).
Each lease renewal reads the lease and then patches it with a resource-version check. TestAfterReleaseRegressionDetectsEachAblatedProductionFence pins that check, so it is a safety control and I did not propose removing it. Merging the jq and date calls in that function would save about 8% of one job, which is not worth changing a fence-critical script.
Waiting for a runner costs far more than this job.
This is one run during a busy period. It shows waiting can dominate, not how often it does. The one wave source I have sized is now monorepo#4775: about 1,800 jobs a day from submodule bumps.
Next action here: read the queue's wait against its run time after #4775 lands, before tuning any more validators.
The dependency grouping in monorepo#4775 has shipped. A first read of two successful overnight protected runs shows detection starting 5 and 16 seconds after run creation, versus 983 seconds in the earlier busy-period sample.
These are different source revisions and off-peak samples, so they do not prove a causal speedup from grouping or replace the planned November 9 observation. Failed deploy samples remain adverse evidence. The complete Go step is still about 10Β½ minutes; the latest native output attributes 493.878s to the existing credential-refresh regression package. No safety check, worker limit or execution capacity changed in this follow-up.
The next measurement should distinguish runner acquisition, prerequisite/concurrency waiting and execution before selecting another optimization. This issue remains open for comparable delivery and reliability evidence.
The performance changes are consolidated in #5841 to require one protected queue deployment. The isolated bridge step was observed at 13s after duplicate removal versus 408s before; the two complete Go partitions passed natively with test-step times of 325s and 193s. A companion unpartitioned run took 691s. Runner conditions varied, so these are validation-step observations, not a controlled end-to-end throughput result.
Delivery remains unfinished: the combined head is in fresh CI and review has not been requested while its required checks are unsettled. GitHub's current primary-language classification admits the organization Go workflow, exposing broader repository-scan failures tracked in #5843. #5846 supplies a locally validated first compatibility repair; the existing #5785 owns lint/dead-code fixes and is already in the protected queue. #5838 was closed only because its complete change is now in #5841.
Keep this issue open until the required baseline is repaired, current-head CI/review pass, the protected merge and default-branch verification finish, and comparable whole-queue measurements demonstrate the intended outcome.
Measurement on the actual queue, 12 November: validation is faster, the whole run is not. This stays open.
The split Go regression jobs (bridge and helpers) are running on main. Four successful merge-queue CI runs from this morning, all at 33 jobs. Times are in seconds and are measured the same way as the baseline, from the first job start to the last job end:
Compared with the baseline (whole run 1,684β1,751, slowest validation job 557β709, deploy 989β1,044):
Validation: the slowest validation job now takes about 390β490 seconds, against 557β709 before. Every check still runs, and the required-checks aggregate still runs at the end.
Whole run: the median is about 1,660 seconds, so there is no material improvement. Most of the difference is time spent waiting: in two of the four runs, the job that waits for the production deployment turn used 755 and 1,351 seconds while earlier queue entries deployed. Deploy time itself varied from 630 to 1,144 seconds.
main is green at 4bad35f9, and the live platform health check passes.
What is left belongs with the deploy turn and the deploy, not with validation. The total cannot reach its goal while entries wait for each other's production deploys one at a time. The next slice needs to decide whether that wait and the deploy duration are in this issue's scope or need their own issue. These are four runs from one morning, not a controlled comparison.
Comparable measurement on the actual queue, 12 November afternoon: the whole run is about 24% faster, and validation and the deploy are both faster. This closes the issue. The deploy-turn wait moves to #6139.
Twelve successful merge-queue CI runs from 10:49 to 18:37 UTC that deployed to production, none of them in the morning sample above. All have 33 jobs. Times are in seconds, from the first job start to the last job end, measured the same way as the baseline:
Against the baseline (whole run 1,684β1,751, slowest validation job 557β709, deploy 989β1,044):
Whole run: median 1,319, range 1,211β1,717. That is about 24% below the baseline median of 1,732. Eleven of the twelve runs beat the fastest baseline run.
Validation: the slowest validation job took 397β501 (median 447), against 557β709.
Deploy: 637β825 (median 749), against 989β1,044.
Why the morning sample read differently. Two things inflated it:
The deploy-turn wait. The wait starts about 20 seconds into the run and runs alongside validation, so it only adds time when it outlasts validation. In the morning, two runs waited 755 and 1,351 seconds, which added 360 and 876 seconds beyond validation. In the afternoon the five runs that waited added 9β396 seconds. This is time a change spends behind other changes' production deploys, not the critical path of the change itself, so it is out of this issue's scope and now has its own issue, Queued changes wait up to 15 minutes after validation for earlier production deploysΒ #6139.
One slow deploy. The 1,144-second deploy in 43199644437 spent 536 seconds in π Stage Flux and consumer GHCR pull credential. Its no-write check stopped at the node inventory, so it ran the full fenced credential transaction. All twelve afternoon runs took the no-write path, in 25β67 seconds. That is 1 fallback in the 14 deploys I read. The node-change case is already tracked in GHCR node credential rollout fails when a node changes during the rolloutΒ #5016.
Every required check, negative control, trust boundary and deployment health check preserved: every job in the baseline run 40639023721 still runs. Three are now split into parallel parts (Publication Generator, Complete Go Regressions and Independent Manifest Regressions), and CI - Required Checks still finishes after the deploy. No check was removed.
Material improvement on comparable runs, validation and total reported separately: the table above.
Protected default-branch run and production recovery:main is green at 3778156c, with Validate Main 43285320900 successful, and the live platform health check passes.
Existing execution capacity only: no runner, worker or paid-capacity setting changed.
These are twelve runs from one afternoon at different source revisions, so they show the queue as it runs today, not a controlled before/after experiment.
Activity
coderabbitai commentedon Oct 28, 2026
π Related PRs
#5366 - perf(ci): partition complete offline regressions [merged]
π Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.π§ͺ Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:π¬ Have feedback or questions? Drop into our discord!
chore: agent-claim #5495 nonce=8e45752493a8c6e6880be2aa6113b347
devantler commentedon Oct 28, 2026
#5510 merged as
db46da4e3c71a26c7310775b0e9d4f4a5204425c. The protected run and post-merge Validate Main passed, including the production convergence check.The bounded publication runner improved the matched local nine-fixture execution from 742s serial to 406s with two isolated workers. In the protected run, the publication validation job took 451s versus baseline job times of 557β709s; its actual fixture runner took 390s. All nine fixtures completed successfully.
The overall readiness result is not yet an improvement. A fresh analysis of the actual workflow dependencies and complete native job timestamps puts the required-gate path at 2,742s, versus 1,657β1,762s for the same ready-to-required-completion measure in the three baseline runs. Its longest observed path was change detection β Helm post-renderer validation β production deployment β required aggregate: 1,594s ready-to-start waiting, 1,147s execution, plus a one-second transition. The post-renderer job alone waited 865s before its 192s execution. Deployment waiting can also include the protected concurrency lock; these timestamps do not establish a hosting-capacity cause.
The later queue-membership observer extends the full run to 3,107s but intentionally does not block the required gate or merging. It is excluded from the next readiness optimization target. This issue stays open: preserve all validators and deployment checks while reducing avoidable waiting or repeated execution within existing capacity.
chore: agent-claim #5495 nonce=86e6545272332256da35dd8656ee442d
chore: agent-claim #5495 nonce=453530848e0c147a5695a550c20cd4cf
23 remaining items
devantler commentedon Nov 2, 2026
The measurement workflow has shipped in #5727. Its first main invocation passed all four arms at the same source: Go 4/8 workers took 556.95/520.90 seconds, and publication 2/4 workers took 600.16/354.39 seconds. All nine publication fixtures passed in each arm. The delivery receipt records CPU cost, exact named outcome reconciliation and the stdout-record gaps. One pair does not establish whole-pipeline causality or retire the adverse samples.
The lookup repair I mentioned earlier is on hold: #5739 is already in the protected queue, and the related shared library has another delivery owner. I removed my empty, unmodified repair worktree rather than creating a competing implementation. The failed #5750 publication fixture is retained and has not been blindly rerun.
The main Go profile points to the credential-refresh regression package as the remaining long pole: 466 seconds with four workers and 434 seconds with eight; the next slowest package took 53/63 seconds. Profiling that fixture's process and execution costs is a stronger next candidate than increasing worker limits again. No validation or capacity change is made by this observation. This issue stays open for reliability and outcome verification.
devantler commentedon Nov 2, 2026
Profile of the slowest validator, and why runner wait is the bigger target. Nothing was changed by this reading.
The Go long pole has no sleeps to remove. I traced one case of the
refresh-flux-ghcr-authpackage locally, which is macOS rather than the CI runner, so the proportions are indicative only.kubectlrenew_sync_leaseTestAfterReleaseRegressionDetectsEachAblatedProductionFencepins that check, so it is a safety control and I did not propose removing it. Merging thejqanddatecalls in that function would save about 8% of one job, which is not worth changing a fence-critical script.Waiting for a runner costs far more than this job.
This is one run during a busy period. It shows waiting can dominate, not how often it does. The one wave source I have sized is now monorepo#4775: about 1,800 jobs a day from submodule bumps.
Next action here: read the queue's wait against its run time after #4775 lands, before tuning any more validators.
devantler commentedon Nov 3, 2026
The dependency grouping in monorepo#4775 has shipped. A first read of two successful overnight protected runs shows detection starting 5 and 16 seconds after run creation, versus 983 seconds in the earlier busy-period sample.
These are different source revisions and off-peak samples, so they do not prove a causal speedup from grouping or replace the planned November 9 observation. Failed deploy samples remain adverse evidence. The complete Go step is still about 10Β½ minutes; the latest native output attributes 493.878s to the existing credential-refresh regression package. No safety check, worker limit or execution capacity changed in this follow-up.
The next measurement should distinguish runner acquisition, prerequisite/concurrency waiting and execution before selecting another optimization. This issue remains open for comparable delivery and reliability evidence.
chore: agent-claim #5495 nonce=4b5a0110686fbeccff799f8e554ff289
chore: agent-claim #5495 nonce=7904521eadca3eab9100b91f6a0aa325
devantler commentedon Nov 3, 2026
The performance changes are consolidated in #5841 to require one protected queue deployment. The isolated bridge step was observed at 13s after duplicate removal versus 408s before; the two complete Go partitions passed natively with test-step times of 325s and 193s. A companion unpartitioned run took 691s. Runner conditions varied, so these are validation-step observations, not a controlled end-to-end throughput result.
Delivery remains unfinished: the combined head is in fresh CI and review has not been requested while its required checks are unsettled. GitHub's current primary-language classification admits the organization Go workflow, exposing broader repository-scan failures tracked in #5843. #5846 supplies a locally validated first compatibility repair; the existing #5785 owns lint/dead-code fixes and is already in the protected queue. #5838 was closed only because its complete change is now in #5841.
Keep this issue open until the required baseline is repaired, current-head CI/review pass, the protected merge and default-branch verification finish, and comparable whole-queue measurements demonstrate the intended outcome.
chore: agent-claim #5495 nonce=9c970a57c0aa602a15d88652c5f5593a
devantler commentedon Nov 12, 2026
Measurement on the actual queue, 12 November: validation is faster, the whole run is not. This stays open.
The split Go regression jobs (bridge and helpers) are running on
main. Four successful merge-queue CI runs from this morning, all at 33 jobs. Times are in seconds and are measured the same way as the baseline, from the first job start to the last job end:Compared with the baseline (whole run 1,684β1,751, slowest validation job 557β709, deploy 989β1,044):
mainis green at4bad35f9, and the live platform health check passes.What is left belongs with the deploy turn and the deploy, not with validation. The total cannot reach its goal while entries wait for each other's production deploys one at a time. The next slice needs to decide whether that wait and the deploy duration are in this issue's scope or need their own issue. These are four runs from one morning, not a controlled comparison.
chore: agent-claim #5495 nonce=6f8a0edaa1a89dc6d87ff3c613e870ce
chore: agent-claim #5495 nonce=ae597d7b9e241d37cfe63f4e181773f5
devantler commentedon Nov 12, 2026
Comparable measurement on the actual queue, 12 November afternoon: the whole run is about 24% faster, and validation and the deploy are both faster. This closes the issue. The deploy-turn wait moves to #6139.
Twelve successful merge-queue CI runs from 10:49 to 18:37 UTC that deployed to production, none of them in the morning sample above. All have 33 jobs. Times are in seconds, from the first job start to the last job end, measured the same way as the baseline:
Against the baseline (whole run 1,684β1,751, slowest validation job 557β709, deploy 989β1,044):
Why the morning sample read differently. Two things inflated it:
π Stage Flux and consumer GHCR pull credential. Its no-write check stopped at the node inventory, so it ran the full fenced credential transaction. All twelve afternoon runs took the no-write path, in 25β67 seconds. That is 1 fallback in the 14 deploys I read. The node-change case is already tracked in GHCR node credential rollout fails when a node changes during the rolloutΒ #5016.The acceptance, item by item:
CI - Required Checksstill finishes after the deploy. No check was removed.mainis green at3778156c, with Validate Main 43285320900 successful, and the live platform health check passes.These are twelve runs from one afternoon at different source revisions, so they show the queue as it runs today, not a controlled before/after experiment.