Skip to content

Reduce CI critical-path time while preserving validationΒ #5495

Closed
5 of 5 issues completed
Closed

Description

@devantler

πŸ€– Generated by the Agentic Engineer

Problem

Recent successful merge-queue runs take 28–29 minutes, delaying delivery and recovery while each change waits for the next deployment.

Baseline

Runs 40639023721, 40641879084 and 40644713318 took 1,732, 1,684 and 1,751 seconds from first job start to last completion. Deployment took 989–1,044 seconds. The publication-generator validation job took 557–709 seconds, and repository guards took 528–608 seconds. These are measurements, not a demonstrated optimization.

Acceptance

  • Measure individual validation helpers and the dependency path before choosing a change. Identify redundant work or independent, isolated fixtures that can run concurrently with bounded resource use.
  • Preserve every required check, negative control, trust boundary and deployment health check. Failures, timeouts and incomplete background work must still fail the required aggregate.
  • Keep logs attributable to each helper and retain an equivalent serial path for comparison and diagnosis.
  • Demonstrate a material improvement on comparable runs, report validation and total critical-path time separately, then verify the protected default-branch run and production recovery.
  • Use existing execution capacity; changing paid capacity or financial settings is outside this change.

Part of #2043

Activity

  1. self-assigned this
    on Oct 28, 2026
  2. coderabbitai commented on Oct 28, 2026

    @coderabbitai
    Contributor
    πŸ”— Related PRs

    #5366 - perf(ci): partition complete offline regressions [merged]


    πŸ“ Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    πŸ§ͺ Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    πŸ’¬ Have feedback or questions? Drop into our discord!

  3. moved this to πŸ“₯ Backlog in 🌊 Project Boardon Oct 28, 2026
  4. moved this from πŸ“₯ Backlog to πŸƒπŸ»β€β™‚οΈ In Progress in 🌊 Project Boardon Oct 28, 2026
  5. added a commit that references this issue on Oct 28, 2026
  6. devantler commented on Oct 28, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    #5510 merged as db46da4e3c71a26c7310775b0e9d4f4a5204425c. The protected run and post-merge Validate Main passed, including the production convergence check.

    The bounded publication runner improved the matched local nine-fixture execution from 742s serial to 406s with two isolated workers. In the protected run, the publication validation job took 451s versus baseline job times of 557–709s; its actual fixture runner took 390s. All nine fixtures completed successfully.

    The overall readiness result is not yet an improvement. A fresh analysis of the actual workflow dependencies and complete native job timestamps puts the required-gate path at 2,742s, versus 1,657–1,762s for the same ready-to-required-completion measure in the three baseline runs. Its longest observed path was change detection β†’ Helm post-renderer validation β†’ production deployment β†’ required aggregate: 1,594s ready-to-start waiting, 1,147s execution, plus a one-second transition. The post-renderer job alone waited 865s before its 192s execution. Deployment waiting can also include the protected concurrency lock; these timestamps do not establish a hosting-capacity cause.

    The later queue-membership observer extends the full run to 3,107s but intentionally does not block the required gate or merging. It is excluded from the next readiness optimization target. This issue stays open: preserve all validators and deployment checks while reducing avoidable waiting or repeated execution within existing capacity.

  7. added 2 commits that reference this issue on Oct 29, 2026
  8. 23 remaining items

  9. devantler commented on Nov 2, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    The measurement workflow has shipped in #5727. Its first main invocation passed all four arms at the same source: Go 4/8 workers took 556.95/520.90 seconds, and publication 2/4 workers took 600.16/354.39 seconds. All nine publication fixtures passed in each arm. The delivery receipt records CPU cost, exact named outcome reconciliation and the stdout-record gaps. One pair does not establish whole-pipeline causality or retire the adverse samples.

    The lookup repair I mentioned earlier is on hold: #5739 is already in the protected queue, and the related shared library has another delivery owner. I removed my empty, unmodified repair worktree rather than creating a competing implementation. The failed #5750 publication fixture is retained and has not been blindly rerun.

    The main Go profile points to the credential-refresh regression package as the remaining long pole: 466 seconds with four workers and 434 seconds with eight; the next slowest package took 53/63 seconds. Profiling that fixture's process and execution costs is a stronger next candidate than increasing worker limits again. No validation or capacity change is made by this observation. This issue stays open for reliability and outcome verification.

  10. devantler commented on Nov 2, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    Profile of the slowest validator, and why runner wait is the bigger target. Nothing was changed by this reading.

    The Go long pole has no sleeps to remove. I traced one case of the refresh-flux-ghcr-auth package locally, which is macOS rather than the CI runner, so the proportions are indicative only.

    Measure, one case Reading
    Wall time 8.05 s, of which 6.66 s was CPU
    Fake commands started 561, of which 502 were kubectl
    Startup of one fake command about 4 ms, the same as an empty Go program
    Share of traced time in renew_sync_lease about 35%, from 121 calls
    • The fake commands are not the cost, and the test's sync interval is already 0.
    • The case is CPU-bound, which fits 8 workers barely beating 4 on these runners (434 s against 466 s).
    • Each lease renewal reads the lease and then patches it with a resource-version check. TestAfterReleaseRegressionDetectsEachAblatedProductionFence pins that check, so it is a safety control and I did not propose removing it. Merging the jq and date calls in that function would save about 8% of one job, which is not worth changing a fence-critical script.

    Waiting for a runner costs far more than this job.

    Run 41500025982, with deploy Reading
    Created to last job finished 4,564 s
    Longest chain of job run times about 1,200 s
    First job, Detect Changes waited 981 s, ran 22 s
    Deploy to Prod waited 366 s, ran 729 s
    Final 7-second job waited 782 s

    This is one run during a busy period. It shows waiting can dominate, not how often it does. The one wave source I have sized is now monorepo#4775: about 1,800 jobs a day from submodule bumps.

    Next action here: read the queue's wait against its run time after #4775 lands, before tuning any more validators.

  11. devantler commented on Nov 3, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    The dependency grouping in monorepo#4775 has shipped. A first read of two successful overnight protected runs shows detection starting 5 and 16 seconds after run creation, versus 983 seconds in the earlier busy-period sample.

    Protected run Creation to last job completion Complete Go regression step Production deploy step
    41579796675 1,942s 635s 721s
    41584234329 2,208s 632s 1,153s

    These are different source revisions and off-peak samples, so they do not prove a causal speedup from grouping or replace the planned November 9 observation. Failed deploy samples remain adverse evidence. The complete Go step is still about 10Β½ minutes; the latest native output attributes 493.878s to the existing credential-refresh regression package. No safety check, worker limit or execution capacity changed in this follow-up.

    The next measurement should distinguish runner acquisition, prerequisite/concurrency waiting and execution before selecting another optimization. This issue remains open for comparable delivery and reliability evidence.

  12. added 2 commits that reference this issue on Nov 3, 2026
  13. devantler commented on Nov 3, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    The performance changes are consolidated in #5841 to require one protected queue deployment. The isolated bridge step was observed at 13s after duplicate removal versus 408s before; the two complete Go partitions passed natively with test-step times of 325s and 193s. A companion unpartitioned run took 691s. Runner conditions varied, so these are validation-step observations, not a controlled end-to-end throughput result.

    Delivery remains unfinished: the combined head is in fresh CI and review has not been requested while its required checks are unsettled. GitHub's current primary-language classification admits the organization Go workflow, exposing broader repository-scan failures tracked in #5843. #5846 supplies a locally validated first compatibility repair; the existing #5785 owns lint/dead-code fixes and is already in the protected queue. #5838 was closed only because its complete change is now in #5841.

    Keep this issue open until the required baseline is repaired, current-head CI/review pass, the protected merge and default-branch verification finish, and comparable whole-queue measurements demonstrate the intended outcome.

  14. added a commit that references this issue on Nov 10, 2026
  15. devantler commented on Nov 12, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    Measurement on the actual queue, 12 November: validation is faster, the whole run is not. This stays open.

    The split Go regression jobs (bridge and helpers) are running on main. Four successful merge-queue CI runs from this morning, all at 33 jobs. Times are in seconds and are measured the same way as the baseline, from the first job start to the last job end:

    Run Whole run Waiting for the deploy turn Slowest validation job Deploy to Prod
    43194875644 1,301 17 394 (Independent Guards) 860
    43188519865 1,645 755 416 (Publication Generator, upgrade) 661
    43199644437 1,675 β€” under 490 1,144
    43200234933 2,150 1,351 463 (Nested RGD Workload Templates) 630

    Compared with the baseline (whole run 1,684–1,751, slowest validation job 557–709, deploy 989–1,044):

    • Validation: the slowest validation job now takes about 390–490 seconds, against 557–709 before. Every check still runs, and the required-checks aggregate still runs at the end.
    • Whole run: the median is about 1,660 seconds, so there is no material improvement. Most of the difference is time spent waiting: in two of the four runs, the job that waits for the production deployment turn used 755 and 1,351 seconds while earlier queue entries deployed. Deploy time itself varied from 630 to 1,144 seconds.

    main is green at 4bad35f9, and the live platform health check passes.

    What is left belongs with the deploy turn and the deploy, not with validation. The total cannot reach its goal while entries wait for each other's production deploys one at a time. The next slice needs to decide whether that wait and the deploy duration are in this issue's scope or need their own issue. These are four runs from one morning, not a controlled comparison.

  16. added 2 commits that reference this issue on Nov 12, 2026
  17. devantler commented on Nov 12, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    Comparable measurement on the actual queue, 12 November afternoon: the whole run is about 24% faster, and validation and the deploy are both faster. This closes the issue. The deploy-turn wait moves to #6139.

    Twelve successful merge-queue CI runs from 10:49 to 18:37 UTC that deployed to production, none of them in the morning sample above. All have 33 jobs. Times are in seconds, from the first job start to the last job end, measured the same way as the baseline:

    Run Whole run Waiting for the deploy turn Slowest validation job Deploy to Prod
    43203738969 1,517 677 445 (Repository Guards) 723
    43205453155 1,633 792 417 (Independent Guards) 739
    43208254811 1,211 489 397 (Nested RGD Workload Templates) 637
    43231673259 1,231 23 445 (Publication Generator, upgrade) 696
    43233842713 1,291 409 422 (Publication Generator, bootstrap) 793
    43238645330 1,240 18 499 (Nested RGD Workload Templates) 666
    43243669384 1,349 22 491 (Independent Guards) 810
    43246082229 1,454 535 501 (Nested RGD Workload Templates) 822
    43252956516 1,346 23 448 (Repository Guards) 790
    43259687052 1,287 24 412 (Independent Guards) 825
    43275915239 1,717 20 479 (Nested RGD Workload Templates) 641
    43281734009 1,269 17 458 (Repository Guards) 759

    Against the baseline (whole run 1,684–1,751, slowest validation job 557–709, deploy 989–1,044):

    • Whole run: median 1,319, range 1,211–1,717. That is about 24% below the baseline median of 1,732. Eleven of the twelve runs beat the fastest baseline run.
    • Validation: the slowest validation job took 397–501 (median 447), against 557–709.
    • Deploy: 637–825 (median 749), against 989–1,044.

    Why the morning sample read differently. Two things inflated it:

    • The deploy-turn wait. The wait starts about 20 seconds into the run and runs alongside validation, so it only adds time when it outlasts validation. In the morning, two runs waited 755 and 1,351 seconds, which added 360 and 876 seconds beyond validation. In the afternoon the five runs that waited added 9–396 seconds. This is time a change spends behind other changes' production deploys, not the critical path of the change itself, so it is out of this issue's scope and now has its own issue, Queued changes wait up to 15 minutes after validation for earlier production deploysΒ #6139.
    • One slow deploy. The 1,144-second deploy in 43199644437 spent 536 seconds in πŸ”‘ Stage Flux and consumer GHCR pull credential. Its no-write check stopped at the node inventory, so it ran the full fenced credential transaction. All twelve afternoon runs took the no-write path, in 25–67 seconds. That is 1 fallback in the 14 deploys I read. The node-change case is already tracked in GHCR node credential rollout fails when a node changes during the rolloutΒ #5016.

    The acceptance, item by item:

    • Measure helpers and the dependency path before choosing a change: the profiles of 2 and 3 November and the measurement workflow from ci: add comparable validation runtime measurementsΒ #5727.
    • Every required check, negative control, trust boundary and deployment health check preserved: every job in the baseline run 40639023721 still runs. Three are now split into parallel parts (Publication Generator, Complete Go Regressions and Independent Manifest Regressions), and CI - Required Checks still finishes after the deploy. No check was removed.
    • Logs attributable to each helper, with an equivalent serial path: ci: add comparable validation runtime measurementsΒ #5727 keeps serial and parallel arms of the Go and publication runners for comparison.
    • Material improvement on comparable runs, validation and total reported separately: the table above.
    • Protected default-branch run and production recovery: main is green at 3778156c, with Validate Main 43285320900 successful, and the live platform health check passes.
    • Existing execution capacity only: no runner, worker or paid-capacity setting changed.

    These are twelve runs from one afternoon at different source revisions, so they show the queue as it runs today, not a controlled before/after experiment.

  18. moved this from πŸƒπŸ»β€β™‚οΈ In Progress to βœ… Done in 🌊 Project Boardon Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Fields

No fields configured for Performance.

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions