Generalize rancher-upgrade pipeline to a TEST_MODULE selector by floatingman · Pull Request #1027 · rancher/tests

floatingman · 2026-10-22T14:09:37Z

Scoped to #968 (pipeline slice; parent PRD #946). Generalizes the rancher-upgrade pipeline from Longhorn-only phases to a per-module selector, so the same pipeline runs any validation/rancherupgrade suite end to end. This also discharges the #986 review discussion asking for generic post-upgrade stages as modules land (khushboo-rancher's comment on Jenkinsfile.rancher-upgrade.e2e:929).

Stacking note: built on #986 (floatingman:rancherupgrade-longhorn-suite), so the file list includes #986's commits until that PR merges. Merge #986 first, or review this PR by commit range (single commit 4fbcd6a8, a squashed rebuild of the selector work on the reworked #986 tip).

Revision 2026-10-29: #968 now targets a single job that runs every phase in-pipeline (no build job: dispatch to the recurring validation job). The map now feeds the in-pipeline runUpgradePhase invocations inherited from the reworked #986 base; the dispatch mechanism is gone (INDIVIDUAL_JOB removed).

Change

New TEST_MODULE parameter (longhorn | neuvector | fleet | observability | all, default longhorn, preserving current behavior exactly; all runs every module suite in one pipeline via broad per-phase regexes).
The three hardcoded Longhorn -run regexes become an UPGRADE_TEST_MODULES map of per-phase selectors consumed by the in-pipeline runUpgradePhase invocations on the single-job base from Add rancherupgrade package: Longhorn pre/post/chart-upgrade validation #986.
The Setup stage resolves the module before any infrastructure spend: unknown module names fail fast with the valid list, and the log records which regex each phase will run (the Slice 3: Dedicated Rancher-upgrade Jenkins job (JJB) and six-stage Jenkinsfile, community leg #968 acceptance criterion: "visible in CI logs which regex each stage ran").
Stage names stay module-agnostic from Add rancherupgrade package: Longhorn pre/post/chart-upgrade validation #986 (Pre Upgrade Tasks / Upgrade Rancher / Post Upgrade Tasks); the build description now carries the module.
fleet ships inside Rancher with no standalone chart, so its map has no chart phase: the chart-upgrade step is skipped inside Post Upgrade Tasks, and the shared Qase run is closed exactly once for every module (the post{always} fallback still covers failure paths).
TEST_MODULE=all threads both the fleet node SSH key and the observability StackState settings (and pre-deploy-guards the license). Its chart phase runs observability first, then the remaining modules, as two distinctly named phases so per-phase result/log artifacts do not collide in the shared workspace: Longhorn's engine live-upgrades stall a concurrent observability rollout's volume reattach (build 52).
The Longhorn-only pre-upgrade functional coverage (validation/longhorn re-run with skipTearDown) is dropped on the generalized branch: it was module-specific extra load, not part of the per-module phase contract.

Verification

Diff is a single commit touching only validation/Jenkinsfile.rancher-upgrade.e2e; with TEST_MODULE=longhorn the three phase regexes are byte-identical to Add rancherupgrade package: Longhorn pre/post/chart-upgrade validation #986's constants; the dispatch mechanism is gone (INDIVIDUAL_JOB removed) since the phases run in-pipeline via runUpgradePhase on the single-job base.
Jenkins declarative linter validates the Jenkinsfile successfully (staging controller).
Regex values for neuvector/fleet/observability match the -list-verified method names from Add NeuVector pre/post/chart-upgrade validation to rancherupgrade package #1024/Add Fleet GitRepo survival and lockstep validation to rancherupgrade package #1025/Add SUSE Observability pre/post/chart-upgrade validation to rancherupgrade package #1026.
Grep confirms zero remaining references to the old Longhorn constants, INDIVIDUAL_JOB, the dispatch helper, or awsNodeTemplate.
Live regression anchor done: go-pit-rancher-upgrade-job build 62 (staging controller, TEST_MODULE=longhorn, community, runtime-resolved 2.14.3 -> 2.15.3). Setup logged the longhorn phase selectors, all three phases ran in-pipeline via runUpgradePhase against the single shared Qase run 7668 (closed once), per-phase artifacts archived (including the name-keyed results-chart-upgrade-longhorn.*), environment destroyed, and no downstream job was triggered. Remaining: per-module runs for the neuvector/fleet/observability slice branches.

Merge order

#986 -> this PR -> #1024 (neuvector) -> #1025 (fleet) -> #1026 (observability). Once #986 merges, this PR retargets cleanly since its only delta is the pipeline file.

Implements the Longhorn vertical of the Rancher server upgrade validation (rancher#967, parent PRD rancher#946). New validation/rancherupgrade package with a LonghornUpgradeTestSuite whose phase methods run as separate go test invocations selected via -run, one per pipeline stage: - TestLonghornPreUpgrade: create project, install the Longhorn chart at the cluster-scoped catalog latest, create a checksum workload (PVC + nginx deployment with deterministic names) on a Longhorn volume. - TestLonghornPostUpgrade: gate on the management server-version setting reaching rancherUpgradeInput.targetVersion, assert the Longhorn chart survived the server upgrade, verify the checksum workload data. - TestLonghornChartUpgrade: gate on server version, resolve latest from the cluster-scoped catalog, upgrade longhorn + longhorn-crd, assert the installed chart version, verify the checksum workload again. Supporting actions: - actions/clusters: WaitRancherVersion polls the server-version setting until it semver-equals the target, retrying transient read errors. - actions/storage: CreateChecksumWorkload/VerifyChecksumWorkload build and verify a deterministically named PVC/deployment pair whose sha256-verified data file must survive across processes; no session cleanup is registered by design. - actions/charts: UpgradeLonghornChart upgrades longhorn and longhorn-crd with the deletingConfirmationFlag value preserved and OpenAPI validation disabled, waits for Deployed, then gates on deployments and daemonsets. TearDownSuite deliberately skips session.Cleanup and the suite is admin-only: upgrade-subject resources must survive into later phases and the pipeline environment teardown is the cleanup. Both deviations are documented in the package README, along with the token-rotation and catalog-follower-restart notes for the pipeline. Includes schemas/pit_schemas.yaml for the three phase methods. Verified live against a 2.14.3 HA env with a node-driver downstream RKE2 cluster: pre-upgrade PASS (Longhorn 109.3.1+up1.11.2, checksum seeded), server upgraded to 2.15.2 via qa-infra-automation, post-upgrade PASS, chart upgrade PASS (109.3.1+up1.11.2 -> 110.1.0+up1.12.1, checksum intact).

…estrator Dedicated pipeline for the Rancher server upgrade validation (issue rancher#968, PRD rancher#946), community leg: deploy 2.14-latest env, pre-upgrade Longhorn phase, in-place server upgrade via make rancher-upgrade with rotated-token re-injection, post-upgrade and chart-upgrade phases against one shared Qase run, destroy-always with failure-only diagnostics. Versions resolve at runtime via make rancher-resolve-version when the params are empty. Structurally cloned from validation/Jenkinsfile.e2e and the airgap Jenkinsfile call shapes; runs go-pit-validation-job per phase with -runs TestLonghornUpgradeTestSuite/<phase> selectors.

The per-phase executor job can land on a different Jenkins agent than the orchestrator (observed in the first rancher-upgrade run: default-n367q vs default-xj00w). The LB ingress on the tofu-created ephemeral SG only allows the orchestrator's detected public IP, so a phase on another agent could not reach the Rancher API and rancher.NewClient timed out after 22 minutes of retries. awsNodeTemplate carries the three keys the executor's Authorize Security Group stage reads (securityGroup, vpcId, region) so it authorizes its own runner IP on tf-<prefix>-sg before dispatching to the API, matching the shape Jenkinsfile.e2e ships for the same reason.

A declarative pipeline re-applies its parameters{} block to the job definition after each run, which silently overwrote the JJB-configured staging values (fork repos + slice branches) with the hard-coded main defaults on the mower job: the first run deployed from the fork, the next one fell back to upstream main and lost make rancher-resolve-version ('No rule to make target'). Defaulting these five params to their current value when set keeps the per-controller JJB configuration authoritative instead of racing it.

The one-minute bind wait raced a freshly installed Longhorn: the first volume's provisioning (engine-image pull, instance-manager startup, CSI registration) can exceed a minute on a cold cluster, which failed the pre-upgrade phase with 'context deadline exceeded' in the mower run (the same step passed when the volume bound faster). Ten minutes matches the bound used for the rest of the storage waits.

Jenkins evaluates declarative post conditions in a fixed order with always before failure, so the failure-block diagnostics ran after the always block had already run infra-down and docker rmi: kubectl/helm had no rancher-infra-tools image left to run in (pull access denied, exit 125) and the cluster was already gone regardless. Diagnostics now run first inside always, gated on a non-SUCCESS result, while the images and the live cluster still exist; the failure block keeps only the DESTROY_ON_FAILURE fallback destroy.

The post-upgrade phase failed with an instant 401 from rancher.NewClient even though the rotated token had been extracted (length 66) and yq-injected into the dispatched config. Two fail-fast checks now pinpoint which side is broken: the injected token must get HTTP 200 from the upgraded Rancher API before any phase is dispatched, and the config about to be dispatched must byte-match the token in generated.tfvars. Both run under set +x so the token value stays out of the xtrace (the previous run echoed it into the build log).

The smoke-check curl died with exit 60 (TLS verification) against pitupgrade.qa.rancher.space before it ever reached auth — the same endpoint the tests already call with insecure: true. Pass -k so the check actually adjudicates the token.

token=$(awk ...) creates an unexported shell variable, but yq's strenv() reads the process environment, so the injection silently wrote an empty string into rancher.adminToken — the post-upgrade phase then authenticated with an empty bearer token (instant 401). curl in the same step still worked because its header uses shell expansion, which is why the smoke-check passed while the config check failed (expected length 66, got length 0). export the variable so the child yq process can see it.

- logrus milestone logs in each upgrade phase (version resolution, chart install/upgrade, workload seed, checksum verification) so the job log shows where each phase is without reading the assertions backwards. - rancher-infra-tools/rancher-go-test now carry a build-scoped tag (pitupgrade-<BUILD_NUMBER>) used by every direct docker invocation in this pipeline, so this job no longer depends on whatever another job last built or removed under the shared :latest tag on the same agent; the :latest alias is still applied for qa-jenkins-library's make.runTarget image resolution.

Jenkins evaluates post conditions in a fixed order (always before failure, cleanup last), so the failure{} destroy block could never work: always had already shredded workspace_name.txt and tfvars and removed the infra-tools image teardown depends on, and with DESTROY_AFTER_TESTS=false + DESTROY_ON_FAILURE=true a failed run skipped teardown entirely, leaking the AWS environment. Fold the destroy decision into always (after diagnostics and Qase completion), gated on DESTROY_AFTER_TESTS or failure+DESTROY_ON_ FAILURE; move the docker rmi and sensitive-artifact shredding into a cleanup block so they run last.

…de pipeline

…ine marker

…ard, flavor plumbing - actions/storage: reject checksum content outside [A-Za-z0-9-] before sh -c interpolation - rancherupgrade: fail fast when resolved latest equals the installed chart version (stale cluster-scoped catalog follower) - Jenkinsfile.rancher-upgrade.e2e: document timeout budget, plumb RANCHER_CHART_REPO_FLAVOR into make args, resolve kubeconfig via INFRA_DIR, select deploy repo by flavor - README + pit_schemas: correct downstream-cluster wording to the Rancher local cluster - actions/clusters: debug logging on server-version read/parse failures

…ersion scope - pvc.go: createNginxDeploymentWithPVC takes an explicit deploymentName, so CreatePVCWorkload keeps random names while CreateChecksumWorkload passes its deterministic one; shared construction replaces ~25 duplicated lines - checksum.go: reuse the shared helper; drop now-unused charts/wloads imports - README: previous-release-line-to-latest wording (pipeline default, not a suite constraint) and a deliberate-deviation bullet for the out-of-band server upgrade

… pre-upgrade - validation/Jenkinsfile.rancher-upgrade.e2e: 11 top-level stages -> 3 high-level phases (Setup, Deploy, Upgrade Validation) per the Jenkins Pipeline Syntax Guide; per-phase work (deploy tasks, test dispatches) becomes steps; the four test dispatches share one dispatchUpgradePhase helper so the stage list stays constant as modules are added - new pre-upgrade functional-coverage step reuses the primary validation/longhorn suite (LONGHORN_FUNCTIONAL_COVERAGE, default on) with a -run selector that excludes TestVolumeEncryption (needs cloud credentials the default config lacks); timeout budget 300 -> 360 minutes - validation/longhorn: TearDownSuite skips session.Cleanup when the config carries the rancherUpgradePipeline marker, so the suite's resources survive the server upgrade; normal runs still clean up - actions/clusters: fold the version-equality check into the poll return - actions/storage: fmt.Sprintf instead of string concatenation

The .golangci.yaml lens builds with tags validation,2.13, which the suite files exclude via !2.13; the unexported helpers moved into rancherupgrade.go read as unused in that view. Carrying the suite tag keeps the package's files in one build unit.

lscalabrini01 · 2026-10-27T21:05:16Z

+    neuvector: [
+        pre:   '-run "TestNeuVectorUpgradeTestSuite/TestNeuVectorPreUpgrade"',
+        post:  '-run "TestNeuVectorUpgradeTestSuite/TestNeuVectorPostUpgrade"',
+        chart: '-run "TestNeuVectorUpgradeTestSuite/TestNeuVectorChartUpgrade"'
+    ],
+    fleet: [
+        pre:  '-run "TestFleetUpgradeTestSuite/TestFleetPreUpgrade"',
+        post: '-run "TestFleetUpgradeTestSuite/TestFleetPostUpgrade"'
+    ],
+    observability: [
+        pre:   '-run "TestObservabilityUpgradeTestSuite/TestObservabilityPreUpgrade"',
+        post:  '-run "TestObservabilityUpgradeTestSuite/TestObservabilityPostUpgrade"',
+        chart: '-run "TestObservabilityUpgradeTestSuite/TestObservabilityChartUpgrade"'
+    ],


We do not have these tests on this branch

We will once all related PRs are merged.

…ding

…atency

Longhorn 1.12's manager gathers volume metrics live per /metrics request, so the endpoint takes 30s or more and every Prometheus scrape times out even with a 60s scrapeTimeout (live-verified across three fresh pipeline environments, runs 56-59). Its series never appear, so the monitoring assertions cannot pass in this pipeline until the exporter latency is fixed upstream. Re-enable via LONGHORN_FUNCTIONAL_TESTCASE.

lscalabrini01

Looks good, but we should wait for the related PRs be merged

…ants Review follow-ups (khushboo-rancher): - Split the single validation stage into Pre Upgrade Tasks / Upgrade Rancher / Post Upgrade Tasks and rename the deploy stage, matching the reviewed six-phase list; the shared Qase run id now crosses the stage boundary via env.SHARED_TEST_RUN_ID - Drop RANCHER_BOOTSTRAP_PASSWORD / RANCHER_ADMIN_PASSWORD and S3_BUCKET_NAME / S3_BUCKET_REGION / S3_KEY_PREFIX as pipeline parameters; both password substitutions resolve to DEFAULT_RANCHER_PASSWORD and the admin password still reaches the token-generation container through the environment block - Rename validation/rancherupgrade/longhorn_test.go to longhorn_upgrade_test.go; the sibling module suites rename their own files in their PRs

…ncher#968) The dispatch to go-pit-validation-job is removed: every phase now runs gotestsum in the pinned go-test image directly in this pipeline, modeled on validation/Jenkinsfile.individual.e2e's Run Tests + Report Results stages. The awsNodeTemplate cross-agent seam in CATTLE_TEST_CONFIG is dropped (phases run on the deploy agent whose IP the deploy already authorized), the INDIVIDUAL_JOB parameter is removed, and per-phase junit/artifact archiving is added to post-always. The shared Qase run stays started once and closed once by the pipeline itself.

…them runUpgradePhase and collectUpgradeDiagnostics are separate Groovy methods; script-level def variables are locals of the implicit run() body and are invisible to them. Live-verified by the first in-pipeline run (build 61 on go-pit-rancher-upgrade-job): the pre-upgrade phase failed with MissingPropertyException: No such property: GO_TEST_IMAGE, and the failure diagnostics hit the same error on INFRA_TOOLS_IMAGE. Binding variables (no def) are visible to both stage bodies and helper methods.

A TEST_MODULE parameter (longhorn | neuvector | fleet | observability | all) now selects the per-phase -run regexes from an UPGRADE_TEST_MODULES map, resolved in the Setup stage with a fail-fast on unknown modules and a log line recording the selected regexes. The selector drives the in-pipeline runUpgradePhase invocations on the single-job base from rancher#986 (squashed rebuild of the previous 6 commits; rancher#968). Module-specific plumbing rides the same flow: fleet threads its node SSH key/user through the test config (its GitRepo phase serves SSH from a downstream node), observability threads StackState connection settings and guards the STACKSTATE_LICENSE credential before any infra spend, and fleet skips the chart-upgrade phase (it ships inside Rancher with no standalone chart), with the shared Qase run still closed exactly once. TEST_MODULE=all runs every module suite in one pipeline via broad per-phase regexes (threading both the fleet key and the observability license, and pre-deploy-guarding the license); its chart phase runs the observability upgrade before the rest, each as a distinctly named phase so per-phase result/log artifacts do not collide: Longhorn's engine live-upgrades stall a concurrent observability rollout's volume reattach (build 52). The Longhorn-only pre-upgrade functional coverage (validation/longhorn re-run with skipTearDown) is dropped on the generalized branch: it was module-specific extra load, not part of the per-module phase contract.

floatingman added 15 commits October 7, 2026 21:41

Address PR review: pit.weekly build tag, skip suite outside the upgra…

84bac3d

…de pipeline

Address PR review: fail on pipeline misconfig via rancherUpgradePipel…

0848729

…ine marker

floatingman requested a review from a team as a code owner October 22, 2026 14:09

floatingman self-assigned this Oct 26, 2026

floatingman requested review from hamistao, khushboo-rancher and lscalabrini01 October 26, 2026 14:12

floatingman added the team/pit-crew slack notifier for pit crew label Oct 26, 2026

floatingman removed the request for review from a team October 27, 2026 12:00

floatingman added 2 commits October 27, 2026 13:48

Move checksum constants and loadUpgradeInput to rancherupgrade.go

f69d6f4

lscalabrini01 reviewed Oct 27, 2026

View reviewed changes

floatingman added 4 commits October 27, 2026 20:20

Address review: skipTearDown config switch, n-1 and S3-credential wor…

15d8f21

…ding

Stabilize TestMonitoringIntegration: poll Prometheus through scrape l…

21309f0

…atency

Raise longhorn ServiceMonitor scrape timeout for 1.12 slow /metrics

85656a7

lscalabrini01 previously approved these changes Oct 28, 2026

View reviewed changes

This was referenced Oct 29, 2026

Add rancherupgrade package: Longhorn pre/post/chart-upgrade validation #986

Open

PRD: Rancher server upgrade tests (2.14 → latest) for interop products: Longhorn, Fleet, NeuVector, SUSE Observability, Harvester #946

Open

floatingman dismissed lscalabrini01’s stale review via dcfd17d October 29, 2026 19:50

floatingman force-pushed the feat/rancher-upgrade-module-selector branch 3 times, most recently from c67f600 to ad09b8b Compare October 29, 2026 19:52

floatingman force-pushed the feat/rancher-upgrade-module-selector branch 2 times, most recently from 6c185f9 to 4fbcd6a Compare October 29, 2026 20:27

lscalabrini01 approved these changes Oct 30, 2026

View reviewed changes

Navigation Menu

Repository navigation

Generalize rancher-upgrade pipeline to a TEST_MODULE selector - #1027

Generalize rancher-upgrade pipeline to a TEST_MODULE selector#1027
floatingman wants to merge 26 commits into
rancher:mainfrom
floatingman:feat/rancher-upgrade-module-selector

floatingman commented Oct 22, 2026 •

edited

Loading

Uh oh!

lscalabrini01 Oct 27, 2026

Uh oh!

floatingman Oct 27, 2026

Uh oh!

lscalabrini01 left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

floatingman commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change

Verification

Merge order

Uh oh!

lscalabrini01 Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

floatingman Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

lscalabrini01 left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

floatingman commented Oct 22, 2026 •

edited

Loading