Repository navigation
Fix Longhorn tests - #1029
Fix Longhorn tests#1029hamistao wants to merge 13 commits into
Conversation
Signed-off-by: hamistao <pedro.ribeiro@suse.com>
Signed-off-by: hamistao <pedro.ribeiro@suse.com>
Signed-off-by: hamistao <pedro.ribeiro@suse.com>
Signed-off-by: hamistao <pedro.ribeiro@suse.com>
Signed-off-by: hamistao <pedro.ribeiro@suse.com>
Signed-off-by: hamistao <pedro.ribeiro@suse.com>
Signed-off-by: hamistao <pedro.ribeiro@suse.com>
Signed-off-by: hamistao <pedro.ribeiro@suse.com>
Signed-off-by: hamistao <pedro.ribeiro@suse.com>
Signed-off-by: hamistao <pedro.ribeiro@suse.com>
Signed-off-by: hamistao <pedro.ribeiro@suse.com>
khushboo-rancher
left a comment
There was a problem hiding this comment.
In general LGTM, just one question.
| "networkPolicies": map[string]any{ | ||
| "restrictInternalTraffic": false, |
There was a problem hiding this comment.
Why do we need this setting? Is it something mandatory for Longhorn chart now?
There was a problem hiding this comment.
This is the core of this PR's fix. It allows us to install Longhorn without network policies that prevents Prometheus from scraping data from the longhorn-system namespace. Creating the namespaces became the default since 1.12.
lscalabrini01
left a comment
There was a problem hiding this comment.
LGTM. one question
| return | ||
| } | ||
|
|
||
| for _, policy := range policies.Data { |
There was a problem hiding this comment.
Should us verify if the policies.Data is empty?
There was a problem hiding this comment.
@lscalabrini01 Not sure I get the point. If it is empty then we just don't issue any warning.
floatingman
left a comment
There was a problem hiding this comment.
Review verdict: Comment, nothing blocking.
Before flagging anything, the risky-looking parts were verified against upstream sources (longhorn-manager, shepherd, rancher/charts):
- Cleanup fix (
backup.NameviabackupController): the old code deleted a Backup name that never existed (snapshot and backup names are independently randomized, guaranteed 404 at teardown) while leaking the real Backup CR; the snapshot keeps its own separate cleanup. Strict fix. backup.Status.Labelsas the encrypted-flag source: the longhorn-manager backup controller propagatesvolume-encrypted: enabledintoStatus.Labelsin the same sync that setsVolumeName, so it is populated once the backup waits complete. ObjectMeta labels orSpec.Labelswould have been the wrong source here.- Restore fail-fast: the failure condition is checked before the done-return on every poll iteration,
RestoreFailureis only set on the faulted path (in-progress usesRestoreInProgress), and the poll aborts on first error, so no false positives and no missed terminal failures. - Timeout bumps (10s interval, 15m for the BackupVolume to appear on a fresh S3 target): proportionate to repo wait conventions.
- ServiceMonitor move to
RancherMonitoringNamespace: selection-neutral (the chart's serviceMonitor selectors are{}and the spec keepsNamespaceSelector: [longhorn-system]), RBAC holds for the cluster-owner standard user, cleanup follows the same variable. - NetworkPolicy warning helper: premise matches the chart (the
longhorn-managerpolicy exists iffrestrictInternalTrafficis on, and it blocks port 9500);List(nil)and the steve type are safe.
Also checked and clean: const adoption complete and value-preserving, v1 alias and map[string]any changes are no-ops, uninstall payload flatten equivalent, empty workload lists return immediately so component-disabled installs will not hang.
Five minor/nit findings inline; one open question that has no inline anchor:
Upgrade path: only InstallRancherMonitoringChart gains the workload-readiness waits; UpgradeRancherMonitoringChart still returns once App state is deployed. If the Prometheus-not-yet-scraping race can also bite after an upgrade, the same three waits belong there. If install-only coverage is deliberate, no action needed.
| } | ||
|
|
||
| _, ok = defaultSettings["deletingConfirmationFlag"] | ||
| defaultSettings["deletingConfirmationFlag"] = true |
There was a problem hiding this comment.
minor: The comment above ("If no specific value for setting deletingConfirmationFlag was provided...") no longer matches the code: this now sets the flag unconditionally, silently overriding a caller that explicitly passed deletingConfirmationFlag: false, and mutating their map in place. If the force is intended, the comment should say so:
| defaultSettings["deletingConfirmationFlag"] = true | |
| // Force deletingConfirmationFlag so chart cleanup always works, and force | |
| // restrictInternalTraffic off so it cannot block Prometheus scraping. | |
| defaultSettings["deletingConfirmationFlag"] = true |
If caller intent should win, the conditional set was lost in the edit: wrap it in if _, ok := defaultSettings["deletingConfirmationFlag"]; !ok { ... }.
| networkPolicies, ok := values["networkPolicies"].(map[string]any) | ||
| if !ok { | ||
| defaultSettings["deletingConfirmationFlag"] = true | ||
| networkPolicies = map[string]any{} |
There was a problem hiding this comment.
minor: Asymmetric validation between the two forced values. A wrong-typed defaultSettings returns an explicit error, but a wrong-typed networkPolicies (e.g. a string or map[string]string) is silently discarded and replaced with an empty map, so the caller's value vanishes with no signal. The sibling keys should behave the same:
| networkPolicies = map[string]any{} | |
| networkPolicies, ok := values["networkPolicies"].(map[string]any) | |
| if !ok { | |
| if _, exists := values["networkPolicies"]; exists { | |
| return errors.New(`Provided values map has invalid value for "networkPolicies"`) | |
| } | |
| networkPolicies = map[string]any{} | |
| values["networkPolicies"] = networkPolicies | |
| } |
| // Wait for the monitoring workloads to be Ready so the chart is actually serving | ||
| // (Prometheus and Alertmanager run as StatefulSets; the operator and Grafana as Deployments). | ||
| // Without this, callers can race against Prometheus not yet scraping any targets. | ||
| err = shepherdCharts.WatchAndWaitDeployments(client, installOptions.Cluster.ID, RancherMonitoringNamespace, metav1.ListOptions{}) |
There was a problem hiding this comment.
minor: All three waits pass bare metav1.ListOptions{}, so they gate on every Deployment, DaemonSet, and StatefulSet in cattle-monitoring-system, not just this release. Any unrelated or leftover workload in that namespace (failed prior cleanup, another chart's components) that never becomes ready will stall the install until the per-workload watch timeout. Scoping to the release costs one line and makes the wait honest:
| err = shepherdCharts.WatchAndWaitDeployments(client, installOptions.Cluster.ID, RancherMonitoringNamespace, metav1.ListOptions{}) | |
| releaseSelector := metav1.ListOptions{LabelSelector: "app.kubernetes.io/instance=" + RancherMonitoringName} | |
| err = shepherdCharts.WatchAndWaitDeployments(client, installOptions.Cluster.ID, RancherMonitoringNamespace, releaseSelector) |
(and the same selector for the DaemonSet and StatefulSet calls).
| return err | ||
| } | ||
|
|
||
| return shepherdCharts.WatchAndWaitStatefulSets(client, installOptions.Cluster.ID, RancherMonitoringNamespace, metav1.ListOptions{}) |
There was a problem hiding this comment.
minor: This readiness gate is weaker than it looks: shepherd's WatchAndWaitDeployments/DaemonSets/StatefulSets discard WatchWait's return value, so a workload that never becomes ready produces a silent nil after the watch timeout rather than an error. The new code guarantees "waited up to the timeout", not "ready". Worth a one-line call-site comment (and ideally a shepherd fix) so future callers are not lulled by the names:
| return shepherdCharts.WatchAndWaitStatefulSets(client, installOptions.Cluster.ID, RancherMonitoringNamespace, metav1.ListOptions{}) | |
| // Note: shepherd WatchAndWait* swallow per-workload watch timeouts; a stuck | |
| // workload delays but does not fail this install. | |
| return shepherdCharts.WatchAndWaitStatefulSets(client, installOptions.Cluster.ID, RancherMonitoringNamespace, releaseSelector) |
| @@ -53,7 +53,7 @@ func chartActionTestOpts() *PayloadOpts { | |||
| return &PayloadOpts{ | |||
| InstallOptions: InstallOptions{Cluster: &clusters.ClusterMeta{ID: "c-test-local"}}, | |||
There was a problem hiding this comment.
nit: This is the one remaining nested InstallOptions: InstallOptions{...} construction; the PR flattens the same shape everywhere else (e.g. UninstallLonghornChart). Both compile, purely a consistency follow-through of the PR's own flattening.
Signed-off-by: hamistao <pedro.ribeiro@suse.com>
Signed-off-by: hamistao <pedro.ribeiro@suse.com>
There was a problem hiding this comment.
🟡 Changes recommended
Multiple critical compile errors and moderate test logic issues must be corrected before approval.
10 open issues
Initialize embedded InstallOptions for Cluster · New Initialize Secret metadata through ObjectMeta · New Initialize PayloadOpts fields through InstallOptions · New Initialize Service metadata through ObjectMeta · New Initialize PayloadOpts through embedded InstallOptions · New Initialize PayloadOpts fields through InstallOptions · New Initialize Secret metadata through ObjectMeta · New Invert alerting chart installation skip condition · New Refresh catalog before resolving monitoring version · New Refresh catalog before resolving chart version · New
What changed in this PR
This pull request fixes Longhorn encryption and monitoring integration tests by improving encrypted backup restoration, chart installation behavior, repository refreshes, cleanup, and workload readiness checks.
Changes:
- Restores encrypted Longhorn backups into encrypted volumes with improved error handling and cleanup.
- Disables restrictive Longhorn network policies for new installations and improves monitoring setup.
- Adds chart repository refresh support and monitoring workload readiness waits.
- Updates related chart installation tests and diagnostics.
| File | Reviewed changes and findings |
|---|---|
validation/longhorn/longhorn_test.go |
Updates Longhorn installation, encrypted backup restoration, and monitoring integration. Critical (2 votes): PayloadOpts uses promoted embedded fields in a composite literal. Critical (2 votes): corev1.Secret uses promoted metadata fields directly. |
validation/longhorn/chartinstall/installation_test.go |
Removes an outdated replica-default comment. |
validation/charts/neuvector_test.go |
Refreshes the chart repository before installation. Critical (2 votes): PayloadOpts uses promoted embedded fields directly. Moderate (2 votes): the repository refresh occurs after the chart version is resolved. |
validation/charts/monitoring_test.go |
Refreshes the repository and updates monitoring resources. Moderate (2 votes): refresh occurs after the version lookup. Critical (2 votes): corev1.Service uses promoted metadata fields directly. |
validation/charts/logging_test.go |
Refreshes the repository and updates Longhorn payload construction. Critical (2 votes): promoted embedded fields are used directly in a PayloadOpts literal. |
validation/charts/installations_test.go |
Refreshes the Rancher chart repository during suite setup. |
validation/charts/alerting_test.go |
Adds conditional Alerting installation. Moderate (4 votes): the installation guard is inverted, preventing installation on a clean cluster. |
actions/kubeapi/longhorn/longhorn.go |
Improves backup cleanup, encrypted-volume restoration, and restore polling. Critical (2 votes): corev1.Secret uses promoted metadata fields directly at multiple locations. |
actions/charts/repo.go |
Adds functionality to force and wait for chart repository updates. |
actions/charts/ranchermonitoring.go |
Waits for monitoring Deployments, DaemonSets, and StatefulSets after installation. |
actions/charts/longhorn.go |
Adds Longhorn network policy defaults and updates CRD handling. Critical (2 votes): PayloadOpts uses Cluster as a direct keyed field despite the embedded InstallOptions. |
actions/charts/action_diagnostics_test.go |
Uses shared monitoring namespace constants and explicit import aliases. |
Review effort: Lite


This aims to fix the currently failing
TestVolumeEncryptionandTestMonitoringIntegration.TestVolumeEncryptionis fixed by restoring an encrypted backup onto an encrypted volume andTestMonitoringIntegrationis fixed by installing Longhorn without the network policies that prevent Prometheus to scrape Longhorn's data.TestMonitoringIntegrationnow also issues a warning if a pre-installed Longhorn includes such policies.Also includes other minor improvements such as improved error handling in
TestVolumeEncryption, fixed backup cleanup and added workload waits when installing the monitoring chart.