Conversation
- Single tasks/ suite: bootstrap.sh + bootstrap/task.yaml tracked in git, guide tasks generated and gitignored (mirrors tests/tutorial/) - Fix task.yaml execute path to $SPREAD_PATH/tasks/<name>.sh - spread.yaml: one suite instead of bootstrap/ + tasks/
- Charmhub channel map exposes per-base revisions; refreshing across bases fails with 'cannot upgrade from single base' - Curated map now tracks (workload version, base) per revision - Auto-fallback 24.04 -> 22.04; different-workload scenario degrades to same-workload revision when no older-workload revision exists on the base
…342/299 - Verified bases and workload versions by downloading each revision - 24.04: rev 342 (2.19.4, same-workload), rev 299 (2.19.2, different-workload) - Baseline deploys at REV_BASELINE; guide asserts against it
The charm's upgrade state machine (blocked -> resume-upgrade) only triggers on a workload version change. A charm-only refresh (same workload) completes instantly and the blocked-status wait times out. Baseline is now the highest revision below the target with a DIFFERENT workload version: 299 (2.19.2) -> 344 (2.19.4), both 2/stable on 24.04.
The app turns blocked as soon as the refresh lands, but the highest unit's workload takes minutes to actually upgrade. Poll its message until '(outdated)' disappears (up to 30 min).
Workload downgrades are impossible in OpenSearch, so refreshing back to the baseline revision after an upgrade leaves a poisoned state (units on new workload under old charm). Each rollback task now redeploys a clean baseline via reset_baseline (destroy model + redeploy at REV_BASELINE). Also poll for the Rollback-incompatible blocked unit instead of checking once (the rollback state takes minutes to surface).
The edit was lost during branch switching (helpers.sh was untracked on this branch; checking out docs-revamp-2, where an older version is tracked, overwrote it). Re-applied, deduplicated after an IDE merge conflict, and committed.
- docs/how-to/upgrade.md: the hidden test:run/test:assert blocks that start a mid-flight upgrade (needed before a same-workload rollback can occur) were placed after the visible rollback command instead of before it. This made the generated test roll back to REV_FROM_SAME before any upgrade had started, then wait_idle on a permanently-blocked app that never gets un-blocked (guaranteed timeout). Reordered so setup runs: start upgrade -> assert blocked mid-upgrade behavior -> rollback -> wait_idle -> assert final revision. Purely a reordering of HTML comments; rendered docs prose is unaffected. - resolve_revisions.py: removed unused TARGET_CHANNEL constant whose comment contradicted the actual resolution logic (which only ever uses CHANNEL=2/stable for both baseline and target lookups).
izmalk
marked this pull request as draft
August 29, 2026 21:13
…rios
The last full-suite run failed 3 of 5 tasks, and the "passing"
upgrade-happy-path was a false pass: the readiness gate matched the
transient message "The OpenSearch service stopped." and resume-upgrade
failed ("Highest number unit has not upgraded yet") without failing the
task, leaving the app blocked with all units still on the old workload.
Test harness (tests/guides/upgrade/):
* helpers.sh: add juju_refresh (retrying refresh, tolerates Juju's
transient "deploy incomplete, please try refresh again" error),
wait_highest_unit_upgraded (real workload-upgrade gate: active/idle,
running version in message, no "(outdated)" marker), juju_run_action
(propagates action failures), clear_node_lock (404-tolerant lock
cleanup). reset_baseline now fails loudly and waits for the model to
be gone before re-adding it.
* extract_guide_tasks.py: generated scripts emit set -x and an ERR trap
so Spread logs attribute failures to a specific line.
* resolve_revisions.py: leave REV_FROM_DIFF empty (with a loud warning)
when no older-workload revision exists below the baseline, instead of
silently aliasing REV_FROM_SAME — a same-workload rollback can never
produce the documented "Rollback incompatible" state.
* spread.yaml: raise VM memory to 24G / disk to 60G (recovery peaks at
5 opensearch containers plus the controller).
* README.md: document the new helpers and the skip behaviour.
Guide annotations (docs/how-to/upgrade.md):
* Happy path: replace the broken "(outdated)" message check with
wait_highest_unit_upgraded; retry resume-upgrade and fail the task if
it keeps failing; final assertion now requires every unit non-outdated
and the app active — green cluster health alone no longer passes.
* Rollback tasks: use juju_refresh for both refreshes; wait for the
highest unit to finish its workload upgrade before rolling back so the
rollback starts from the documented mid-upgrade state; skip cleanly
when REV_FROM_DIFF is empty; broaden the blocked-unit poll and dump
juju debug-log on timeout.
* Recovery task: build the broken state deterministically in its own
setup instead of inheriting the previous task's leftovers; detect the
stuck unit by status (message text varies) with a highest-ordinal
fallback; skip literal example commands (remove-unit opensearch/2,
DELETE /index1, node-lock DELETE) that only exist in example output;
re-resolve OS_UNIT_IP before final assertions.
The recovery section previously ordered the steps as: add a replacement unit, remove the rolled back unit, remove the node lock. That order deadlocks in practice: the broken unit holds the .charm_node_lock document, so the freshly added replacement unit blocks forever on "Requesting lock on operation: start" and can never reach active/idle. This was observed directly in the recover-from-rollback Spread run (2026-08-29): after `juju add-unit opensearch -n 1`, the new unit opensearch/3 sat in `waiting` with "Requesting lock on operation: start" for the full 1800s timeout while the broken opensearch/2 remained blocked — the task timed out without ever reaching the remove/lock steps that would have unblocked it. Move the "Add a new unit" section to run after "Remove rolled back unit" and "Remove lock", so the departed unit is gone and the stale lock document is deleted before the replacement unit tries to acquire the lock. The "Remove lock" prose is updated to explain the lock must be cleared *before* adding the replacement, rather than as a remedy for an already-stuck unit.
juju show-unit --format=json does not expose workload-status at all: its UnitInfo struct (cmd/juju/application/showunit.go) only has workload-version, machine, public-address, charm, leader, life and relation-info. The two 'no unit still outdated' assertions in the upgrade guide therefore crashed with KeyError: 'workload-status'. Add an assert_no_outdated_units helper that reads unit messages from juju status --format=json (which does expose workload-status and is already used successfully elsewhere in the same tests), and swap both call sites to it. Also fix the same latent bug in the unused wait_unit_message helper.
juju run's process exit code does not reflect action failure: the CLI only exits non-zero when the action's result map contains a numeric return-code >= 1 (cmd/juju/action/common.go, convertActionOutput). pre-upgrade-check refuses to run mid-upgrade via action-fail without setting a return-code, so the mid-upgrade assertion recorded rc=0 and falsely reported 'pre-upgrade-check unexpectedly succeeded' even though the action had failed with 'Upgrade already in progress'. Add an assert_action_fails helper that detects failure via the 'Action id N failed:' line the CLI always prints for failed actions, plus the expected message substring, and swap the mid-upgrade pre-upgrade-check assertion to it. Also rewrite juju_run_action to use the same text-based detection.
A rollback happens mid-upgrade, when the highest unit already runs REV_TO's workload. Resolving REV_FROM_SAME relative to the baseline's workload (297 = 2.19.2) made the 'same-workload' rollback a workload DOWNGRADE for that unit — OpenSearch refuses downgrades, so the unit stayed stuck at 'Waiting for OpenSearch to start...' until wait_idle timed out after 3600s (rollback-same-workload failure, 2026-08-30 run). Redefine both rollback targets relative to REV_TO's workload version: - REV_FROM_SAME = highest revision below REV_TO with the SAME workload as REV_TO (now 342 = 2.19.4): the upgraded unit matches cleanly and the guide's documented 'all units active/idle' outcome is achievable. - REV_FROM_DIFF = highest revision below REV_TO with an OLDER workload than REV_TO (now 299 = 2.19.2): non-empty on this base, so the different-workload rollback and recovery scenarios actually run their documented 'Rollback incompatible -> force-refresh-start' flows instead of skipping. Also move the REV_FROM_DIFF skip guards into the setup blocks so a skipped scenario exits before the ~40-minute reset_baseline rebuild instead of after it.
…recovery flow
Three fixes from the 2026-08-30 full-suite run (3/5 passed):
1. rollback-different-workload skipped falsely: the REV_FROM_DIFF guard
moved to the top of the setup block ran before /root/revisions.env
was sourced, so the variable was always empty. Source the env file
before the guard.
2. rollback-same-workload failed at the final assertion: after
'juju refresh' back to a same-workload revision, the app stays
blocked ('Upgrading. Verify highest unit is healthy & run
resume-upgrade') with the remaining units still '(outdated)' — the
rollback only reverts the charm code; the workload rollout must be
completed via resume-upgrade under the rolled-back charm (matches
the charm's own integration-test helper). Document the step and add
it to the test with a retry wrapper.
3. recover-from-rollback failed twice: (a) the blocked-unit poll only
matched 'Rollback incompatible'/'Rollback unsupported', but the
actual state was 'An error occurred during the start of the
OpenSearch service.' — extend the match; (b) 'juju remove-unit'
without --force never completes on a stuck unit (hooks can't run),
so the lock cleanup ran while the unit still held the lock and the
replacement unit deadlocked on 'Requesting lock on operation:
start'. Force the removal, destroy its storage, and wait for the
unit to disappear before clearing the lock. Also tolerate
force-refresh-start refusing ('No rollback in progress') and bump
the add-unit settle timeout to 3600s.
The rollback-same-workload task failed because the generated script ran
the visible 'juju run opensearch/leader resume-upgrade' command from the
guide BEFORE the hidden retry block: the extractor emits both, and the
bare command timed out at the CLI level ('ERROR timed out waiting for
results from: unit opensearch/1') because the leader's action queue was
backed up behind upgrade hooks right after the refresh.
Two fixes:
1. docs/how-to/upgrade.md: mark the visible resume-upgrade fence with
<-- test:skip --> so only the retry block executes. The command
stays in the rendered guide for readers.
2. tests/guides/upgrade/helpers.sh: juju_run_action now also fails on a
non-zero 'juju run' exit code, not just on 'Action id N failed:'
text. The CLI exits non-zero for transport-level errors (timeouts)
while action-fail exits 0 — both signals must be treated as failure.
The retry block now uses juju_run_action instead of bare juju run,
so a charm-side refusal ('Highest number unit has not upgraded yet')
no longer false-succeeds the retry.
Vale's US-spellcheck rule flags 'rollout' as a misspelling; use 'rolling upgrade' instead, which is also the term the guide uses elsewhere.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of issue or feature:
Automatic
Solution:
Heavily automated script to determine revisions for testing, generate Spread testing tasks and run them in a multipass VM (similar to the tutorial).
See the README file in
tests/guides/upgrade.How was this change tested?
Checklist