Partial multi-cloud upload failure discards successful uploads and skips relation-data update, risking stale/pruned images

[cbartz]

Bug Description

When --upload-clouds is configured with more than one cloud, _upload_to_clouds() in app/src/github_runner_image_builder/openstack_builder.py (lines ~840-877) uploads to each cloud sequentially in a loop, calling store.upload_image() per cloud. If a single cloud's upload raises UploadImageError (e.g. a network blip), the exception propagates immediately and the function never returns — discarding the images list entries for any clouds that already uploaded successfully in that same call.

This exception propagates up through openstack_builder.run() → the charm's builder.run() (re-raised as BuilderRunError) → charm.py::_run():

cloud_images = builder.run(config_matrix=..., static_config=...)
self.image_observer.update_image_data(cloud_images=cloud_images)

Because builder.run() raises, update_image_data() is never called — relation data is not updated for any cloud, including ones whose upload actually succeeded in this run.

Compounding this, store.py::upload_image() (lines 59-92) prunes older images for a cloud immediately after that cloud's own upload succeeds, with no coordination with whether the batch as a whole (or the relation-data update) ever completes:

image = connection.create_image(...)
logger.info("Pruning older images %s, keeping %s.", image_name, keep_revisions)
_prune_old_images(connection=connection, image_name=image_name, num_revisions=keep_revisions)

Net effect: if one upload-cloud consistently fails (e.g. a flaky network path to that cloud's Glance endpoint) while another cloud consistently succeeds, every retry keeps uploading + pruning on the healthy cloud but never advances relation data (since the batch always errors out on the unhealthy cloud). After enough successful-but-unrecorded builds on the healthy cloud, keep_revisions pruning can delete the very image that relation data still (silently, stale) points to — leaving consumers with a reference to an image that no longer exists in Glance.

Note: src/image.py::update_image_data() already supports partial data cleanly — it keys strictly by cloud_id and updates each relation independently, so no relation-data model changes are required. The fix is to have the upload path surface partial successes instead of swallowing them via an unhandled exception, and to update relation data with whatever succeeded even when the overall batch partially fails.

Impact

High (major functionality broken, no workaround)

Impact Rationale

This caused a full outage of s390x runner spawning on ps6 (production self-hosted runners) for ~4 days (31 Aug - 04 Sep). The image-builder unit kept retrying every cron cycle, but every run failed while uploading to the ps7 cross-site cloud (persistent SSLEOFError on glance.ps7.canonical.com), so relation data was never refreshed even though local ps6 uploads sometimes succeeded. Runner-manager eventually referenced a pruned/missing image ID, causing openstack.exceptions.NotFoundException: No Image found for <image_id> and 100% runner spawn failures for the s390x fleet on ps6. Root-caused and worked around manually via relation-set; see canonical Jira ISD-6561 / ISD-5444 for the incident writeup.

To Reproduce

1. Deploy github-runner-image-builder with --upload-clouds pointing at two (or more) OpenStack clouds.
2. Cause uploads to one of the clouds to reliably fail (e.g. block/reset the TLS connection to that cloud's Glance endpoint) while the other cloud remains healthy.
3. Trigger repeated builds (cron run event) over several cycles, with keep_revisions set low (e.g. 5).
4. Observe: the healthy cloud accumulates new images and prunes old ones each cycle, but the image relation data is never updated (every run raises BuilderRunError due to the unhealthy cloud), until the once-referenced image on the healthy cloud is eventually pruned away.

Environment

• Charm: github-runner-image-builder-operator, deployed via Juju on OpenStack (multi-cloud upload configuration, s390x arch).
• Affected component: app/src/github_runner_image_builder/openstack_builder.py::_upload_to_clouds, app/src/github_runner_image_builder/store.py::upload_image, src/charm.py::_run.

Relevant log output

2026-09-01 07:47:17,338 - ERROR - Error while uploading image.
urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:2437)
...
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='glance.ps7.canonical.com', port=9292): Max retries exceeded with url: /v2/images/6cc7e346-2a07-46e2-a9ad-6ee109870f19/file (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol')))
...
openstack.exceptions.SDKException: Image creation failed: SSL exception connecting to https://glance.ps7.canonical.com:9292/v2/images/6cc7e346-2a07-46e2-a9ad-6ee109870f19/file: ...

2026-09-04 05:18:00,416 - ERROR - github_runner_manager.manager.runner_manager - Failed to spawn a runner.
...
openstack.exceptions.NotFoundException: No Image found for 944c3715-6c31-486e-a20b-d4bafaf7c18f

Additional context

Suggested fix direction:

• _upload_to_clouds(): catch per-cloud UploadImageError, continue attempting remaining clouds, and return whatever succeeded (plus signal the partial failure) instead of aborting on the first exception.
• charm.py::_run(): call self.image_observer.update_image_data(cloud_images=cloud_images) with the partial results before surfacing/logging the overall failure, so relation data always advances for clouds that did succeed.

AI-generated message.