Skip to content

fix: generate and persist SECRET_KEY for pulpcore 3.105.13 compatibility - #287

Merged
rooftopcellist merged 4 commits into
ansible:mainfrom
lucas-benedito:fix/secret-key-generation
Sep 10, 2026
Merged

rooftopcellist merged 4 commits into
ansible:mainfrom
lucas-benedito:fix/secret-key-generation

Conversation

@lucas-benedito

Copy link
Copy Markdown
Member
SUMMARY

pulpcore 3.105.13 removed the insecure SECRET_KEY = "SECRET" default (pulp/pulpcore#7873), causing all galaxy-ng pods to enter CrashLoopBackOff on startup:

django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

The galaxy-operator previously relied on this fallback. This fix adds secret_key_configuration.yml to the common role, following the same pattern as db_fields_encryption_configuration.yml.

Behavior:

  • On first install: generates a 50-character random key via lookup('password', ...), stores it in a <name>-secret-key Kubernetes Secret with ownerReferences removed (so it survives CR deletion), and merges it into pulp_combined_settings so it lands in /etc/pulp/settings.py as SECRET_KEY
  • On subsequent reconciliations: reads and reuses the stored value (idempotent)
  • If pulp_settings.secret_key is set in the CR spec (e.g. by the AAP installer via hub_secret_key): that value is used as-is and no Secret is created
ISSUE TYPE
  • Bug, Docs Fix or other nominal change
COMPONENT NAME
  • Operator
STEPS TO REPRODUCE AND EXTRA INFO

Deploy galaxy-operator with quay.io/ansible/galaxy-ng:latest (any build after 2026-09-01). All api/content/worker pods crash immediately with SECRET_KEY setting must not be empty.

pulpcore 3.105.13 removed the insecure SECRET_KEY = "SECRET" default
(pulp/pulpcore#7873), causing all galaxy-ng pods to enter CrashLoopBackOff
with "The SECRET_KEY setting must not be empty" on startup.

Add secret_key_configuration.yml to the common role. On first install it
generates a 50-character random key, stores it in a <name>-secret-key
Kubernetes Secret with ownerReferences removed (so it survives CR
deletion), and merges it into pulp_combined_settings so it lands in
/etc/pulp/settings.py as SECRET_KEY. Subsequent reconciliations reuse
the stored value. If the user or installer provides secret_key in
pulp_settings, that value is used as-is and no Secret is created.

Assisted-by: Claude
Signed-off-by: Lucas Benedito <lbenedit@redhat.com>
@lucas-benedito
lucas-benedito force-pushed the fix/secret-key-generation branch from addb743 to 2a3b072 Compare September 7, 2026 16:10
@jerabekjiri

Copy link
Copy Markdown
Contributor

thank you, that's exactly we need 👍

Following the awx-operator pattern, store the secret key secret name
in the Galaxy CR status so backup/restore can preserve it across
reinstalls. Without this, a restore generates a new random SECRET_KEY,
invalidating all active Django sessions and breaking the restored
instance.

Changes:
- secret_key_configuration.yml: store secret_key_secret_name fact
- galaxy-status/tasks/main.yml: write secretKeySecret to CR status
- backup/tasks/secrets.yml: add secretKeySecret to the backup dump list

The restore role already handles secrets generically via secrets.yml.j2,
so no restore-side changes are needed.

Assisted-by: Claude
Signed-off-by: Lucas Benedito <lbenedit@redhat.com>
@lucas-benedito
lucas-benedito force-pushed the fix/secret-key-generation branch from 7dc0834 to 3f69eec Compare September 8, 2026 14:02
The "Store secret key secret name" task was gated on
`pulp_combined_settings.secret_key is not defined`, but the two tasks
immediately before it already set that same key. By the time "Store
secret key secret name" ran, the condition was always false, so
secret_key_secret_name was never set, status.secretKeySecret was never
written, and roles/backup/tasks/dump_generated_secret.yml failed with
"'dict object' has no attribute 'secretKeySecret'" on every CI job.

Capture whether the user supplied a literal secret_key once, into its
own fact, before anything mutates pulp_combined_settings, and gate all
downstream tasks on that stable fact instead - the same separation of
"secret name" from "secret value" already used by
db_fields_encryption_configuration.yml.

Also declare status.secretKeySecret and the new spec.secret_key_secret
override on the Galaxy CRD. Both are new structural-schema properties;
without them the Kubernetes API server silently prunes the field on
write, so status.secretKeySecret would still never persist even after
the logic fix above. secret_key_secret is also now honored as the
Secret's name (falling back to the previous <name>-secret-key default),
needed so a restored deployment can be pinned back to its original
secret name.

Signed-off-by: Christian M. Adams <chadams@redhat.com>
The previous commit's intent was for backup/restore to preserve
SECRET_KEY across reinstalls, matching db_fields_encryption_secret,
admin_password_secret, postgres_configuration_secret and
container_token_secret. That linkage was incomplete: the secret name
was never recorded on the GalaxyBackup CR's own status, never read
back during restore, and never pinned into the restored CR's spec.

As a result, SECRET_KEY continuity only survived a restore to the same
deployment name. Restoring under a different GalaxyRestore.spec.deployment_name
(an explicitly supported flow) would still generate a brand-new random
key, invalidating sessions - the exact failure mode this feature exists
to prevent.

- roles/backup/tasks/init.yml: read secretKeySecret from the live
  Galaxy CR status, same as the other generated secrets.
- roles/backup/tasks/update_status.yml: write it onto the GalaxyBackup
  CR's own status so restore can find it.
- config/crd/bases/galaxy_v1beta1_galaxybackup_crd.yaml: declare the
  new status field (same structural-schema pruning issue as the Galaxy
  CRD).
- roles/backup/tasks/custom_resource.yml: pin the actual secret name
  into the backed-up CR spec so a rename-restore still resolves to it.
  Only done when a secret was actually generated (secret_key_secret can
  legitimately be absent when pulp_settings.secret_key was provided as
  a literal value).
- roles/restore/tasks/init.yml, roles/restore/tasks/cleanup.yml: read
  the name back from the GalaxyBackup status and strip its
  ownerReferences after restore, same as the other generated secrets.

Signed-off-by: Christian M. Adams <chadams@redhat.com>
@rooftopcellist

Copy link
Copy Markdown
Member

Pushed two follow-up commits to get CI green and finish out the backup/restore piece:

f904def — fix the CI failure. The "Store secret key secret name" task was gated on pulp_combined_settings.secret_key is not defined, but the two tasks right before it already set that value — so the condition was always false by the time it ran, secret_key_secret_name never got set, status.secretKeySecret was never written, and the backup role's dump_generated_secret.yml failed on the undefined status key on every job. Fixed by capturing "was a value provided by the user" into its own fact before anything mutates pulp_combined_settings, same separation of "secret name" vs "secret value" that db_fields_encryption_configuration.yml already uses.

That alone wasn't quite enough, though: status.secretKeySecret also wasn't declared in the Galaxy CRD's status schema, and since it's a structural OpenAPI schema, the API server would've silently pruned the field on write even with the logic fixed. Added it alongside the existing dbFieldsEncryptionSecret-style fields, plus a matching spec.secret_key_secret override (used below).

355bb63 — finish the backup/restore linkage. The goal of the second commit here was for backup/restore to preserve SECRET_KEY the same way it does for db_fields_encryption_secret, admin_password_secret, etc., but a few pieces were still missing: the name wasn't written to the GalaxyBackup CR's own status, wasn't read back on restore, and wasn't pinned into the restored CR's spec. Net effect was that SECRET_KEY continuity only worked for a restore to the same deployment name — restoring under a different GalaxyRestore.spec.deployment_name would still generate a fresh random key. Wired up the remaining pieces (status write/read on the backup CR, spec pinning in custom_resource.yml, ownerReferences cleanup on restore) following the same pattern as the other generated secrets.

CI is running now on the updated branch — will confirm once it's green.

@rooftopcellist
rooftopcellist merged commit dc46e03 into ansible:main Sep 10, 2026
11 of 12 checks passed
@lucas-benedito
lucas-benedito deleted the fix/secret-key-generation branch September 11, 2026 08:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants