Skip to content

keycloak-rbac-bootstrap job fails: expected hub client id does not match what nebari-operator provisions #203

Description

@tylerpotts

Summary

On a fresh deploy against nebari-operator v0.1.0-alpha.20, the keycloak-rbac-bootstrap job (PostSync hook) fails on every attempt because the hub OIDC client id it derives does not match the client the operator actually provisions in Keycloak.

  • Job expects: jupyterhub-data-science-pack-nebari-data-science-pack (the chart's jupyterhub-<fullname> convention)
  • Operator provisions: data-science-pack-data-science-pack-nebari-data-science-pack (<namespace>-<NebariApp name>, confirmed via the client-id key of the operator-written oidc-client Secret in the release namespace)

Environment

  • Chart: nebari-data-science-pack 0.1.1 (also reproduces with current main; observed on a branch containing only unrelated changes)
  • nebari-operator: v0.1.0-alpha.20 (NIC-pinned)
  • Release name data-science-pack, namespace data-science-pack, fresh NIC hetzner-k3s cluster, Keycloak realm nebari
  • kcAdminCredentialSecret set, so the job runs

Failure

2026-07-30 21:54:52,102 INFO ==> 1. group-membership mapper on 'groups' scope
2026-07-30 21:54:52,251 INFO scope 'groups' group-membership mapper already in desired state
2026-07-30 21:54:52,251 INFO ==> 2. hub OIDC client + service account
Traceback (most recent call last):
  File "/scripts/keycloak_rbac_bootstrap.py", line 521, in <module>
    sys.exit(main())
  File "/scripts/keycloak_rbac_bootstrap.py", line 516, in main
    run(config, kc)
  File "/scripts/keycloak_rbac_bootstrap.py", line 479, in run
    hub_uuid = kc.get_client_uuid(config.realm, config.hub_client_id)
  File "/scripts/keycloak_rbac_bootstrap.py", line 265, in get_client_uuid
    raise RuntimeError(
RuntimeError: client 'jupyterhub-data-science-pack-nebari-data-science-pack' not found in realm 'nebari'

The job retries to backoffLimit and the Application reports the failed hook. Step 1 (groups mapper) succeeds, so login and basic hub use are unaffected; whatever step 2+ configures (hub client service account, role-gated shared mount wiring) never completes.

Analysis

The chart derives the hub client id by its own convention (jupyterhub-<fullname>, see the hubClientID helper and the values comment describing the nebi-<releaseName>-nebari-nebi-pack operator naming convention), but operator v0.1.0-alpha.20 names the provisioned client <namespace>-<NebariApp name> with no app-type prefix. Any deploy where the two conventions disagree fails the same way. The same skew presumably affects nebi-client-id derivation and the hub <-> nebi token-exchange config (03-nebi-envs.py), which reads the same chart-derived ids; that path was not exercised here because no remote nebi pack is deployed.

Suggested fix directions

  • Most robust: stop deriving the client id by convention. The operator already writes the authoritative id to the oidc-client Secret (<fullname>-oidc-client, key client-id) in the release namespace; the job (and the chart-derived defaults) could read it from there.
  • Alternatively: update the helpers to match the operator's current naming, and pin/document the minimum compatible operator version.

Evidence commands

kubectl get secret <fullname>-oidc-client -n <release-ns> -o jsonpath='{.data.client-id}' | base64 -d
kubectl logs -n keycloak job/<fullname>-rbac-bootstrap

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Start date

    None yet

    Target date

    None yet

    Size

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions