Cannot integrate with Ceph - Missing option to provide endpoint SSL CA

[gustavosr98]

Bug Description

A big portion on-premise deployments like Openstack, Kubernetes and Ceph use certificates generated by Vault, and not by a publicly trusted CA

Now, there is no option in the charm to provide the SSL CA cert of an on premise Ceph S3 endpoint

Affecting most Kubeflow deployments on-premise

To Reproduce

Deploy Minio in gateway mode
Point to an on-prem S3 endpoint that requires CA certs

I tried using ssl-ca but I assume that one only works for the Minio when on server mode

  minio:
    charm: minio
    channel: ckf-1.10/stable
    revision: 459
    base: ubuntu@20.04/stable
    resources:
      oci-image: 547
    scale: 1
    options:
      access-key: include-file://../../secrets/kubeflow/s3-kubeflow__access
      secret-key: include-file://../../secrets/kubeflow/s3-kubeflow__secret
      console-port: 9001
      gateway-storage-service: s3
      mode: gateway
      port: 9000
      ssl-ca: "include-base64://../../secrets/k8s/vault-root-ca.crt" # CA of https://ceph-radosgw.maas
      ssl-cert: ""
      ssl-key: ""
      storage-service-endpoint: https://ceph-radosgw.maas
    constraints: arch=amd64

Environment

Kubeflow 1.10/stable
Minio ckf-1.10/stable - revision: 459
S3: Ceph radosgw (on premise)

Relevant Log Output

# kk logs mlflow-minio-0
Defaulted container "minio" out of: minio, juju-pod-init (init)
2025-06-06T20:31:24.410Z [pebble] Started daemon.
2025-06-06T20:31:24.415Z [pebble] POST /v1/services 3.088395ms 202
2025-06-06T20:31:24.419Z [pebble] Service "minio" starting: docker-entrypoint.sh minio [ gateway s3 https://ceph-radosgw.maas --console-address :9001 ]
2025-06-06T20:31:24.697Z [pebble] Service "minio" stopped unexpectedly with code 1
2025-06-06T20:31:24.697Z [pebble] Service "minio" on-failure action is "restart", waiting ~500ms before restart (backoff 1)
2025-06-06T20:31:24.699Z [pebble] Change 1 task (Start service "minio") failed: service start attempt: exited quickly with code 1, will restart
2025-06-06T20:31:24.703Z [pebble] GET /v1/changes/1/wait 286.729015ms 200
2025-06-06T20:31:24.703Z [pebble] Started default services with change 1.
2025-06-06T20:31:25.233Z [pebble] Service "minio" starting: docker-entrypoint.sh
[..]
2025-06-06T21:29:25.587Z [pebble] Service "minio" stopped unexpectedly with code 1
2025-06-06T21:29:25.587Z [pebble] Service "minio" on-failure action is "restart", waiting ~30s before restart (backoff 115)
2025-06-06T21:29:56.241Z [pebble] Service "minio" starting: docker-entrypoint.sh minio [ gateway s3 https://ceph-radosgw.maas --console-address :9001 ]
2025-06-06T21:29:56.557Z [pebble] Service "minio" stopped unexpectedly with code 1
2025-06-06T21:29:56.557Z [pebble] Service "minio" on-failure action is "restart", waiting ~30s before restart (backoff 116)

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-7494.

This message was autogenerated

[gustavosr98]

Workaround

Transparently inject the SSL CA cert to Minio pods in Kubeflow namepsace with Kyverno

# ----- Kyverno
H="juju ssh -m $K8S_MODEL k8s/leader -- sudo k8s helm"

$H repo add kyverno https://kyverno.github.io/kyverno/
$H repo update
$H install kyverno kyverno/kyverno --namespace kyverno --create-namespace --set replicaCount=3 --set nodeSelector.juju-application=k8s-cpu-worker

# ----- Add Vault CA to some pods
# Minio (kubeflow, mlflow) pods
# Jupyter notebooks pods? Pipelines? 

CA_FILE="secrets/k8s/vault-root-ca.crt"
NAMESPACE="kubeflow"

echo "Creating/updating ConfigMap with CA cert..."
kubectl create configmap vault-root-ca \
  --namespace $NAMESPACE \
  --from-file=vault-root-ca.crt=$CA_FILE \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl apply -f - << EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: inject-vault-root-ca
spec:
  rules:
    - name: inject-vault-root-ca
      match:
        resources:
          kinds:
            - Pod
          namespaces:
            - kubeflow
          selector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - minio
                  - mlflow-minio
      mutate:
        patchStrategicMerge:
          spec:
            volumes:
              - name: vault-root-ca
                configMap:
                  name: vault-root-ca
                  optional: true
              - name: ca-certs
                emptyDir: {}
            initContainers:
              - name: update-ca-certificates
                image: debian:bullseye
                command: ["/bin/sh", "-c"]
                args:
                  - |
                    apt-get update && \
                    apt-get install -y --no-install-recommends ca-certificates && \
                    update-ca-certificates && \
                    cp /etc/ssl/certs/ca-certificates.crt /ca-certs/ca-certificates.crt
                volumeMounts:
                  - name: vault-root-ca
                    mountPath: /usr/local/share/ca-certificates/vault-root-ca.crt
                    subPath: vault-root-ca.crt
                  - name: ca-certs
                    mountPath: /ca-certs
            containers:
              - (name): "*"
                volumeMounts:
                  - name: vault-root-ca
                    mountPath: /usr/local/share/ca-certificates/vault-root-ca.crt
                    subPath: vault-root-ca.crt
                    readOnly: true
                  - name: ca-certs
                    mountPath: /etc/ssl/certs/ca-certificates.crt
                    subPath: ca-certificates.crt
EOF

kubectl -n kubeflow delete po minio-0 mlflow-minio-0

ubuntu@juju-8966e0-1-lxd-1:~$ mc ls kubeflow
[2025-06-06 22:20:14 UTC]     0B mlpipeline/

ubuntu@juju-8966e0-1-lxd-1:~$ mc ls mlflow
[2025-06-06 22:22:13 UTC]     0B mlflow/

# kubectl logs -n kubeflow pod/minio-0
Defaulted container "minio" out of: minio, update-ca-certificates (init), juju-pod-init (init)
2025-06-06T22:20:02.254Z [pebble] Started daemon.
2025-06-06T22:20:02.258Z [pebble] POST /v1/services 2.624342ms 202
2025-06-06T22:20:02.261Z [pebble] Service "minio" starting: docker-entrypoint.sh minio [ gateway s3 https://ceph-radosgw.maas --console-address :9001 ]
2025-06-06T22:20:03.266Z [pebble] GET /v1/changes/1/wait 1.007619983s 200
2025-06-06T22:20:03.267Z [pebble] Started default services with change 1.