Skip to content

unable to use velero with self-signed CA and server certificate on a server #235

Description

@Wodel

Hi,

I've configured a seaweedfs s3 storage and used a server certificate generated from a self-signed CA.
I've used the charmed s3-integrator from channel 2.
This charm supports the use of "tls-ca-chain", so the s3-integraror can connect to my s3 storage.
But the rest of velero pods won't.

user05@mk8s01:~$ juju run s3-integrator/0 status-detail
Running operation 15 with 1 task
  - task 16 on unit-s3-integrator-0

Waiting for task 16...
18:50:30 Stored statuses:
18:50:31                      App Statuses                      
┏━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ Status ┃ Component Name ┃ Message ┃ Action ┃ Reason ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ Active │ general        │         │ N/A    │ N/A    │
│ Active │ s3-provider    │         │ N/A    │ N/A    │
└────────┴────────────────┴─────────┴────────┴────────┘
18:50:31                      Unit Statuses                     
┏━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ Status ┃ Component Name ┃ Message ┃ Action ┃ Reason ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ Active │ general        │         │ N/A    │ N/A    │
│ Active │ s3-provider    │         │ N/A    │ N/A    │
└────────┴────────────────┴─────────┴────────┴────────┘

json-output:
  app: '[{"Status": "Active", "Component Name": "general", "Message": "", "Action":
    "N/A", "Reason": "N/A"}, {"Status": "Active", "Component Name": "s3-provider",
    "Message": "", "Action": "N/A", "Reason": "N/A"}]'
  unit: '[{"Status": "Active", "Component Name": "general", "Message": "", "Action":
    "N/A", "Reason": "N/A"}, {"Status": "Active", "Component Name": "s3-provider",
    "Message": "", "Action": "N/A", "Reason": "N/A"}]'

user05@mk8s01:~$ juju status
Model          Controller  Cloud/Region      Version  SLA          Timestamp
velero-backup  uk8sx       my-k8s/localhost  3.6.21   unsupported  18:50:35Z

App              Version  Status   Scale  Charm            Channel      Rev  Address         Exposed  Message
s3-integrator             active       1  s3-integrator    2/edge       550  10.152.183.105  no       
velero-operator           blocked      1  velero-operator  1.15/stable  421  10.152.183.235  no       Velero Storage location is not ready: BackupStorageLocation is unavailable

Unit                Workload  Agent  Address       Ports  Message
s3-integrator/0*    active    idle   10.1.111.113         
velero-operator/0*  blocked   idle   10.1.111.118         Velero Storage location is not ready: BackupStorageLocation is unavailable
kubectl logs -n velero-backup velero-operator-0 -f
2026-05-13T18:48:47.189Z [container-agent] 2026-05-13 18:48:47 ERROR juju-log Uncaught exception while in charm code:
2026-05-13T18:48:47.189Z [container-agent] Traceback (most recent call last):
2026-05-13T18:48:47.189Z [container-agent]   File "/var/lib/juju/agents/unit-velero-operator-0/charm/venv/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
2026-05-13T18:48:47.189Z [container-agent]     yield
2026-05-13T18:48:47.189Z [container-agent]   File "/var/lib/juju/agents/unit-velero-operator-0/charm/venv/httpx/_transports/default.py", line 236, in handle_request
2026-05-13T18:48:47.189Z [container-agent]     resp = self._pool.handle_request(req)
2026-05-13T18:48:47.189Z [container-agent]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-05-13T18:48:47.189Z [container-agent]   File "/var/lib/juju/agents/unit-velero-operator-0/charm/venv/httpcore/_sync/connection_pool.py", line 256, in handle_request
2026-05-13T18:48:47.189Z [container-agent]     raise exc from None
2026-05-13T18:48:47.189Z [container-agent]   File "/var/lib/juju/agents/unit-velero-operator-0/charm/venv/httpcore/_sync/connection_pool.py", line 236, in handle_request
2026-05-13T18:48:47.189Z [container-agent]     response = connection.handle_request(
2026-05-13T18:48:47.189Z [container-agent]                ^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-05-13T18:48:47.189Z [container-agent]   File "/var/lib/juju/agents/unit-velero-operator-0/charm/venv/httpcore/_sync/connection.py", line 101, in handle_request
2026-05-13T18:48:47.189Z [container-agent]     raise exc
....
2026-05-13T18:48:47.189Z [container-agent]   File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
2026-05-13T18:48:47.189Z [container-agent]     self.gen.throw(value)
2026-05-13T18:48:47.189Z [container-agent]   File "/var/lib/juju/agents/unit-velero-operator-0/charm/venv/httpcore/_exceptions.py", line 14, in map_exceptions
2026-05-13T18:48:47.189Z [container-agent]     raise to_exc(exc) from exc
2026-05-13T18:48:47.189Z [container-agent] httpcore.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)
kubectl logs -n velero-backup velero-57f8998cf4-xjw89 -f
time="2026-05-13T18:50:19Z" level=info msg="plugin process exited" backupLocation=velero-backup/default cmd=/plugins/velero-plugin-for-aws controller=backup-sync id=832 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws
time="2026-05-13T18:51:08Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero-backup/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:142"
time="2026-05-13T18:51:12Z" level=error msg="fail to validate backup store" backup-storage-location=velero-backup/default controller=backup-storage-location error="rpc error: code = Unknown desc = operation error S3: ListObjectsV2, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get \"https://10.30.0.20/velero-backups?delimiter=%2F&list-type=2&prefix=\": tls: failed to verify certificate: x509: certificate signed by unknown authority" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:206" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).IsValid" logSource="pkg/controller/backup_storage_location_controller.go:145"
time="2026-05-13T18:51:12Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero-backup/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121"
time="2026-05-13T18:51:12Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"default\" is unavailable: rpc error: code = Unknown desc = operation error S3: ListObjectsV2, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get \"https://10.30.0.20/velero-backups?delimiter=%2F&list-type=2&prefix=\": tls: failed to verify certificate: x509: certificate signed by unknown authority)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:179"
time="2026-05-13T18:51:12Z" level=info msg="plugin process exited" backup-storage-location=velero-backup/default cmd=/plugins/velero-plugin-for-aws controller=backup-storage-location id=841 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws

Is it possible to make it work with a self-signed certificate??

Regards.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions