Skip to content

Error while integrating upstream rock: cannot create directory mkdir /var/lib/pebble #205

Closed
Closed
Error while integrating upstream rock: cannot create directory mkdir /var/lib/pebble#205
Labels
bugSomething isn't working

Description

@mvlassis
mvlassis
opened on Apr 3, 2026

Bug Description

While trying to use the upstream rock in #203, I ran into a permission issue with pebble:

ops.pebble.PathError: permission-denied - cannot create directory: mkdir /var/lib/pebble.mkdir-new: permission denied

Relevant Log Output

unit-envoy-0: 13:29:07 INFO unit.envoy/0.juju-log grpc:0: execute_components execution loop complete.
unit-envoy-0: 13:29:07 INFO unit.envoy/0.juju-log grpc:0: Status of all CharmReconciler Components:
unit-envoy-0: 13:29:07 INFO unit.envoy/0.juju-log grpc:0: Status: WaitingStatus('[envoy-component] Waiting for Pebble services (envoy).  If this persists, it could be a blocking configuration error.')
unit-envoy-0: 13:29:07 INFO unit.envoy/0.juju-log grpc:0: Status: ActiveStatus('[leadership-gate] ')
unit-envoy-0: 13:29:07 INFO unit.envoy/0.juju-log grpc:0: Status: ActiveStatus('[grpc] ')
unit-envoy-0: 13:29:07 INFO unit.envoy/0.juju-log grpc:0: Status: ActiveStatus('[istio-relations-conflict-detector] ')
unit-envoy-0: 13:29:07 INFO unit.envoy/0.juju-log grpc:0: Status: ActiveStatus('[relation:ingress] ')
unit-envoy-0: 13:29:07 INFO unit.envoy/0.juju-log grpc:0: Status: ActiveStatus('[ambient-ingress-requirer] ')
unit-envoy-0: 13:29:07 INFO unit.envoy/0.juju-log grpc:0: Status of unit set to: WaitingStatus('[envoy-component] Waiting for Pebble services (envoy).  If this persists, it could be a blocking configuration error.')
unit-envoy-0: 13:29:07 INFO juju.worker.uniter.operation ran "grpc-relation-changed" hook (via hook dispatching script: dispatch)
unit-envoy-0: 13:28:44 INFO juju.cmd running containerAgent [3.6.20 8da14f62e48aca8627da67f8cad47d0a4fb60587 gc go1.25.8]
unit-envoy-0: 13:28:44 INFO juju.cmd.containeragent.unit start "unit"
unit-envoy-0: 13:28:44 INFO juju.worker.upgradesteps upgrade steps for 3.6.20 have already been run.
unit-envoy-0: 13:28:44 INFO juju.worker.probehttpserver starting http server on 127.0.0.1:65301
unit-envoy-0: 13:28:44 INFO juju.api connection established to "wss://10.152.183.110:17070/model/8b7bab76-6f89-4724-8d26-65a65ddd2e56/api"
unit-envoy-0: 13:28:44 INFO juju.worker.apicaller [8b7bab] "unit-envoy-0" successfully connected to "wss://10.152.183.110:17070"
unit-envoy-0: 13:28:44 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-envoy-0: 13:28:44 INFO juju.worker.logger logger worker started
unit-envoy-0: 13:28:44 INFO juju.worker.leadership envoy/0 promoted to leadership of envoy
unit-envoy-0: 13:28:44 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-envoy-0: 13:28:44 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-envoy-0: 13:28:44 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-envoy-0: 13:28:44 INFO juju.worker.caasupgrader unblocking abort check
unit-envoy-0: 13:28:44 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-envoy-0
unit-envoy-0: 13:28:44 INFO juju.worker.uniter unit "envoy/0" started
unit-envoy-0: 13:28:44 INFO juju.worker.uniter resuming charm install
unit-envoy-0: 13:28:44 INFO juju.worker.uniter.charm downloading local:envoy-0 from API server
unit-envoy-0: 13:28:44 INFO juju.downloader downloading from local:envoy-0
unit-envoy-0: 13:28:44 WARNING juju.worker.s3caller Response has no supported checksum. Not validating response payload.%!(EXTRA []interface {}=[])
unit-envoy-0: 13:28:44 INFO juju.downloader download complete ("local:envoy-0")
unit-envoy-0: 13:28:45 INFO juju.downloader download verified ("local:envoy-0")
unit-envoy-0: 13:28:48 INFO juju.worker.uniter hooks are retried true
unit-envoy-0: 13:28:48 INFO juju.worker.uniter found queued "install" hook
unit-envoy-0: 13:28:52 WARNING unit.envoy/0.juju-log The ``kubernetes_service_patch v1`` library is DEPRECATED and will be removed in October 2025. For patching the Kubernetes service created by Juju during the deployment of a charm, ``ops.Unit.set_ports`` functionality should be used instead.
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Starting `execute_components` for event 'EnvoyOperator/on/install[1]'
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Executing component: 'leadership-gate'
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Executing component: 'grpc'
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Execution for component 'grpc' complete.  Component now has status 'BlockedStatus('Missing relation with a k8s service info provider. Please add the missing relation.')'
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Executing component: 'istio-relations-conflict-detector'
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Execution for component 'istio-relations-conflict-detector' complete.  Component now has status 'ActiveStatus('')'
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Executing component: 'relation:ingress'
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Execution for component 'relation:ingress' complete.  Component now has status 'ActiveStatus('')'
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Executing component: 'ambient-ingress-requirer'
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Execution for component 'ambient-ingress-requirer' complete.  Component now has status 'ActiveStatus('')'
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log execute_components execution loop complete.
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Status of all CharmReconciler Components:
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Status: BlockedStatus('[grpc] Missing relation with a k8s service info provider. Please add the missing relation.')
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Status: MaintenanceStatus("[envoy-component] Execution pending - waiting on grpc (BlockedStatus('Missing relation with a k8s service info provider. Please add the missing relation.')).")
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Status: ActiveStatus('[leadership-gate] ')
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Status: ActiveStatus('[istio-relations-conflict-detector] ')
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Status: ActiveStatus('[relation:ingress] ')
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Status: ActiveStatus('[ambient-ingress-requirer] ')
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log Status of unit set to: BlockedStatus('[grpc] Missing relation with a k8s service info provider. Please add the missing relation.')
unit-envoy-0: 13:28:52 INFO unit.envoy/0.juju-log HTTP Request: GET https://10.152.183.1/api/v1/namespaces/kubeflow/services/envoy "HTTP/1.1 200 OK"
unit-envoy-0: 13:28:53 INFO unit.envoy/0.juju-log HTTP Request: PATCH https://10.152.183.1/api/v1/namespaces/kubeflow/services/envoy "HTTP/1.1 200 OK"
unit-envoy-0: 13:28:53 INFO unit.envoy/0.juju-log Kubernetes service 'envoy' patched successfully
unit-envoy-0: 13:28:53 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-envoy-0: 13:28:54 WARNING unit.envoy/0.juju-log grpc:0: The ``kubernetes_service_patch v1`` library is DEPRECATED and will be removed in October 2025. For patching the Kubernetes service created by Juju during the deployment of a charm, ``ops.Unit.set_ports`` functionality should be used instead.
unit-envoy-0: 13:28:54 INFO juju.worker.uniter.operation ran "grpc-relation-created" hook (via hook dispatching script: dispatch)
unit-envoy-0: 13:28:54 INFO juju.worker.uniter found queued "leader-elected" hook
unit-envoy-0: 13:28:56 WARNING unit.envoy/0.juju-log The ``kubernetes_service_patch v1`` library is DEPRECATED and will be removed in October 2025. For patching the Kubernetes service created by Juju during the deployment of a charm, ``ops.Unit.set_ports`` functionality should be used instead.
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Starting `execute_components` for event 'EnvoyOperator/on/leader_elected[11]'
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Executing component: 'leadership-gate'
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Executing component: 'grpc'
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Execution for component 'grpc' complete.  Component now has status 'ActiveStatus('')'
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Executing component: 'istio-relations-conflict-detector'
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Execution for component 'istio-relations-conflict-detector' complete.  Component now has status 'ActiveStatus('')'
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Executing component: 'relation:ingress'
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Execution for component 'relation:ingress' complete.  Component now has status 'ActiveStatus('')'
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Executing component: 'ambient-ingress-requirer'
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Execution for component 'ambient-ingress-requirer' complete.  Component now has status 'ActiveStatus('')'
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Executing component: 'envoy-component'
unit-envoy-0: 13:28:56 ERROR unit.envoy/0.juju-log execute_components caught unhandled exception when executing configure_charm for envoy-component
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-envoy-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/charm_reconciler.py", line 92, in reconcile
    component_item.component.configure_charm(event)
  File "/var/lib/juju/agents/unit-envoy-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/component.py", line 50, in configure_charm
    self._configure_unit(event)
  File "/var/lib/juju/agents/unit-envoy-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/pebble_component.py", line 274, in _configure_unit
    self._push_files_to_container()
  File "/var/lib/juju/agents/unit-envoy-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/pebble_component.py", line 245, in _push_files_to_container
    container.push(**container_file_template.get_inputs_for_push())
  File "/var/lib/juju/agents/unit-envoy-0/charm/venv/lib/python3.12/site-packages/ops/model.py", line 2853, in push
    self._pebble.push(
  File "/var/lib/juju/agents/unit-envoy-0/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2731, in push
    self._raise_on_path_error(typing.cast('_FilesResponse', resp), path)
  File "/var/lib/juju/agents/unit-envoy-0/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2670, in _raise_on_path_error
    raise PathError(error['kind'], error['message'])
ops.pebble.PathError: permission-denied - cannot create directory: mkdir /var/lib/pebble.mkdir-new: permission denied
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log execute_components execution loop complete.
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Status of all CharmReconciler Components:
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Status: WaitingStatus('[envoy-component] Waiting for Pebble services (envoy).  If this persists, it could be a blocking configuration error.')
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Status: ActiveStatus('[leadership-gate] ')
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Status: ActiveStatus('[grpc] ')
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Status: ActiveStatus('[istio-relations-conflict-detector] ')
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Status: ActiveStatus('[relation:ingress] ')
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Status: ActiveStatus('[ambient-ingress-requirer] ')
unit-envoy-0: 13:28:56 INFO unit.envoy/0.juju-log Status of unit set to: WaitingStatus('[envoy-component] Waiting for Pebble services (envoy).  If this persists, it could be a blocking configuration error.')
unit-envoy-0: 13:28:56 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)

Additional Context

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions