All pods are in pending, no logs shown

[rinormaloku]

vcluster list shows the cluster, when querying debug logs it says that it doesn't exist, all pods are in pending.

$ vcluster list

       NAME    | STATUS  | CONNECTED | AGE
  -------------+---------+-----------+------
    my-cluster | running | True      | 66s


$ vcluster debug collect
? Please specify vcluster name my-cluster
06:40:09 fatal failed to find vCluster my-cluster: couldn't find vcluster my-cluster


$ k get pods -A
NAMESPACE            NAME                                      READY   STATUS    RESTARTS   AGE
kube-system          coredns-75bb76df-h7n5d                    0/1     Pending   0          59s
local-path-storage   local-path-provisioner-6f6fd5d9d9-84hd7   0/1     Pending   0          59s

not sure how to troubleshoot this

[Didier-Lane]

Hello 👋
Can you share the output of the events ?
You can get them with

kubectl get events --sort-by=".lastTimestamp" --watch --all-namespaces

It seems like you're using the kubectl plugin for oh-my-zsh, you can use the following alias

kgew -A

[rinormaloku]

$ kubectl get events --sort-by=".lastTimestamp" --watch --all-namespaces
warning: --watch requested, --sort-by will be ignored for watch events received
NAMESPACE            LAST SEEN   TYPE      REASON              OBJECT                                         MESSAGE
kube-system          85s         Warning   FailedScheduling    pod/coredns-75bb76df-bhnwb                     no nodes available to schedule pods
kube-system          80s         Warning   FailedScheduling    pod/coredns-75bb76df-bhnwb                     no nodes available to schedule pods
local-path-storage   85s         Warning   FailedScheduling    pod/local-path-provisioner-6f6fd5d9d9-r454w    no nodes available to schedule pods
local-path-storage   80s         Warning   FailedScheduling    pod/local-path-provisioner-6f6fd5d9d9-r454w    no nodes available to schedule pods
kube-system          91s         Normal    LeaderElection      lease/kube-controller-manager                  my-cluster_a3dd4663-f1f7-4a79-a5fb-093a927e1cb3 became leader
kube-system          91s         Normal    LeaderElection      lease/kube-scheduler                           my-cluster_03dc3076-fd25-4c1b-b560-987d67189c4b became leader
kube-system          85s         Normal    SuccessfulCreate    replicaset/coredns-75bb76df                    Created pod: coredns-75bb76df-bhnwb
kube-system          85s         Normal    ScalingReplicaSet   deployment/coredns                             Scaled up replica set coredns-75bb76df from 0 to 1
kube-system          85s         Normal    LeaderElection      lease/vcluster-control-plane                   my-cluster-vcluster-control-plane became leader
local-path-storage   85s         Normal    SuccessfulCreate    replicaset/local-path-provisioner-6f6fd5d9d9   Created pod: local-path-provisioner-6f6fd5d9d9-r454w
local-path-storage   85s         Normal    ScalingReplicaSet   deployment/local-path-provisioner              Scaled up replica set local-path-provisioner-6f6fd5d9d9 from 0 to 1

These are the events

[Didier-Lane]

Thanks, can you please provide the output of the following commands

Display the nodes

kubectl get nodes -o wide

Describe the coredns pod

kubectl -n kube-system describe pod/coredns-75bb76df-bhnwb

Describe the local-path-provisioner pod

kubectl -n local-path-storage describe pod/local-path-provisioner-6f6fd5d9d9-r454w

Note

Make sure that pod/coredns-75bb76df-bhnwb reflects the name of the current coredns pod name, same for the local-path-provisioner pod ☝️

[karanpreetsingh1990]

I see the same issue on my end. Sharing the details requested above:

> k get nodes -o wide
NAME             STATUS   ROLES                  AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
argos-vcluster   Ready    control-plane,master   8m48s   v1.35.0   172.18.0.2    <none>        Ubuntu 24.04.4 LTS   6.19.6-2-cachyos   containerd://2.1.6
worker-1         Ready    <none>                 8m38s   v1.35.0   172.18.0.3    <none>        Ubuntu 24.04.4 LTS   6.19.6-2-cachyos   containerd://2.1.6

NAMESPACE            NAME                                      READY   STATUS    RESTARTS   AGE
kube-flannel         kube-flannel-ds-74wml                     1/1     Running   0          11m
kube-flannel         kube-flannel-ds-g66m5                     1/1     Running   0          11m
kube-system          coredns-79cf5f4c56-7jtbm                  0/1     Pending   0          11m
kube-system          kube-proxy-kmn4c                          1/1     Running   0          11m
kube-system          kube-proxy-p2865                          1/1     Running   0          11m
local-path-storage   local-path-provisioner-5b9b9995f4-p8jc9   0/1     Pending   0          11m

Output of core-dns describe:

Name:             coredns-79cf5f4c56-7jtbm
Namespace:        kube-system
Priority:         0
Service Account:  coredns
Node:             <none>
Labels:           k8s-app=vcluster-kube-dns
                  pod-template-hash=79cf5f4c56
Annotations:      <none>
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    ReplicaSet/coredns-79cf5f4c56
Containers:
  coredns:
    Image:      coredns/coredns:1.14.1
    Port:       <none>
    Host Port:  <none>
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      cpu:     1
      memory:  170Mi
    Requests:
      cpu:        20m
      memory:     64Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /etc/coredns/custom from custom-config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-thb4p (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  custom-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns-custom
    Optional:  true
  kube-api-access-thb4p:
    Type:                     Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:   3607
    ConfigMapName:            kube-root-ca.crt
    Optional:                 false
    DownwardAPI:              true
QoS Class:                    Burstable
Node-Selectors:               kubernetes.io/os=linux
Tolerations:                  node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded for selector k8s-app=vcluster-kube-dns
Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  7m58s                  default-scheduler  no nodes available to schedule pods
  Warning  FailedScheduling  7m54s (x2 over 7m57s)  default-scheduler  no nodes available to schedule pods
  Warning  FailedScheduling  2m34s                  default-scheduler  0/2 nodes are available: 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.

Output of local-path-provisioner describe:

Name:             local-path-provisioner-5b9b9995f4-p8jc9
Namespace:        local-path-storage
Priority:         0
Service Account:  local-path-provisioner-service-account
Node:             <none>
Labels:           app=local-path-provisioner
                  pod-template-hash=5b9b9995f4
Annotations:      <none>
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    ReplicaSet/local-path-provisioner-5b9b9995f4
Containers:
  local-path-provisioner:
    Image:      rancher/local-path-provisioner:v0.0.31
    Port:       <none>
    Host Port:  <none>
    Command:
      local-path-provisioner
      --debug
      start
      --config
      /etc/config/config.json
    Environment:
      POD_NAMESPACE:      local-path-storage (v1:metadata.namespace)
      CONFIG_MOUNT_PATH:  /etc/config/
    Mounts:
      /etc/config/ from config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-66gwr (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      local-path-config
    Optional:  false
  kube-api-access-66gwr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  8m28s                  default-scheduler  no nodes available to schedule pods
  Warning  FailedScheduling  8m23s (x2 over 8m26s)  default-scheduler  no nodes available to schedule pods
  Warning  FailedScheduling  3m4s                   default-scheduler  0/2 nodes are available: 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.

[soumen321]

I already share the same issue