在k8s部署kuscia中心模式，执行测试任务一直running

[magic-hya]

Issue Type

Running

Search for existing issues similar to yours

Yes

OS Platform and Distribution

CentOS Linux 7

Kuscia Version

latest

Deployment

k8s

deployment Version

k8s 1.22.2

App Running type

secretflow

App Running version

latest

Configuration file used to run kuscia.

参照官方k8s部署中心化kuscia配置部署

What happend and What you expected to happen.

## 任务发起成功
[root@kuscia-master-76c5b5bc7b-s84k8 kuscia]# scripts/user/create_example_job.sh
/home/kuscia /home/kuscia
With JOB_EXAMPLE=PSI, job via APP_IMAGE=secretflow-image creating ...
kusciajob.kuscia.secretflow/secretflow-task-20240620111611 created
Job 'secretflow-task-20240620111611' created successfully. You can use the following command to display job status:
  kubectl get kj -n cross-domain

## 查看任务一直在运行
[root@kuscia-master-76c5b5bc7b-s84k8 kuscia]# kubectl get kj -n cross-domain
NAME                             STARTTIME   COMPLETIONTIME   LASTRECONCILETIME   PHASE
secretflow-task-20240620111611   12s                          11s                 Running

Kuscia log output.

## 查看alice和bob pod运行状况有error
[root@kuscia-master-76c5b5bc7b-s84k8 kuscia]# kubectl logs secretflow-task-20240620111611-single-psi-0 -n alice
Error from server: Get "https://192.168.62.11:10250/containerLogs/alice/secretflow-task-20240620111611-single-psi-0/secretflow": proxy error from 0.0.0.0:6443 while dialing 192.168.62.11:10250, code 502: 502 Bad Gateway

[root@kuscia-master-76c5b5bc7b-s84k8 kuscia]# kubectl logs secretflow-task-20240620111611-single-psi-0 -n bob
Error from server: Get "https://192.168.62.12:10250/containerLogs/bob/secretflow-task-20240620111611-single-psi-0/secretflow": proxy error from 0.0.0.0:6443 while dialing 192.168.62.12:10250, code 502: 502 Bad Gateway

[zimu-yuxi]

alice和bob的路由看下呢？
kubectl get cdr -A

[magic-hya]

[root@kuscia-master-76c5b5bc7b-s84k8 kuscia]# kubectl get cdr -A
NAME SOURCE DESTINATION HOST AUTHENTICATION READY
alice-bob alice bob kuscia-lite-bob.lite-bob.svc.cluster.local Token True
bob-alice bob alice kuscia-lite-alice.lite-alice.svc.cluster.local Token True
alice-kuscia-system alice kuscia-system Token True
bob-kuscia-system bob kuscia-system Token True

[zimu-yuxi]

[root@kuscia-master-76c5b5bc7b-s84k8 kuscia]# kubectl get cdr -A NAME SOURCE DESTINATION HOST AUTHENTICATION READY alice-bob alice bob kuscia-lite-bob.lite-bob.svc.cluster.local Token True bob-alice bob alice kuscia-lite-alice.lite-alice.svc.cluster.local Token True alice-kuscia-system alice kuscia-system Token True bob-kuscia-system bob kuscia-system Token True

/home/kuscia/var/stdout目录下看下有没有相关的日志输出

[magic-hya]

目录下没有内容
[root@kuscia-master-76c5b5bc7b-s84k8 stdout]# pwd
/home/kuscia/var/stdout
[root@kuscia-master-76c5b5bc7b-s84k8 stdout]# ls -l
total 0

[magic-hya]

我想进bob容器看一看的，也报这个错，这错误是不是有关联
[root@kuscia-master-76c5b5bc7b-s84k8 stdout]# kubectl exec -it secretflow-task-20240620111611-single-psi-0 -n bob -- bash
Error from server: error dialing backend: proxy error from 0.0.0.0:6443 while dialing 192.168.62.12:10250, code 502: 502 Bad Gateway

[aokaokd]

请检查下你的网关是否配置了安全策略。
也可以看下bob 容器的状态

kubectl describe pod   {bob容器} -n  xxx

[magic-hya]

[root@kuscia-master-76c5b5bc7b-s84k8 stdout]# kubectl describe pod secretflow-task-20240620111611-single-psi-0 -n bob
Name:             secretflow-task-20240620111611-single-psi-0
Namespace:        bob
Priority:         0
Service Account:  default
Node:             kuscia-lite-bob-69bd6df646-k8krs/192.168.62.12
Start Time:       Thu, 20 Jun 2024 11:17:57 +0800
Labels:           kuscia.secretflow/communication-role-client=true
                  kuscia.secretflow/communication-role-server=true
                  kuscia.secretflow/controller=kusciatask
                  kuscia.secretflow/pod-identity=0cddc892-e259-4af0-b393-fc7b1d66bf40-0
                  kuscia.secretflow/pod-role=
                  kuscia.secretflow/task-resource-uid=56940863-4cf4-4978-81b0-2483eb75f62f
                  kuscia.secretflow/task-uid=0cddc892-e259-4af0-b393-fc7b1d66bf40
Annotations:      kuscia.secretflow/config-template-volumes: config-template
                  kuscia.secretflow/image-id: abc
                  kuscia.secretflow/initiator: alice
                  kuscia.secretflow/task-id: secretflow-task-20240620111611-single-psi
                  kuscia.secretflow/task-resource: secretflow-task-20240620111611-single-psi-f6623141c609
                  kuscia.secretflow/task-resource-group: secretflow-task-20240620111611-single-psi
Status:           Pending
IP:
IPs:              <none>
Containers:
  secretflow:
    Container ID:
    Image:         secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/secretflow-lite-anolis8:1.6.0b0
    Image ID:
    Ports:         21320/TCP, 21321/TCP, 21316/TCP, 21317/TCP, 21318/TCP, 21319/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      sh
    Args:
      -c
      python -m secretflow.kuscia.entry ./kuscia/task-config.conf
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      KUSCIA_DOMAIN_ID:                   bob
      TASK_ID:                            secretflow-task-20240620111611-single-psi
      TASK_CLUSTER_DEFINE:                {"parties":[{"name":"bob","role":"","services":[{"portName":"spu","endpoints":["secretflow-task-20240620111611-single-psi-0-spu.bob.svc"]},{"portName":"fed","endpoints":["secretflow-task-20240620111611-single-psi-0-fed.bob.svc"]},{"portName":"global","endpoints":["secretflow-task-20240620111611-single-psi-0-global.bob.svc:21316"]}]},{"name":"alice","role":"","services":[{"portName":"spu","endpoints":["secretflow-task-20240620111611-single-psi-0-spu.alice.svc"]},{"portName":"fed","endpoints":["secretflow-task-20240620111611-single-psi-0-fed.alice.svc"]},{"portName":"global","endpoints":["secretflow-task-20240620111611-single-psi-0-global.alice.svc:21565"]}]}],"selfPartyIdx":0,"selfEndpointIdx":0}
      ALLOCATED_PORTS:                    {"ports":[{"name":"spu","port":21320,"scope":"Cluster","protocol":"GRPC"},{"name":"fed","port":21321,"scope":"Cluster","protocol":"GRPC"},{"name":"global","port":21316,"scope":"Domain","protocol":"GRPC"},{"name":"node-manager","port":21317,"scope":"Local","protocol":"GRPC"},{"name":"object-manager","port":21318,"scope":"Local","protocol":"GRPC"},{"name":"client-server","port":21319,"scope":"Local","protocol":"GRPC"}]}
      TASK_INPUT_CONFIG:                  {"sf_datasource_config":{"alice":{"id":"default-data-source"},"bob":{"id":"default-data-source"}},"sf_cluster_desc":{"parties":["alice","bob"],"devices":[{"name":"spu","type":"spu","parties":["alice","bob"],"config":"{\"runtime_config\":{\"protocol\":\"REF2K\",\"field\":\"FM64\"},\"link_desc\":{\"connect_retry_times\":60,\"connect_retry_interval_ms\":1000,\"brpc_channel_protocol\":\"http\",\"brpc_channel_connection_type\":\"pooled\",\"recv_timeout_ms\":1200000,\"http_timeout_ms\":1200000}}"},{"name":"heu","type":"heu","parties":["alice","bob"],"config":"{\"mode\": \"PHEU\", \"schema\": \"paillier\", \"key_size\": 2048}"}],"ray_fed_config":{"cross_silo_comm_backend":"brpc_link"}},"sf_node_eval_param":{"domain":"data_prep","name":"psi","version":"0.0.4","attr_paths":["input/receiver_input/key","input/sender_input/key","protocol","precheck_input","bucket_size","curve_type","left_side"],"attrs":[{"ss":["id1"]},{"ss":["id2"]},{"s":"PROTOCOL_ECDH"},{"b":true},{"i64":"1048576"},{"s":"CURVE_FOURQ"},{"is_na":false,"ss":["alice"]}]},"sf_input_ids":["alice-table","bob-table"],"sf_output_ids":["psi-output"],"sf_output_uris":["psi-output.csv"]}
      KUSCIA_PORT_SPU_NUMBER:             21320
      KUSCIA_PORT_FED_NUMBER:             21321
      KUSCIA_PORT_GLOBAL_NUMBER:          21316
      KUSCIA_PORT_NODE_MANAGER_NUMBER:    21317
      KUSCIA_PORT_OBJECT_MANAGER_NUMBER:  21318
      KUSCIA_PORT_CLIENT_SERVER_NUMBER:   21319
    Mounts:
      /root/kuscia/task-config.conf from config-template (rw,path="task-config.conf")
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config-template:
    Type:        ConfigMap (a volume populated by a ConfigMap)
    Name:        secretflow-task-20240620111611-single-psi-configtemplate
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kuscia.secretflow/namespace=bob
Tolerations:     kuscia.secretflow/agent:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

[magic-hya]

其实还有个地方我有疑问的，在配置lite的runtime配置runk时，我使用了默认的kubeconfigFile
然后我也没有配置RBAC，不知道是否会影响到

    # 当 runtime 为 runk 时配置
    runk:
      # 任务调度到指定的机构 k8s namespace 下
      namespace: lite-bob
      # 机构 k8s 集群的 pod dns 配置, 用于解析节点的应用域名, runk 拉起 pod 所使用的 dns 地址，应配置为 kuscia-lite service 的 clusterIP, 此处以 "1.1.1.1" 为例
      dnsServers:
      # - kuscia-dns-lb-server
        - 10.105.200.142
      # k8s 集群的 kubeconfig, 不填默认 serviceaccount; 当前请不填，默认使用 serviceaccount
      kubeconfigFile:

[aokaokd]

你的kubeconfigFile 没有问题，但是需要配置下 RBAC，参考：
RBAC
另外
可以看到bob的 event内没有信息 ,也可以看一下bob的日志

kubectl logs secretflow-task-20240620111611-single-psi-0 -n bob

如果没有日志信息，可以看下k8s节点的状态：

kubectl get nodes

[magic-hya]

[root@kuscia-master-76c5b5bc7b-s84k8 stdout]# kubectl logs secretflow-task-20240620111611-single-psi-0 -n bob
Error from server: Get "https://192.168.62.12:10250/containerLogs/bob/secretflow-task-20240620111611-single-psi-0/secretflow": proxy error from 0.0.0.0:6443 while dialing 192.168.62.12:10250, code 502: 502 Bad Gateway
[root@kuscia-master-76c5b5bc7b-s84k8 stdout]# kubectl get nodes
NAME                                STATUS     ROLES   AGE   VERSION
kuscia-lite-bob-69bd6df646-k8krs    NotReady   agent   29h   578bc84
kuscia-lite-alice-b69cbcc76-f2lg6   NotReady   agent   46h   578bc84

我先把rbac配置了再看看

[yushiqie]

我想进bob容器看一看的，也报这个错，这错误是不是有关联 [root@kuscia-master-76c5b5bc7b-s84k8 stdout]# kubectl exec -it secretflow-task-20240620111611-single-psi-0 -n bob -- bash Error from server: error dialing backend: proxy error from 0.0.0.0:6443 while dialing 192.168.62.12:10250, code 502: 502 Bad Gateway

kuscia 中 pod ，不支持 kubectl logs 和 exec 功能（应安全要求），如果要查看任务 pod 日志，请到 lite 节点侧查看 https://www.secretflow.org.cn/zh-CN/docs/kuscia/v0.8.0b0/deployment/logdescription

[magic-hya]

启动了rbac后，执行任务，现在开始报错了

kubectl describe pod secretflow-task-20240620172156-single-psi-0 -n alice

...
Events:
  Type     Reason            Age    From              Message
  ----     ------            ----   ----              -------
  Warning  FailedScheduling  4m29s  kuscia-scheduler  0/2 nodes are available: waiting for task resource. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling., can not find related task resource.
  Warning  FailedScheduling  4m28s  kuscia-scheduler  0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. reject the pod secretflow-task-20240620172156-single-psi-0 even after PostFilter, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
  Warning  FailedScheduling  2m56s  kuscia-scheduler
  Warning  FailedScheduling  2m52s  kuscia-scheduler  0/2 nodes are available: task resource alice/secretflow-task-20240620172156-single-psi-64164e39cac5 status phase is Failed, skip scheduling pod. last failed scheduling result: . preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling., reject the pod secretflow-task-20240620172156-single-psi-0 even after PostFilter.

[aokaokd]

启动了rbac后，执行任务，现在开始报错了

kubectl describe pod secretflow-task-20240620172156-single-psi-0 -n alice

...
Events:
  Type     Reason            Age    From              Message
  ----     ------            ----   ----              -------
  Warning  FailedScheduling  4m29s  kuscia-scheduler  0/2 nodes are available: waiting for task resource. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling., can not find related task resource.
  Warning  FailedScheduling  4m28s  kuscia-scheduler  0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. reject the pod secretflow-task-20240620172156-single-psi-0 even after PostFilter, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
  Warning  FailedScheduling  2m56s  kuscia-scheduler
  Warning  FailedScheduling  2m52s  kuscia-scheduler  0/2 nodes are available: task resource alice/secretflow-task-20240620172156-single-psi-64164e39cac5 status phase is Failed, skip scheduling pod. last failed scheduling result: . preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling., reject the pod secretflow-task-20240620172156-single-psi-0 even after PostFilter.

你的两个节点存在污点，可以执行一下命令看看是什么污点

kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, taints: .spec.taints}'
或者
kubectl describe node kuscia-lite-bob-69bd6df646-k8krs

[magic-hya]

之前容器内的alice和pod的任务节点卡住了，我就用--force删除了，有可能是这个原因

[root@kuscia-master-76c5b5bc7b-s84k8 kuscia]# kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, taints: .spec.taints}'
{
  "name": "kuscia-lite-bob-69bd6df646-k8krs",
  "taints": [
    {
      "effect": "NoSchedule",
      "key": "kuscia.secretflow/agent",
      "value": "v1"
    }
  ]
}
{
  "name": "kuscia-lite-alice-b69cbcc76-f2lg6",
  "taints": [
    {
      "effect": "NoSchedule",
      "key": "kuscia.secretflow/agent",
      "value": "v1"
    },
    {
      "effect": "NoSchedule",
      "key": "node.kubernetes.io/unreachable",
      "timeAdded": "2024-06-20T10:03:45Z"
    },
    {
      "effect": "NoExecute",
      "key": "node.kubernetes.io/unreachable",
      "timeAdded": "2024-06-20T10:03:50Z"
    }
  ]
}

[root@kuscia-master-76c5b5bc7b-s84k8 kuscia]# kubectl describe node kuscia-lite-bob-69bd6df646-k8krs
Name:               kuscia-lite-bob-69bd6df646-k8krs
Roles:              agent
Labels:             beta.kubernetes.io/arch=x86_64
                    beta.kubernetes.io/os=linux
                    domain=bob
                    kubernetes.io/apiVersion=0.26.6
                    kubernetes.io/arch=x86_64
                    kubernetes.io/hostname=kuscia-lite-bob-69bd6df646-k8krs
                    kubernetes.io/os=linux
                    kubernetes.io/role=agent
                    kuscia.secretflow/namespace=bob
                    kuscia.secretflow/runtime=runk
Annotations:        node.alpha.kubernetes.io/ttl: 0
CreationTimestamp:  Wed, 19 Jun 2024 11:22:44 +0800
Taints:             kuscia.secretflow/agent=v1:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  kuscia-lite-bob-69bd6df646-k8krs
  AcquireTime:     <unset>
  RenewTime:       Thu, 20 Jun 2024 18:09:32 +0800
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                  Message
  ----                 ------  -----------------                 ------------------                ------                  -------
  NetworkUnavailable   False   Wed, 19 Jun 2024 11:22:44 +0800   Wed, 19 Jun 2024 11:22:44 +0800   RouteCreated            RouteController created a route
  PIDPressure          False   Wed, 19 Jun 2024 11:22:44 +0800   Wed, 19 Jun 2024 11:22:44 +0800   AgentHasSufficientPID   Agent has sufficient PID available
  Ready                True    Thu, 20 Jun 2024 18:07:49 +0800   Thu, 20 Jun 2024 17:17:54 +0800   AgentReady              Agent is ready
Addresses:
  InternalIP:  192.168.62.12
Capacity:
  cpu:      4
  memory:   4Gi
  pods:     500
  storage:  100Gi
Allocatable:
  cpu:      4
  memory:   4Gi
  pods:     500
  storage:  100Gi
System Info:
  Machine ID:                 c0e61932-d279-5b11-bf44-42bf1a8a4f09
  System UUID:
  Boot ID:                    1709087859-1718767364341055751
  Kernel Version:             5.4.181-1.el7.elrepo.x86_64
  OS Image:                   docker://linux/anolis:23 (guest)
  Operating System:           linux
  Architecture:               x86_64
  Container Runtime Version:
  Kubelet Version:            578bc84
  Kube-Proxy Version:
PodCIDR:                      10.42.1.0/24
PodCIDRs:                     10.42.1.0/24
Non-terminated Pods:          (0 in total)
  Namespace                   Name    CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----    ------------  ----------  ---------------  -------------  ---
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests  Limits
  --------           --------  ------
  cpu                0 (0%)    0 (0%)
  memory             0 (0%)    0 (0%)
  ephemeral-storage  0 (0%)    0 (0%)
  storage            0         0
Events:              <none>

[magic-hya]

删除污点后，重新启动任务发现2个问题分别是：
bob节点

[root@kuscia-master-76c5b5bc7b-s84k8 kuscia]# kubectl get pods -n bob
NAME                                          READY   STATUS             RESTARTS   AGE
secretflow-task-20240621113741-single-psi-0   0/1     ImagePullBackOff   0          4m39s

Events:
  Type     Reason            Age    From              Message
  ----     ------            ----   ----              -------
  Warning  FailedScheduling  4m35s  kuscia-scheduler  0/2 nodes are available: waiting for task resource. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling., can not find related task resource.
  Normal   Scheduled         4m12s  kuscia-scheduler  Successfully assigned bob/secretflow-task-20240621113741-single-psi-0 to kuscia-lite-bob-69bd6df646-k8krs

alice节点

[root@kuscia-master-76c5b5bc7b-s84k8 kuscia]# kubectl get pods -n alice
NAME                                          READY   STATUS              RESTARTS   AGE
secretflow-task-20240621113741-single-psi-0   0/1     ContainerCreating   0          5m53s

Events:
  Type     Reason                  Age    From              Message
  ----     ------                  ----   ----              -------
  Warning  FailedScheduling        2m52s  kuscia-scheduler  0/2 nodes are available: waiting for task resource. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling., can not find related task resource.
  Warning  FailedScheduling        2m51s  kuscia-scheduler  0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling., reject the pod secretflow-task-20240621113741-single-psi-0 even after PostFilter.
  Normal   Scheduled               2m29s  kuscia-scheduler  Successfully assigned alice/secretflow-task-20240621113741-single-psi-0 to kuscia-lite-alice-b69cbcc76-f2lg6
  Warning  FailedCreatePodSandBox  2m28s  Agent             Failed to sync pod to k8s: failed to create configmap secretflow-task-20240621113741-single-psi-0-secretflow-task-20240621113741-single-psi-configtemplate, detail-> failed to create resource(*v1.ConfigMap) secretflow-task-20240621113741-single-psi-0-secretflow-task-20240621113741-single-psi-configtemplate, detail-> configmaps is forbidden: User "system:serviceaccount:lite-alice:default" cannot create resource "configmaps" in API group "" in the namespace "lite-alice"