Operator could not create Keeper Pods

[yildizozgur]

Hi,
I have deployed altinity-clickhouse-operator:0.25.6 and deployed altinity clickhouse helm-chart with enabled clickhouse installation(CHI) and keeper(CHK) to manage repliacation between clickhouse pods. Operator and CR deployments are in same k8s cluster but in different namespaces. Operator deploys ClickhouseInstallation successfully. However it does not deploy Keeper pods.I have checked operator logs, events, and states of resources. There is no significant information or issue to point out issue. Keeper states InProgress. It seems Operator couldnot or does not try to deploy. Do you know whether there is some configuration is missing or is this an operator/helmchart bug.

❯ k get clickhousekeeperinstallations.clickhouse-keeper.altinity.com -w NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE SUSPEND sentry-dev-clickhouse-keeper 1 1 InProgress 2m32s

`❯ kgp
NAME READY STATUS RESTARTS AGE
chi-sentry-dev-clickhouse-sentry-any-dev-0-0-0 1/1 Running 0 11m

❯ k get clickhouseinstallation
NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE SUSPEND
sentry-dev-clickhouse 1 1 Completed 12m
`

Operator Logs -->
❯ k logs -n obe-clickhouse-operator-qa clickhouse-operator-5bbfbd6c65-fm4wz | grep -i chk Defaulted container "altinity-clickhouse-operator" out of: altinity-clickhouse-operator, metrics-exporter common: chk/keeper_config.d host: chk/conf.d user: chk/users.d common: chk/keeper_config.d host: chk/conf.d user: chk/users.d common: /etc/clickhouse-operator/chk/keeper_config.d host: /etc/clickhouse-operator/chk/conf.d user: /etc/clickhouse-operator/chk/users.d common: /etc/clickhouse-operator/chk/keeper_config.d host: /etc/clickhouse-operator/chk/conf.d user: /etc/clickhouse-operator/chk/users.d I0113 22:43:08.886470 1 worker-reconciler-chk.go:49] worker-reconciler-chk.go:49:reconcileCR():start:unknown I0113 22:43:10.524836 1 worker-reconciler-chk.go:102] unknown:IPs of the CR obe-sentry-dev/sentry-dev-clickhouse-keeper: len: 0 [] I0113 22:43:10.524956 1 worker-reconciler-chk.go:154] logSWVersion():Host:0-keeper-0[0/0]:obe-sentry-dev/sentry-dev-clickhouse-keeper:Host software version: 0-keeper-0 99.99.99[99.99.99/so far so] I0113 22:43:10.524971 1 worker-reconciler-chk.go:157] logSWVersion():unknown:CR software versions [min, max]: 99.99.99[99.99.99/] 99.99.99[99.99.99/so far so] I0113 22:43:10.532506 1 worker-reconciler-chk.go:55] unknown: I0113 22:43:10.532548 1 worker-reconciler-chk.go:59] reconcileCR():unknown:ActionPlan has actions - continue reconcile

keeper-describe.yaml

[joshleecreates]

Can you share your values.yaml? Thanks

[gbawade]

Facing the same. I think Clickhouse-eks doesnt have keeper and standalone clickhouse doesnt create the pods either with the operator

nameOverride: ""
fullnameOverride: "clickhouse-sentry-dev"

namespaceDomainPattern: ""

clickhouse:
  defaultUser:
    password: "REDACTED_PASSWORD"
    password_secret_name: ""
    allowExternalAccess: true
    hostIP: 127.0.0.1/32

  users: []
  profiles: {}
  settings: {}

  replicasCount: 1
  shardsCount: 1

  clusterSecret:
    enabled: false
    secure: false
    auto: true
    value: ""
    valueFrom:
      secretKeyRef:
        name: ""
        key: "secret"

  zones:
    - us-east-1a
    - us-east-1b
    - us-east-1c

  antiAffinity: true
  antiAffinityScope: "ClickHouseInstallation"

  keeper:
    host: ""
    port: 2181

  persistence:
    enabled: true
    size: 100Gi
    accessMode: ReadWriteOnce
    storageClass: "gp3-sc"
    logs:
      enabled: false
      size: 10Gi
      accessMode: ReadWriteOnce

  image:
    repository: altinity/clickhouse-server
    pullPolicy: IfNotPresent
    tag: "25.3.6.10034.altinitystable"

  service:
    type: ClusterIP
    serviceAnnotations: {}
    serviceLabels: {}

  lbService:
    enabled: false
    loadBalancerSourceRanges: []
    serviceAnnotations: {}
    serviceLabels: {}

  imagePullSecrets: []

  serviceAccount:
    create: false
    annotations: {}
    name: ""

  podAnnotations: {}
  podLabels: {}
  podSecurityContext: {}
  securityContext: {}
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  extraPorts: []

  extraConfig: |
    <clickhouse>
    </clickhouse>
  extraUsers: |
    <clickhouse>
    </clickhouse>

  configurationFiles: {}
  extraContainers: []
  extraVolumes: []
  extraVolumeMounts: []

  initScripts:
    enabled: false
    configMapName: ""
    alwaysRun: true
  resources: {}

keeper:
  enabled: true
  replicaCount: 3
  image: "altinity/clickhouse-keeper"
  tag: "25.3.6.10034.altinitystable"
  settings: {}
  localStorage:
    size: 5Gi
    storageClass: "gp3-sc"
  nodeSelector: {}
  tolerations: []
  podAnnotations: {}
  volumeClaimAnnotations: {}
  zoneSpread: false
  metricsPort: ""
  resources:
    cpuRequestsMs: 100
    memoryRequestsMiB: 512Mi
    cpuLimitsMs: 500
    memoryLimitsMiB: 1Gi
  
  podSecurityContext: {}
  securityContext: {}

operator:
  enabled: true