Headline: 3-node Kubernetes (kind) HA deployment by default

[dshapi]

v1.0.1 — May 2026

This release moves dev from a single-node kind cluster to a
production-shaped HA topology that mirrors the prod target one-for-one.
A single ./deploy/scripts/bootstrap-cluster.sh brings up:

• 3 control-plane Kubernetes nodes running on Docker Desktop via
  kind. No worker nodes — control-plane taints lifted on dev so
  application pods can schedule cluster-wide.
• CNPG (CloudNativePG) Postgres with 1 primary + 2 replicas,
  automatic failover, streaming replication, and a spm-db-rw
  Service that always points to the current primary.
• Bitnami Redis Sentinel with 3 nodes for cache + session HA.
• MinIO distributed object storage (4 servers, 4 disks) backing
  Flink checkpoints and other large blobs. Replaces Longhorn.
• Istio service mesh with sidecar injection, mTLS PeerAuthentication,
  AuthorizationPolicies per service, and a single ingress gateway at
  https://aispm.local (port 443, browser-trusted via mkcert).
• gVisor (runsc) RuntimeClass as the default for customer-uploaded
  agent pods — per-pod kernel sandboxing for untrusted code.
• Local container registry at localhost:5001, mirrored into every
  kind node so dev image rebuilds land in seconds.

The chart applies in 6 phased tiers (infra → data → data-init →
platform → compute → frontend), serially, with auto-recovery from
immutable-PVC errors, failed Jobs, and stale registry state.
A single canonical seeder (scripts/seed_all.py) populates models,
posture history, integrations, cases, alerts, and policies via one
idempotent K8s Job.

Bootstrap & operability

• One entry point: ./deploy/scripts/bootstrap-cluster.sh. Helper
  scripts (kind-cluster.sh, kind-databases-ha.sh,
  install-gvisor.sh, build-images.sh) are now invoked from
  bootstrap; never directly.
• Cluster-destroy prompt at the start. FORCE_DESTROY /
  FORCE_KEEP / FORCE_CREATE env overrides for CI.
• Verbose-by-default: every step prints what it does. xtrace goes to
  /tmp/bootstrap-xtrace-<pid>.log on bash 4.1+.
• Phased apply runs serially to keep kind-on-Docker load manageable.
• Auto-recovery from immutable-PVC errors, failed/stuck Jobs, stuck
  pod templates, stale Docker registry containers, and racing
  cert-manager Certificate reconcile loops.
• .env keys (Postgres password, Anthropic, Tavily, Groq, etc.)
  auto-merge into platform-secrets every bootstrap — no more
  re-entering them in the Integrations UI after a rebuild.

TLS / WSS

• Chart's aispm-tls-certificate.yaml now gates on
  ingress.certManager, so cert-manager doesn't fight the
  bootstrap's mkcert Secret in dev.
• Bootstrap actively repairs the WSS cert chain on every run —
  re-upserts the mkcert Secret, restarts istio-ingressgateway,
  verifies the wire issuer is mkcert development CA. Safari
  WebSocket Secure connections now work end-to-end with no manual
  steps.

Schema drift fixes (alembic 010 / 011 / 012)

• New agent_kind enum + agents.kind column.
• agents.risk migrated from risk_level to model_risk_tier;
  model_risk_tier extended with low/medium/critical.
• agents.policy_status migrated from policy_status enum to
  policy_coverage; values mapped (covered → full).
• model_provider extended with aws, azure, gcp, internal.
• All migrations idempotent across fresh, hand-patched, and re-run
  database states.

Resource sizing

• spm-api ships with explicit 2Gi memory limits (was inheriting
  the namespace LimitRange's 512Mi and getting OOMKilled every
  ~8 minutes).
• db-seed Job memory bumped to 2Gi.

macOS dev caveat

gVisor's runsc doesn't work on Docker Desktop's Linuxkit kernel —
sandbox init crashes. values.dev.yaml overrides
agentRuntime.runtimeClassName to "" so agent pods use runc on
Mac. On a Linux dev host or in prod the gVisor sandbox is enforced
unchanged.

---

This discussion was created from the release Headline: 3-node Kubernetes (kind) HA deployment by default.