Secure Kubernetes Platform on GKE, with an AI Triage Agent

Archived lab record. Raw shutdown exports have been withdrawn and textual cloud identifiers replaced with example values. Historical screenshots may still show retired environment details. See privacy cleanup.

A Kubernetes platform on GKE, built to explore what it takes to run workloads securely and reliably. Terraform builds the cluster, GitHub Actions deploys to it, and two web apps run behind GKE Gateway. A part counts as done once its failure and recovery are recorded, not only its success. The worklogs hold the commands and their output.

With the platform in place, the next question was whether an agent could help operate it. Security Command Center had been raising findings that nobody read, so an AI triage agent now works through them and records its verdicts. It has no Kubernetes API access and cannot change the cluster.

Note

Closed on 2026-09-22. Shut down on 2026-09-25. The infrastructure and the sindrg.com DNS records are deleted. The implementation is retained with example cloud identifiers after privacy cleanup. See the shutdown worklog.

At a glance

Layer	Built with
Cluster	GKE Standard, private Shielded nodes in three zones, DNS-only control plane, Dataplane V2, Workload Identity
Kubernetes	Deployments, HPA, PodDisruptionBudgets, topology spread, ResourceQuota, LimitRange, namespaced RBAC
Infrastructure	Terraform, custom VPC, Cloud NAT, Artifact Registry
Edge	GKE Gateway, global load balancer, managed TLS, Cloud Armor, Cloudflare DNS
Delivery	GitHub Actions, Workload Identity Federation, Artifact Registry, deploys by digest
Workload security	Pod Security restricted, default-deny NetworkPolicies, Checkov, kube-linter, Dependabot
AI agent	Security Command Center, Pub/Sub, Vertex AI gemini-2.5-flash, a Go worker
Evidence	Cloud Monitoring, k6 load tests, failure drills, worklogs with the command behind each number

Merge to Ready	Peak load, no failures	Connection failures in a rollout	Finding to verdict	Threat model
59 s	125 rps on 8 replicas	0, from 72	2.9 s	11 of 12 closed

Sources are in Measured and the threat model.

Start here

Time	Read
2 min	This page, down to Kubernetes
15 min	Networking, then the rollouts and autoscaling worklogs
Kubernetes	Kubernetes concepts, using the objects this cluster runs, and kubectl
Security	IAM and federation, threat model
AI agent	The triage worker, then the model scope and triage idempotency decisions
Proof	The worklogs, with the commands and their output
Failures	Troubleshooting, indexed by symptom

Architecture

Terraform builds everything in Google Cloud: a custom VPC, private nodes across three zones, Cloud NAT, and a control plane reachable only by DNS. The cluster is drawn in more detail below.

Kubernetes

Cluster

flowchart TB
    lb["Global load balancer<br/>Google ranges 130.211.0.0/22, 35.191.0.0/16"]
    gh["GitHub Actions<br/>k8-lab-deploy identity"]
    apis["Google APIs<br/>Pub/Sub, Vertex AI"]

    subgraph cluster["GKE Standard, control plane in europe-north1-a, DNS endpoint only"]
        subgraph pool["Node pool general: 2 to 3 e2-standard-2, zones a, b, c"]
            subgraph demo["demo namespace: restricted, default deny, quota"]
                gw["Gateway external<br/>HTTPRoutes"]
                nginx["nginx<br/>2 replicas, PDB"]
                sky["sky<br/>2 to 8 replicas, HPA, PDB"]
            end
            subgraph agents["agents namespace: restricted, default deny, quota"]
                worker["triage-worker<br/>1 replica, Recreate"]
            end
        end
    end

    lb -- ":8080 to Pod IPs, NEG" --> nginx
    lb -- ":8080 to Pod IPs, NEG" --> sky
    gw -. "programs" .-> lb
    gh -- "Role deploy: patch Deployments, smoke Pod" --> demo
    worker -- "Workload Identity, :443" --> apis

    classDef ext fill:#FEF7E0,stroke:#E37400,stroke-width:1.5px,color:#3C2A00
    classDef gcp fill:#E8F0FE,stroke:#1A73E8,stroke-width:1.5px,color:#0B2E6B
    classDef k8s fill:#E6F4EA,stroke:#188038,stroke-width:1.5px,color:#0B3D1C
    class gh ext
    class lb,apis gcp
    class gw,nginx,sky,worker k8s
    style cluster fill:none,stroke:#1A73E8,stroke-width:1.5px,color:#1A73E8
    style pool fill:none,stroke:#5F6368,stroke-width:1px,stroke-dasharray:3 3,color:#5F6368
    style demo fill:none,stroke:#188038,stroke-width:1.5px,stroke-dasharray:6 4,color:#188038
    style agents fill:none,stroke:#188038,stroke-width:1.5px,stroke-dasharray:6 4,color:#188038

Loading

Setting	Value	Why
Mode	GKE Standard, zonal control plane	Direct control over nodes, scaling, networking and upgrades. See Cluster
Nodes	One pool of e2-standard-2, 2 to 3 nodes, BALANCED across three zones	Two so an evicted replica has somewhere to land. Three zones after one ran out of capacity
Node image	Container-Optimized OS with containerd, Shielded VMs with secure boot, no external IP	Private nodes reach Google APIs through Private Google Access
Networking	VPC-native, Dataplane V2 (eBPF)	Enforces NetworkPolicy with no extra add-on. See Networking
Control plane	DNS endpoint only, IP endpoints off	IAM authorizes each call, so GitHub Actions needs no bastion
Upgrades	Regular channel, auto-upgrade, surge 1 and unavailable 0, window at 01:00 UTC	A drain happens at night and never removes a node before its replacement exists
Identity	Workload Identity, node metadata server GKE_METADATA	A Pod gets a Google identity from its ServiceAccount, with no key file
Add-ons	Gateway API (standard channel), Managed Service for Prometheus, vulnerability scanning	Installed by GKE, declared in Terraform

Namespaces

Each namespace is a trust boundary. demo holds no Google identity. agents holds one and reaches Google APIs, so it is kept apart.

Control	demo	agents
Pod Security	restricted, enforced, warned and audited	Same
NetworkPolicy	Default deny, then DNS, load balancer ranges to :8080, and labelled clients	Default deny, then DNS, the metadata server, and :443 outside private ranges
ResourceQuota	5 CPU requested, 13 CPU limit, 16 Pods	1 CPU requested, 2 CPU limit, 4 Pods
LimitRange	Defaults for a container that declares none	Same
RBAC	The pipeline's deploy Role: patch Deployments, run a smoke-test Pod	None for the pipeline. An operator applies the worker

The demo quota is sized for sky at its HPA maximum during a rollout: 8 replicas, a surge Pod, two Pods still in their preStop sleep, nginx, and a smoke test.

Workloads

Workload	Replicas	Rollout	Disruption budget	Probes
nginx	2	Rolling, surge 1, unavailable 0	minAvailable: 1	Readiness and liveness on /healthz
sky	2 to 8, HPA at 70% of a 350m CPU request	Rolling, surge 1, unavailable 0	minAvailable: 1	Readiness and liveness on /health
triage-worker	1	Recreate, so two workers never pull at once	None	Readiness on /readyz, liveness on /healthz

Every container runs as non-root with a read-only root filesystem, all capabilities dropped, no privilege escalation, the RuntimeDefault seccomp profile, and no mounted ServiceAccount token. Images are pulled by digest. For nginx and sky, topology spread prefers one replica per node and counts only the current ReplicaSet, so the spread survives a rollout.

Rollouts that drop nothing

A rollout dropped 72 connections at 20 rps while the load balancer detached each old Pod. The preStop sleep took that to zero. The other two changes fixed what the next runs found:

Change	Value	Effect
preStop sleep	35 s, grace period 60 s	A stopping Pod keeps serving until the load balancer lets go. 20 s was not enough: a proxy sent requests 28.6 s after the stop
Backend keep-alive	620 s	Longer than the load balancer's 600 s reuse, so a backend never closes a connection first
One deploy at a time	Concurrency group deploy-demo	Two overlapping rollouts filled the quota with Pods in their preStop sleep and refused the smoke test

GKE also gates each Pod with cloud.google.com/load-balancer-neg-ready, so a Pod counts as ready only once the load balancer agrees. See Phase 12c.

Scaling

Step	Result
2 fixed replicas	40 rps settled, no failures
HPA, quota too small	Capped at 5 of 8 replicas, 60 rps
HPA, one zone	A third node failed with ZONE_RESOURCE_POOL_EXHAUSTED. 4 of 8 replicas, 100 rps
HPA, three zones	8 of 8 replicas, 125 rps, no failures
Spike from two nodes	97 s from the HPA decision to a Pod on a new node, 2m51s to eight available

The fixes were the quota resized for eight replicas, the CPU limit raised to one core for a single-process server, and the node pool spread over three zones. See Phase 12d.

Serving

flowchart LR
    user(["User"])
    dns["Cloudflare DNS<br/>DNSSEC, CAA"]
    lb["Global load balancer<br/>managed TLS, Cloud Armor"]
    gw["GKE Gateway<br/>HTTPRoutes"]
    uptime["Uptime check<br/>3 regions"]

    subgraph demo["demo namespace"]
        nginx["nginx<br/>2 replicas"]
        sky["sky<br/>2 to 8 replicas, HPA"]
    end

    user -. "DNS lookup" .-> dns
    user -- "HTTPS" --> lb
    lb -- "/" --> nginx
    lb -- "/sky, /api, /static" --> sky
    gw -. "configures" .-> lb
    uptime -. "/healthz every 60 s" .-> lb

    classDef actor fill:#FFFFFF,stroke:#5F6368,stroke-width:1.5px,color:#202124
    classDef ext fill:#FEF7E0,stroke:#E37400,stroke-width:1.5px,color:#3C2A00
    classDef gcp fill:#E8F0FE,stroke:#1A73E8,stroke-width:1.5px,color:#0B2E6B
    classDef k8s fill:#E6F4EA,stroke:#188038,stroke-width:1.5px,color:#0B3D1C
    class user actor
    class dns ext
    class lb,uptime gcp
    class gw,nginx,sky k8s
    style demo fill:none,stroke:#188038,stroke-width:1.5px,stroke-dasharray:6 4,color:#188038

Loading

The Gateway programs a global load balancer that sends traffic straight to Pod IPs through network endpoint groups.

Delivery

%%{init: {"sequence": {"mirrorActors": false}}}%%
sequenceDiagram
    autonumber
    actor Dev as Developer
    participant GH as GitHub Actions
    participant WIF as Workload Identity<br/>Federation
    participant AR as Artifact Registry
    participant GKE as GKE

    Dev->>GH: Merge to main
    GH->>WIF: OIDC token, main only
    WIF-->>GH: Short-lived credential
    GH->>AR: Push image, tagged by commit
    AR-->>GH: Digest
    GH->>GKE: Patch Deployment by digest
    GKE-->>GH: Rollout status, smoke test

Loading

No key is stored anywhere. In demo, the pipeline can patch Deployments and run a smoke-test Pod, and can create nothing else. See operations.

AI triage agent

Security Command Center publishes each finding change to Pub/Sub. A worker in the agents namespace counts vulnerability findings and does not triage them, because a vulnerability is fixed by an image pin, not by a decision. It settles every other finding with a reviewed mapping first, and asks Vertex AI only when no mapping matches and the finding is complete. It writes every verdict to an append-only ledger before it notifies anyone.

Speed and accuracy

A finding change became a recorded verdict in 2.9 s on the rules path, most of it spent reaching the worker. See slice 8. The model path adds a model call of 1.3 s at p50, from slice 13.

The model beats the rules alone only once a contradiction has to land on a control that covers the finding's resource. The evaluation is 25 cases, and one person wrote the holdout. See slice 14.

Guardrails

The model	Enforced by
Cannot accept a finding	accepted is absent from the schema, refused by the worker, and rejected by the validator
Cannot act	No tools are declared, and a tool call in the reply is rejected
Cannot overspend	1.00 USD a UTC day. Each call reserves its worst case in the ledger first, so the ceiling survives restarts
Cannot lose a finding to a crash	Record, then notify, then acknowledge. A crash at each boundary was drilled. A message that fails five deliveries is parked on the dead letter topic with its body intact, and an alert fires
Was not steered by an injected instruction	An instruction in the fields a resource's creator controls gave the same verdict as without it. The detector-written fields are covered by unit tests only

The worker holds no Security Command Center permission and no cluster credential. Its code lives in ai-k8s. How it works is in the triage worker reference.

What it does

Platform

Area	What exists	Why	Evidence
Foundation	Custom VPC, private nodes, Cloud NAT, DNS-only control plane, Dataplane V2, zonal GKE Standard	Networking, Cluster	Phase 1
Workloads	nginx and sky in demo, with probes, limits and ClusterIP Services	Configuration	Phase 2
Ingress	GKE Gateway, managed TLS, HTTP to HTTPS redirect, path routing	Ingress and TLS	Phase 7
Observability	Uptime check, dashboard as code, alerts for availability, findings and dead letters	Observability	Phase 8
Resilience	Node floor of two, disruption budgets, maintenance window, two failure drills	Cluster	Phase 9, Phase 10
Load	k6 harness, clean rollouts, HPA on sky, nodes across three zones	Load and scaling	12a, 12b, 12c, 12d

Delivery

Area	What exists	Why	Evidence
CI	Credential-free pull request checks, required on main	Delivery	Phase 3
Images	Private registry, immutable tags, non-root images pulled by digest	Supply chain	Phase 5
Delivery	Keyless deploys scoped to main, namespaced pipeline RBAC, gated rollout	Delivery	Phase 6
Upstream pins	A scheduled workflow proposes each pin bump, and the merge is the review	Pin automation	Notes

Security and the agent

Area	What exists	Why	Evidence
Guardrails	Pod Security restricted, namespace quota, default-deny NetworkPolicies	Workload security	Phase 4
Hardening	Default network removed, vulnerability scanning, Log Analytics	Workload security	Phase 11
Security baseline	Threat model with 12 findings: 11 closed, 1 accepted	Workload security	Phase 13, Phase 14
Finding triage	Findings over Pub/Sub, settled by reviewed mapping then by model, in an append-only ledger	Agents	Phase 15
Patching	Dependabot moves each base image, and a scanner counts what remains	Residual vulnerabilities	Phase 15b

Phases 16 to 19 are optional and were not built. The plan says what would justify each.

Measured

Area	Metric	Result
Delivery	Merge to Ready workload	59 s
Delivery	Deploy duration, median of 12 runs	70.5 s
Observability	Alert detection floor	About 3 min
Cost	Running cost	kr461.81 a week, covered by credits
Load	sky saturation, 2 replicas	40 rps, p95 230 ms
Load	sky saturation, 8 replicas	125 rps, p95 394 ms, no failures
Load	HPA decision to a Pod on a new node	97 s
Rollouts	Failed requests at 20 rps, before preStop	1.14%
Rollouts	Connection failures after preStop	0 of 3 rollouts, from 72
Rollouts	Closed-connection 503s after keep-alive	0 of 7,150, from 6
Triage	Finding change to message on the subscription	About 2 s
Triage	Finding change to recorded verdict, rules path	2.9 s
Triage	Model call p50 and p95, and cost	1.3 s and 1.9 s, about 0.003 USD
Triage	Crash to redelivery, then four ledger states written	About 25 s, then 285 ms
Triage	Right on all 5 runs, rules plus model	16 of 18 dev, 7 of 7 holdout
Triage	Right on all 5 runs, rules alone	14 of 18 dev, 5 of 7 holdout
Triage	False contradictions from the model	0 of 125 runs, from 15
Images	CRITICAL and HIGH vulnerabilities	frontend 0, from 17. sky 6 with no fix, from 22

The command behind each number is in the worklogs.

Known limitations

Limitation	Reason
Terraform state is local	One operator, no automated apply. See the decision gate
Bootstrap order not rehearsed on an empty project	Reconstructed from the worklogs in operations
Zonal control plane	Cost. The node pool spans three zones
agents egress admits any host on 443	NetworkPolicy cannot match hostnames. See boundary 3
Six HIGH vulnerabilities in sky	No fixed package. Accepted
Evaluation is 25 cases, and one person wrote the holdout	A check against tuning, not an independent sample. See slice 13
Event Threat Detection needs Security Command Center Premium	Ran on a trial. Standard drops threat findings

Documentation

Document	Holds
plan.md	What was built, phase by phase, and what stayed optional
decisions.md	Why each choice was made, its cost, and what was rejected
worklog/	The commands and output behind every claim
troubleshooting.md	Failures worth not repeating
loadtest/	The k6 harness
Operations	Where values live, bootstrap order, recovery
Networking	How a request reaches a Pod, and what blocks everything else
IAM and federation	Who can do what, and how GitHub gets a token
Threat model	Trust boundaries and their findings
Triage worker	How a finding becomes a verdict
Kubernetes concepts	How Kubernetes works, using the objects this cluster runs
kubectl	Commands to operate and inspect the platform
Diagram renderer	Renders the README images from their HTML sources in scripts/diagrams/