Skip to content

AFTER.md

Latest commit

 

History

History
155 lines (113 loc) · 5.48 KB

AFTER.md

File metadata and controls

155 lines (113 loc) · 5.48 KB
Raw

NeuroScale — After: The Transformed Platform

What exists now — a self-service, policy-enforced, GitOps-driven AI inference platform.

The Platform Is Finished

NeuroScale is now a production-hardened self-service AI inference platform. A developer fills in a Backstage form, the platform creates a pull request, CI validates it against schema and policy rules, ArgoCD deploys it through GitOps, and a KServe inference endpoint is live — with cost attribution, drift control, and policy guardrails enforced automatically at every stage.

21 verified checks across 6 milestones. 0 failures. Deterministic. Repeatable. On any machine.

What Works Now

1. Self-Service Golden Path

Developer fills Backstage form
  → PR created automatically (apps/<name>/inference-service.yaml)
  → CI validates schema + policies + resource delta
  → Merge triggers ArgoCD sync
  → ApplicationSet auto-discovers new model folder
  → KServe InferenceService reaches Ready=True
  → Prediction endpoint live

No kubectl. No YAML editing. No tribal knowledge. One form, one PR, one working endpoint.

2. GitOps Drift Control

$ kubectl delete deploy nginx-test -n default
# Wait 20 seconds...
$ kubectl get deploy nginx-test -n default
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
nginx-test   1/1     1            1           8s   ← auto-recreated by ArgoCD

ArgoCD continuously reconciles. Manual cluster changes are automatically reverted within seconds. Git is the single source of truth. Drift is impossible.

3. Policy Guardrails (Shift-Left + Shift-Down)

Admission-time (Kyverno):

$ kubectl apply -f bad-model.yaml
Error from server: admission webhook "validate.kyverno.svc" denied the request:
  InferenceService resources must set metadata.labels.owner and metadata.labels.cost-center

PR-time (CI):

Guardrails Checks — Policy Simulation
| Check                    | Result     |
|--------------------------|------------|
| Kyverno policy simulation | ❌ failure |
→ PR blocked. Unsafe workloads cannot merge.

Five enforced policies: required labels, required resource limits, no :latest tags, no root containers.

4. Stable Inference Endpoints

$ curl -sS -H "Content-Type: application/json" \
  -d '{"instances":[[6.8,2.8,4.8,1.4],[6.0,3.4,4.5,1.6]]}' \
  http://127.0.0.1:8082/v1/models/demo-iris-2:predict

{"predictions":[1,1]}

KServe with Kourier ingress. Working. Reproducible. Sub-200MB memory footprint vs Istio's 1GB+.

5. Automated CI Pipeline

Every PR is validated by:

Check Tool Purpose
Schema validation kubeconform Catches malformed YAML before merge
Policy simulation kyverno-cli Catches policy violations before merge
Helm rendering helm template Catches Helm values hierarchy bugs
Resource delta Python + PyYAML Shows CPU/memory impact as PR comment

6. Cost Attribution

Every workload carries owner and cost-center labels (enforced by Kyverno). OpenCost reads these labels via Prometheus and provides per-team cost breakdowns.

7. Operational Recovery

# ArgoCD repo-server recovery (documented runbook)
kubectl -n argocd rollout restart deploy/argocd-repo-server
kubectl -n argocd rollout status deploy/argocd-repo-server --timeout=120s
# → All applications recover within 3 minutes

Documented runbooks for every failure mode encountered during development. The platform is operable under failure.

8. One-Command Bootstrap

$ bash scripts/bootstrap.sh
# 5 minutes later: entire platform running on any machine with Docker + k3d

9. Deterministic Smoke Test

$ bash scripts/smoke-test.sh

━━━ Milestone A — GitOps Spine ━━━
  [✓ PASS] All ArgoCD pods are Running
  [✓ PASS] ArgoCD Applications: 7/7 Healthy and Synced
  [✓ PASS] Drift self-heal: nginx-test recreated in ~20s

━━━ Milestone B — AI Serving Baseline ━━━
  [✓ PASS] KServe controller-manager: 1 replica available
  [✓ PASS] InferenceServices: 2/2 Ready=True
  [✓ PASS] Inference request: demo-iris-2 → {"predictions":[1,1]}

━━━ Milestone C — Golden Path ━━━
  [✓ PASS] Backstage deployment: 1 replica available
  [✓ PASS] demo-iris-2 InferenceService exists
  [✓ PASS] demo-iris-2 ArgoCD Application exists

━━━ Milestone D — Guardrails ━━━
  [✓ PASS] Kyverno ClusterPolicies installed: 5 policies
  [✓ PASS] Non-compliant InferenceService correctly denied

━━━ Milestone F — Production Hardening ━━━
  [✓ PASS] ApplicationSet generates 3 child Applications
  [✓ PASS] ResourceQuota exists in default namespace
  [✓ PASS] OpenCost deployment healthy

  PASS 21 / FAIL 0 / SKIP 1

Summary: The After State

Aspect Status
Developer self-service Backstage Golden Path — one form, one PR
Deployment safety 5 Kyverno policies + CI simulation
Configuration drift Auto-healed by ArgoCD in ~20 seconds
CI/CD validation Schema + policy + resource delta on every PR
Inference endpoints Working — Kourier ingress, predictions verified
Cost visibility owner/cost-center labels + OpenCost dashboard
Operational runbooks Documented for every failure mode
Environment reproducibility One-command bootstrap (scripts/bootstrap.sh)
Platform health monitoring 21-check smoke test (scripts/smoke-test.sh)

This is a platform. Self-service, policy-guarded, operationally credible, and reproducible on any machine.