ML Platform Infrastructure

A local ML platform reference implementation — an inference service, its full MLflow/PostgreSQL/MinIO lifecycle, GitOps, autoscaling, security hardening and observability, all running on Kubernetes and validated with real drills instead of descriptions. AWS is designed as code but has not been built yet.

Status: local-v1.0.0. The local implementation (M0–M12) is validated and frozen. AWS work (M13+) has not started — no cloud resource has been created, and Terraform is currently at the design/static-validation level only (fmt, validate, tflint; no plan, no apply).

Try it

git clone https://github.com/negativexq/ml-platform-infrastructure.git
cd ml-platform-infrastructure
make local-up      # fresh kind cluster → GitOps → ML lifecycle → observability, ~15 min
make local-test    # 11-check acceptance suite
make local-down    # tear it all down

No manual step, no pre-existing cluster resource, no registry. Proved for real in M11: cluster, images, and build cache destroyed first, then local-up → local-test from the repo alone.

Milestone status

Milestone		Evidence
M0	Containerized inference service	gate
M1	MLflow + PostgreSQL + MinIO lifecycle	gate
M2	Kubernetes with kind	gate
M3	Helm packaging	gate
M4	GitOps with Argo CD	gate
M5	Observability & failure engineering	gate
M6	Terraform & AWS migration design	gate
M7	Full local Kubernetes platform (no Compose)	gate
M8	Stateful persistence & recovery	gate
M9	Security hardening	gate
M10	Scaling, SLO & alerting	gate
M11	End-to-end reproducibility	gate
M12	Local Release Candidate — this freeze	gate
M13+	AWS — not started, no cloud resource has been created	—

Architecture

                                              client
                                                │
                                                ▼
┌────────────────────── Git (source of truth) ─────────────────────────────────┐
│  helm/ml-platform/   helm/platform-local/   gitops/   infra/terraform/       │
└──────────────────────────────────────────────────────────────────────────────┘
                          │  poll ~3 min · watch + self-heal ~1.4 s
                   ┌──────▼──────┐
                   │   Argo CD   │   Applications: platform-local, inference-local
                   └──────┬──────┘
                          │ apply
┌───────────────────── kind cluster · Pod Security Standards: restricted ──────────────────────┐
│                                                                                              │
│ namespace: ml-platform ──────────────────────────────────────────────────────────────────────│
│                                                                                              │
│  Service ──▶ inference    Deployment · HPA 2↔6 on CPU · PDB minAvailable=1                   │
│                  │  GET /health   GET /ready   POST /predict   GET /metrics                  │
│                  │                                                                           │
│                  │  NetworkPolicy: default-deny + explicit allow-list                        │
│                  ├── allowed ──▶ MLflow ──▶ PostgreSQL   StatefulSet, PVC                    │
│                  │                     └──▶ MinIO        StatefulSet, PVC                    │
│                  └── denied  ──▶ PostgreSQL directly                                         │
│                                                                                              │
│ namespace: observability ────────────────────────────────────────────────────────────────────│
│                                                                                              │
│  Prometheus ── scrapes /metrics ──▶ inference (above)                                        │
│  Prometheus ──▶ Grafana        Prometheus ──▶ Alertmanager  5 rules, promtool-tested         │
│                                                                                              │
└──────────────────────────────────────────────────────────────────────────────────────────────┘

Everything runs inside the cluster; Docker Compose was retired in M7. Argo CD watches live cluster state continuously (drift reverted in ~1.4 s) and polls Git independently (default ~3 min) — the two paths have very different latency, which is why a Git commit lands slower than a manual edit gets reverted (M4). NetworkPolicy denies inference → PostgreSQL directly, verified rather than assumed (M9). Full component notes, the /health vs /ready design decision, and the repository layout: docs/architecture.md.

Key results

Measured on this local kind cluster, not estimated:

k6 load test	645,809 requests, 0% errors
Saturated throughput	2,935 req/s
Saturated /predict p95	32.9 ms
HPA scale-up under load	2 → 6 replicas in 71 s
HPA scale-down after load	6 → 2 in ~230 s, stepped
Pod deleted → replacement serving	12–15 s
Argo drift → reconciled	~1.4 s
Fresh cluster → 11/11 acceptance	901 s (~15 min)

Autoscaling: M10 · pod recovery: M2 · GitOps reconciliation: M4 · reproducibility: M11.

Failure engineering

Eight faults injected on purpose against the running cluster — not simulated. Representative scenarios:

Scenario	Observed	Recovery
Pod crash	ReplicaSet notices, surviving replica keeps serving	replacement ready in 12–15 s
Invalid model artifact	readiness 503, pod held out of Service endpoints	Git revert
Artifact store outage	100% of requests still 200 while unready	background recheck, 0 restarts
Config drift	Argo marks OutOfSync the moment it diverges	self-heal in ~1.4 s
Bad rollout	maxUnavailable: 0 keeps old replicas serving	Git revert, bad ReplicaSet pruned
Node drain (stateful pod on it)	surviving inference pod absorbs traffic	Postgres/MinIO reschedule automatically

Full 8-row table with detection/containment detail: docs/failure-engineering.md.

Real defects this project found in itself

Not written around — found by running the automation (one of them by someone just asking), then fixed. Full writeups: docs/failure-engineering.md.

• Blocking startup — model loading blocked the process; a slow artifact store took /health down with it. Fixed: moved to a background thread.
• Dropped request during rolling update — kubectl rollout status said success while an external probe measured 1 failure in 90. Fixed: preStop drain.
• Empty dashboard panel — a labelled counter emits no series until its first increment, so "zero errors" and "not instrumented" looked identical. Fixed twice with or vector(0).
• Security drill false positive — PSS restricted rejected the drill's own probe pods, and every admission rejection was misread as a network DENY. Fixed: PSS-compliant probe pods.
• MLflow CVE vs. memory trade-off — the 297 MiB version carried 7 unpatched CRITICAL CVEs; every fix lands only in a version with a 1.46 GiB floor. Paid the memory.
• Broken GitHub Actions Trivy scan — an action tag missing its v prefix never resolved, so image-scan silently failed for four milestones while the gate table said PASS. Fixed the pin and corrected the M9 evidence.

Engineering evidence

• Milestone roadmap
• Architecture
• Failure engineering
• SLOs and alerting
• AWS migration design
• Cost model
• M0–M12 evidence transcripts

Current scope

• local-v1.0.0 is a validated local reference implementation — AWS has never been applied.
• Terraform fmt/validate/tflint pass; a real terraform plan/apply against an AWS account has not happened. Detail: docs/aws-architecture.md.
• PostgreSQL and MinIO run single-replica by design — intentionally non-HA for a local lab.
• Alerting uses static thresholds; there is no burn-rate/error-budget alerting yet.
• local-up/local-test were proven manually from a destroyed-and-rebuilt environment (M11) but do not yet run automatically in CI.
• AWS (M13+) will replace each local dependency with its managed equivalent (kind→EKS, MinIO→S3, PostgreSQL→RDS, local image→ECR) and re-run the equivalent gates — it does not change anything above.

Security note

All credentials committed in this repository are disposable local-development defaults (e.g. MinIO's own upstream minioadmin/minioadmin). No production credentials, cloud secrets, proprietary code, or customer data are included.

License

MIT