Phase 6: MCP server, Docker/K8s, observability, CI

[rlienard]

Context

Sixth and final phase. Closes the production-readiness arc: the agent is reachable from any MCP-aware client, the stack deploys in one command (locally or on K8s), every service emits Prometheus metrics + JSON logs, and CI gates lint + a real Postgres test run + a scanned image push.

What's in this PR

services/mcp_server/ — MCP server

Two transports, one shared registry (server.py):

• stdio (python -m services.mcp_server.stdio) — for Claude Code / Desktop and other local clients.
• streamable HTTP (uvicorn services.mcp_server.http:app) — for LibreChat and remote MCP clients.

14 tools spanning runs, classifications, the SGT dictionary, proposals, and threat-intel:

Group	Tools
Runs	start_run, ingest_lines, classify_run, build_matrix, list_runs, get_run, list_classifications, list_missing_sgts
SGT dict	list_sgt_entries, set_sgt_name (gated by --allow-dictionary-edit)
Proposals	list_proposals, get_proposal, create_proposal, approve_proposal, reject_proposal
Threat intel	lookup_threat_intel

Tools are stateless — every call takes an explicit tenant_id / run_id. Per-tenant authorization is enforced by the repo layer (established in Phase 1).

deploy/ — one-command stack

• Dockerfile — multi-stage, non-root (uid 10001), readOnlyRootFilesystem-friendly. One image, every role; pod CMD picks the role.
• docker-compose.yml — postgres + redis + migrate (one-shot) + api + worker + scheduler + mcp-http always-on. Optional ui, webex, threat behind Compose profiles.
• k8s/base/ (kustomize) — Deployments + Services + HPAs (CPU-based for now; KEDA-on-Redis-Streams is a follow-up), a minAvailable: 1 PDB on the scheduler so leader handover is fast during rollouts, postgres + redis as single-replica StatefulSets, migration as a pre-install Job (helm + argocd sync wave annotations), cert-manager + nginx-ingress for the four hostnames, example NetworkPolicy stack with default-deny. PSAS restricted on the namespace.

core/observability/

• logging.py — JSON structured logs (toggleable via SCOPILOT_LOG_FORMAT).
• metrics.py — Prometheus counters for the high-signal events: scopilot_flow_unknown_published_total, scopilot_classifications_total, scopilot_proposals_total, scopilot_threat_lookups_total, plus an http_request_duration_seconds histogram. API exposes /metrics.
• The scheduler scan path now increments flow_unknown_published as proof of plumbing — downstream services adopt the rest as they need it.

.github/workflows/ci.yml

• ruff clean (the cleanup applied 98 fixes across the tree — including a real bug: a duplicate 14: key in the AbuseIPDB category map silently masked "port_scan" behind "scanner").
• Alembic migration applied against a real postgres:16 service container.
• pytest in SQLite (the suite is hermetic).
• Docker build + push to ghcr.io/$REPO (on push to main only) + Trivy scan (warn for now; fail-on-CRITICAL is a follow-up).

Tests — 79/79 passing (+4 new)

• test_mcp_server.py (4): full run lifecycle through the MCP tool surface (start_run → ingest_lines → classify_run → build_matrix); dictionary-edit gating proves set_sgt_name is hidden by default; create_proposal → approve_proposal → list_proposals end-to-end; lookup_threat_intel returns a graceful {"error": …} when no providers are configured.

What this PR explicitly does not do (out of scope)

• OIDC verifier with JWKS — the auth dependency surface from Phase 2 is JWKS-ready; swapping in the verifier is plumbing-only, deferred to a follow-up because it needs an IdP to integration-test against.
• mTLS via Linkerd — short-lived JWTs cover service-to-service for now.
• KEDA Redis-Streams scaler — CPU-based HPAs ship today; KEDA is the next iteration.
• Loki/Tempo exporters — the structured logs and Prometheus metrics drop into any backend; specific exporters are deployment choices.
• GDPR purge cron job — schema + retention guidance is in place; the actual DELETE … WHERE ingestion_ts < now() - 90d cron lives in operations.

End-to-end sanity check

docker compose -f deploy/docker-compose.yml up -d
# api on :8000, mcp-http on :8002, worker + scheduler in the background

# Drive the agent from the CLI client
SCOPILOT_API_BASE=http://localhost:8000 scopilot health
SCOPILOT_API_BASE=http://localhost:8000 scopilot run start tests/fixtures/sample.log

# Or from any MCP client
python -m services.mcp_server.stdio

Test plan

• pytest — 79/79 passing
• ruff check . — clean
• MCP tool surface covered by tests against the FastMCP call_tool API
• Docker image structure validates (multi-stage, non-root, copies app.py)
• Real Docker Compose smoke test (deferred — environment doesn't run Docker)
• Real kubectl apply -k smoke (deferred — no cluster)

---

Generated by Claude Code