Order Agent Reliability Lab

A deliberately small and breakable order-status agent for learning MCP, OpenTelemetry, observability, evaluation, and k3s as one system.

Open architecture explorer · View full-size overview SVG

The lab contains five fake customers, ten fake orders, three MCP tools, twenty golden cases, request-scoped fault injection for offline evaluation, local JSONL traces, and a Kustomize deployment for k3s.

Design invariants

• The MCP server exposes only find_customer, get_order, and get_shipping_status.
• No write endpoint or write tool exists. Mixed read/write requests can complete the read, but the write action is explicitly refused.
• The model chooses tools, but customer prose is rendered from typed ERP evidence. A model draft cannot become an order-status claim.
• Missing, ambiguous, malformed, timed-out, or unavailable data fails closed.
• Customer text and ERP text are untrusted data. Raw prompts and unfiltered ERP payloads are not copied into spans or customer responses; only validated, allowlisted order fields reach the public response.
• The trusted customer_id field is a lab scoping fixture, not authentication.

Quick start on macOS

Install uv if needed, then create the pinned environment and run the key-free suite:

brew install uv
make install
make eval

make eval uses the scripted reference planner. It validates orchestration, grounding, failure behavior, tracing, and the evaluators without spending model tokens. It does not measure real model tool-selection quality. Results and traces are written under artifacts/evals/.

Run the automated checks:

make test
make lint

Run the distributed system locally

Start each process in a separate terminal:

make run-erp
make run-mcp
OPENAI_API_KEY="..." make run-agent

Ask a question:

uv run order-agent \
  --customer-id cust_001 \
  "Where is order 1042, and when will it arrive?"

For a key-free API smoke test, start the agent with ORDER_AGENT_PLANNER=scripted make run-agent. The real planner uses the OpenAI Responses API with gpt-5.6-luna by default and keeps the model configurable. The model selection was checked against the OpenAI model guide on 2026-07-10.

Run the live golden suite or compare the deliberately weak and hardened prompts:

OPENAI_API_KEY="..." make eval-live
OPENAI_API_KEY="..." make compare-prompts

The live experiment records the model, prompt hash, usage, latency, trace ID, tool attempts, structured response, and every evaluator result. A zero score delta is still a measured result; do not assume a prompt change helped without the report.

Repository map

Path	Purpose
src/order_agent_lab/fake_erp.py	GET-only fake ERP API
src/order_agent_lab/mcp_server.py	FastMCP server and three read-only tools
src/order_agent_lab/agent.py	Policy gate, tool audit, and grounded renderer
src/order_agent_lab/planner.py	Scripted and OpenAI planner adapters
src/order_agent_lab/telemetry.py	OTLP, OpenInference, JSONL, and test exporters
evals/golden_cases.json	Twenty hand-written reliability cases
src/order_agent_lab/evaluation.py	Deterministic evaluator and prompt comparator
deploy/k8s/	k3s namespace, workloads, collector, and Phoenix
docs/architecture/	Single-page architecture guide and detailed SVG assets
docs/lesson-01-monitoring-vs-evaluation.md	First interactive lesson

k3s

The manifests deploy the agent, MCP server, fake ERP, OpenTelemetry Collector, and Phoenix as separate workloads in agent-lab. First render them without changing the cluster:

make kustomize

Build the shared application image with make docker-build, import it into the k3s node or point Kustomize at a registry, and then follow deploy/k8s/README.md. No cluster changes are made by the local verification commands.

Instrumentation experiment

The application initializes one OpenTelemetry provider. FastMCP 3.4.4 then emits native MCP client/server spans and propagates trace context between them. The OpenAI adapter is instrumented with OpenInference while the lab's stable lab.operation attributes make evaluator assertions independent of library span names. The Collector exports every lab trace to Phoenix.

After the native baseline is understood, add OpenLIT Operator injection to a separate run. Do not enable native OpenAI instrumentation and injected OpenAI instrumentation together unless duplicate spans are the behavior being studied.

Current interfaces and pins were verified on 2026-07-10 against the FastMCP telemetry guide, FastMCP 3.4.4 release, OpenInference OpenAI instrumentation, and the OpenAI function-calling guide.

Security boundary

Everything here is fake and intended for a private learning cluster. Phoenix has no authentication in this scaffold and remains a ClusterIP service. Keep real credentials out of the repository. If a plaintext credential from earlier notes is still active, rotate it before using this lab.